A multi-modal false information identification and credit punishment system and method
By combining multimodal neural network models and blockchain technology, accurate identification and credible credit punishment of cross-modal misinformation have been achieved, solving the problems of single-modality misinformation detection and disconnection between cross-platform governance in existing technologies, and improving the accuracy and long-term effectiveness of misinformation governance.
Patent Information
- Application Number
- CN202511377419.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-25
- Publication Date
- 2026-01-09
AI Technical Summary
Existing technologies for detecting false information are limited to a single modality, making it difficult to deal with complex cross-modal deception methods. Multimodal fusion methods are superficial, with a disconnect between identification and governance, and a lack of a cross-platform trusted credit system.
A multimodal neural network model is used to extract and fuse features from text and image information. A credit punishment system is built by combining blockchain technology. The system dynamically adjusts user points through credit assessment and punishment modules, and uses blockchain to store tamper-proof credit records to achieve trusted sharing across platforms.
It has improved the accuracy and robustness of identifying false information, built a cross-platform long-term credit punishment mechanism, and enhanced the binding force and transparency of false information governance.
Smart Images
Figure CN121303535A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of internet information security, and in particular to a multimodal system and method for identifying and punishing false information. Background Technology
[0002] With the rapid development of the internet, mobile communications, and social media, information dissemination has entered an unprecedented era of high speed and multimodal operation. While cyberspace has become the primary arena for people to obtain information and exchange ideas, it has also become a breeding ground for false information, rumors, and deepfake content. This false information not only seriously misleads public perception and disrupts social order, but may also pose a substantial threat to personal reputation and corporate goodwill. Currently, known existing technological solutions mainly focus on single-modal information detection. In terms of combating false information, existing technologies primarily emphasize content deletion or account banning within platforms. These post-incident measures lack sufficient punitive power, and due to data isolation between platforms, violators can easily register repeatedly across platforms, making it difficult to consolidate the governance effect. Essentially, this is because existing technologies lack a reliable, quantifiable, and tamper-proof credit record and punishment mechanism that can be shared across platforms. Although blockchain technology has precedents in other fields such as credit reporting, there is no mature and effective solution in existing technologies for how to creatively combine it with multimodal false information identification technology to build a closed-loop credit punishment ecosystem.
[0003] In summary, existing technologies have the following main shortcomings: the detection modality for false information is singular, making it difficult to deal with complex cross-modal deception methods; the multimodal fusion methods are superficial and fail to fully explore the deep correlations and consistency between modalities; the identification and governance processes are disconnected, lacking a cross-platform, long-term punishment mechanism based on a trusted credit system. Summary of the Invention
[0004] The main objective of this invention is to provide a multimodal system and method for identifying and punishing misinformation, which can effectively solve the problems of single-modality detection, shallow multimodal fusion, and disconnect between identification and governance in existing technologies.
[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:
[0006] A multimodal misinformation identification and credit punishment system includes: a multimodal misinformation identification module for receiving user-submitted information to be identified, wherein the information to be identified includes at least text and image modalities; this module integrates a trained multimodal neural network model for feature extraction and fusion of the information in the text and image modalities, and outputs the misinformation identification result; a credit assessment and punishment module, communicatively connected to the multimodal misinformation identification module, for triggering dynamic adjustment of the credit score of the information publisher user based on the misinformation identification result; this module has pre-set credit score calculation rules and reward and punishment strategies; a blockchain storage module, communicatively connected to the credit assessment and punishment module, for distributively storing user credit score change records and key data hash values used as the basis for misinformation identification, utilizing the immutability of blockchain technology; and a user interaction and management module for providing users with an information submission interface, a credit score query interface, and an appeal feedback channel, and providing administrators with a community monitoring, credit adjustment review, and system configuration interface.
[0007] Preferably, the multimodal fake information identification module includes: a text processing submodule, used to preprocess the input text information and extract deep semantic features using a pre-trained natural language processing model BERT to generate a text feature vector; an image processing submodule, used to preprocess the input image information and extract visual features using a pre-trained convolutional neural network model VGG to generate an image feature vector; a feature fusion submodule, using an attention-based fusion network to weightedly fuse the text feature vector and the image feature vector to generate a comprehensive feature vector; and a classification output submodule, used to input the comprehensive feature vector into a classifier and output the probability that the information to be identified is fake information or real information and the final identification result.
[0008] Preferably, the pre-set credit score calculation rules in the credit assessment and punishment module include: basic behavior score rules, which quantify and score users' basic behaviors such as posting information and participating in interactions; quality contribution score rules, which award additional points based on content quality and community contribution; and violation punishment score rules, which deduct credit points from users confirmed to have posted false information according to preset standards based on the output results of the multimodal false information identification module. The reward and punishment strategy includes: linking users' credit scores to community rights, with users with high scores enjoying priority participation and redemption rights, while users with low scores are subject to functional restrictions and permission constraints.
[0009] Preferably, the blockchain storage module is specifically used to: form a data block by recording each credit score change operation, the hash value of the associated false information content, and the operation timestamp; and encrypt the data block through a consensus mechanism and append it to the existing blockchain to form a traceable and tamper-proof credit operation log.
[0010] Preferably, the system further includes: an information tracing module, used to trace the source and dissemination path of false information after it is identified, and record the associated user identifiers; and a credit assessment and punishment module, which, based on the tracing results, implements differentiated credit score punishment measures against the original publisher of the false information and those who knowingly spread false information on a large scale.
[0011] A multimodal misinformation identification and credit punishment method is applied to the system described above. The method includes: Step S1: Receiving multimodal information to be identified submitted by a user through the user interaction and management module; Step S2: The multimodal misinformation identification module extracts the text features and image features of the information, performs fusion analysis, and generates an identification result; Step S3: The credit assessment and punishment module, based on the identification result, calls a preset scoring rule to calculate and generate a credit score adjustment scheme for the corresponding user; Step S4: The credit score adjustment scheme and key data hash value are stored as evidence through the blockchain storage module; Step S5: The user interaction and management module notifies the user of the score adjustment result and provides an appeal portal.
[0012] Preferably, the specific steps of feature fusion analysis in step S2 include: Step S21: Encoding text information using a pre-trained BERT model to obtain a text feature vector; Step S22: Encoding image information using a pre-trained VGG model to obtain an image feature vector; Step S23: Inputting the text feature vector and the image feature vector into a fusion layer based on an attention mechanism to calculate the attention weights of each modality feature; Step S24: Weighting and summing the text and image features according to the attention weights to obtain a fused comprehensive feature vector; Step S25: Inputting the comprehensive feature vector into a classifier to obtain the final false information identification result.
[0013] Preferably, after generating the credit score adjustment scheme in step S3 and before step S4, the following steps are included: Step S3a: For users who are first determined to have published false information and whose score deduction reaches a certain threshold, a manual review process is triggered; Step S3b: After review and confirmation by the administrator, the final score adjustment scheme and review record are sent to the blockchain storage module for evidence storage.
[0014] Compared with the prior art, the present invention has the following beneficial effects:
[0015] 1. This invention provides a multimodal misinformation identification and credit punishment system and method. The text processing submodule uses a pre-trained BERT model to extract deep semantic features of the text, and the image processing submodule uses a pre-trained VGG model to extract visual features of the image. The feature fusion submodule based on the attention mechanism performs weighted fusion of text and image features to explore deep correlations between modalities. Finally, the classification output submodule outputs accurate identification results, which effectively improves the accuracy and robustness of cross-modal misinformation identification and can cope with complex scenarios of text and image collusion forgery.
[0016] 2. This invention provides a multimodal system and method for identifying and punishing false information. It pre-sets three types of rules—basic behavior points, quality contribution points, and violation punishment points—through a credit assessment and punishment module. Combined with an information tracing module, it tracks the source and spread path of false information, imposing differentiated credit punishments on the original publisher and those who knowingly spread false information on a large scale. Credit points are linked to community rights. Simultaneously, relying on a blockchain storage module, it achieves reliable sharing of credit records across platforms, preventing violators from repeatedly registering and committing malicious acts across platforms. This constructs a closed-loop governance ecosystem of identification, assessment, punishment, and evidence storage, significantly improving the long-term effectiveness and binding force of false information governance.
[0017] 3. This invention provides a multimodal system and method for identifying and punishing false information. It uses a blockchain storage module to form data blocks containing each credit score change record, the hash value of the associated false information content, and the operation timestamp. These blocks are then encrypted using a consensus mechanism and appended to the blockchain, creating an immutable and traceable credit operation log. This design ensures the integrity and credibility of user credit data, providing reliable data support for cross-platform credit assessment, while also guaranteeing the traceability of the basis for identifying false information, effectively enhancing the credibility and transparency of the system's credit punishment. Attached Figure Description
[0018] Figure 1 This is a schematic diagram of the system architecture of the present invention;
[0019] Figure 2 This is a schematic diagram of the method flow of the present invention;
[0020] Figure 3 This is a schematic diagram of the input and output of the BERT pre-training of this invention;
[0021] Figure 4 This is a diagram of the Transformer model architecture of the present invention;
[0022] Figure 5 This is an example diagram of the VGG model of the present invention. Detailed Implementation
[0023] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below in conjunction with specific embodiments.
[0024] like Figure 1-5 As shown, this invention provides a multimodal system for identifying and punishing false information, which mainly includes the following four core modules:
[0025] Multimodal misinformation identification module
[0026] This module is used to receive user-submitted information containing at least text and image modalities, extract and fuse features from the text and image information by integrating a pre-trained multimodal neural network model, and output the result of identifying false information; this module further includes:
[0027] The text processing submodule uses a pre-trained BERT model to extract deep semantic features from the text, generating text feature vectors. BERT is a Transformer-based neural network model, such as... Figure 3 As shown, the Transformer is a neural network structure based on a self-attention mechanism. It can capture dependencies in a sequence and generate encoding vectors with powerful expressive capabilities, such as... Figure 4 As shown, Transformer is essentially an encoder-decoder architecture;
[0028] Image processing submodule: Uses a pre-trained VGG model to extract visual features from images, generating image feature vectors. The VGG model extracts low- to high-level features from the image by stacking multiple convolutional layers, pooling layers, and fully connected layers. This is a crucial step in extracting useful information from the image. Figure 5 As shown, the output of a specific convolutional or pooling layer in the VGG model is selected as the result of feature extraction. These feature maps contain local information and spatial structure in the image and can serve as feature representations of the image. The steps for feature extraction using a pre-trained CNN model are as follows:
[0029] a. Loading a pre-trained model: First, load a pre-trained CNN model from the open-source library TensorFlowHub;
[0030] b. Selecting feature layers: In a pre-trained model, convolutional layers of different levels will extract features of different levels. Lower-level convolutional layers usually extract low-level features such as edges and textures, while higher-level convolutional layers extract more abstract and higher-level features. Depending on the task requirements, appropriate feature layers can be selected for feature extraction.
[0031] c. Forward propagation: The preprocessed image is input into the pre-trained model, and the output of the selected feature layer is obtained through forward propagation. These outputs are the feature representations of the image.
[0032] Feature fusion submodule: Employs an attention-based fusion network to weight and fuse text and image feature vectors to generate a comprehensive feature vector. The feature fusion submodule is a key part of the multimodal fake information recognition model, responsible for effectively integrating features from different modalities to form a comprehensive and powerful feature representation.
[0033] Classification output submodule: Input the comprehensive feature vector into the classifier and output the probability of false information and the final recognition result.
[0034] Credit assessment and punishment module
[0035] It communicates with the recognition module and triggers dynamic adjustments to the user's credit score based on the recognition results. This module has the following preset rules and strategies:
[0036] Credit score calculation rules:
[0037] Basic Behavior Points: Quantifying and awarding points for basic behaviors such as posting information and participating in interactions;
[0038] Quality Contribution Points: Additional rewards based on content quality and community contribution;
[0039] Violation penalty points: Users confirmed to have published false information will have points deducted according to preset standards;
[0040] Reward and punishment strategy: Link credit scores to user rights, with high-scoring users enjoying priority and redemption rights, while low-scoring users are restricted;
[0041] Blockchain storage module
[0042] Key data such as credit score change records, hash values of false information content, and timestamps are used for distributed storage. The data is encrypted through a consensus mechanism and appended to the blockchain to form an immutable and traceable credit operation log.
[0043] User Interaction and Management Module
[0044] It provides users with interfaces for information submission, credit inquiry, and appeal feedback; and provides administrators with interfaces for community monitoring, credit review, and system configuration.
[0045] Information traceability module
[0046] It is used to track the source and spread of false information, record associated user identifiers, and support differentiated credit punishment.
[0047] In this embodiment, it is necessary to define the feature representations output by the text processing module and the image processing module. The text processing module obtains text feature vectors through Word2Vec and BERT, while the image processing module obtains image feature vectors through VGG and ROI pooling. These two feature vectors represent key information in the text and image, respectively. These two feature vectors are then concatenated to form a joint feature vector. This joint feature vector contains information from both text and image modalities and forms the basis for subsequent feature fusion. An attention mechanism layer is then introduced to achieve feature fusion. The core of the attention mechanism layer is a weight matrix, which automatically adjusts the importance of different features through learning. Specifically, the dimension of the weight matrix matches the dimension of the joint feature vector, and each element represents a corresponding feature. During training, the model adjusts the weight matrix based on feedback from fake information identification. Through backpropagation and gradient descent optimizer, the model gradually learns which features are more important for identifying fake information and adjusts the weight matrix accordingly. After obtaining the weight matrix, it performs a dot product with the joint feature vector to obtain a weighted feature vector. This weighting operation actually adjusts the importance of different features, giving greater weight to features that contribute more to the identification task. Finally, the weighted feature vector is further processed through a fully connected layer or other nonlinear transformation layer to obtain the final fused feature representation. This fused feature representation includes both textual and image information, and the importance of different features is reasonably adjusted.
[0048] like Figure 1-5 As shown, this invention provides a method for multimodal misinformation identification and credit punishment, the method comprising the following steps:
[0049] Step S1: Information Submission
[0050] Users submit multimodal information (text + image) to be recognized through the interactive module;
[0051] Step S2: Multimodal Feature Extraction and Fusion
[0052] S21: Use the BERT model to encode the text and obtain the text feature vector;
[0053] S22: Encode the image using the VGG model to obtain the image feature vector;
[0054] S23: Input text and image features into the attention-based fusion layer and calculate the attention weights for each modality;
[0055] S24: Calculate the weighted summation based on the weights to obtain the comprehensive feature vector;
[0056] S25: Input the comprehensive feature vector into the classifier and output the recognition result;
[0057] Step S3: Credit Score Adjustment
[0058] Based on the identification results, the system calls the preset points rules to generate a credit score adjustment plan; if a user is judged to have published false information for the first time and the points deducted reach the threshold, the manual review process is triggered; after the administrator reviews and confirms, the final adjustment plan and review record are sent to the blockchain storage module.
[0059] Step S4: Blockchain Evidence Storage
[0060] The credit score adjustment plan and related data hash values will be distributed and stored through a blockchain storage module.
[0061] Step S5: Notification of Results and Appeals
[0062] The user interaction module will notify users of the points adjustment results and provide an appeal portal.
[0063] In this embodiment, multimodal datasets such as weibo23 are used for training; the text unimodal model uses BERT, the image unimodal model uses VGG, and the multimodal model uses an attention mechanism for fusion; training parameters: learning rate 0.0005~0.001, batch size 32~64, number of iterations 100 rounds, optimizer is Adam, and loss function is cross-entropy; evaluation metrics include accuracy, recall, F1 score, etc., performance is evaluated using validation and test sets, and an early stopping strategy is used to prevent overfitting; unimodal (text / image) and multimodal detection interfaces are provided; after the user uploads information and clicks prediction, the system returns the recognition result; the recognition result and credit score changes are synchronized to the blockchain in real time, and users can query historical records and file appeals. Credit scores are updated in real time and are directly related to user behavior; reward and punishment measures such as credit score redemption, permission adjustment, and function restriction are supported; a transparent credit score change record and appeal mechanism are provided; all credit operation records are stored on the blockchain in the form of hash values to ensure that the data is tamper-proof and traceable; consensus mechanisms such as PoW and PoS are used to ensure distributed consistency.
[0064] The working principle of this multimodal misinformation identification and credit punishment system will be explained in detail below.
[0065] like Figure 1-5As shown, users submit information containing text and images to be identified through the interaction module. In the multimodal recognition module, semantic features of text are extracted using BERT, and visual features of images are extracted using VGG. The features are then fused through an attention mechanism and classified to output false / true results. The credit assessment module generates a credit adjustment plan based on basic behavior, quality contribution, and violation penalty points. The first deduction triggers manual review. At the same time, the information traceability module tracks the source and spread path of false information, and the credit module implements differentiated penalties accordingly. The blockchain module encrypts and stores the point change records, information hash values, and timestamps to form an immutable log. Finally, the interaction module notifies users of the results and provides an appeal channel, as well as a monitoring and review interface for administrators, forming a closed-loop process.
[0066] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.
Claims
1. A multimodal system for identifying and punishing misinformation, characterized in that: include: A multimodal misinformation identification module is used to receive information to be identified submitted by a user, wherein the information to be identified includes at least text modality and image modality; This module integrates a trained multimodal neural network model, which is used to extract and fuse features from the text modality and image modality, and output the results of false information identification. The credit assessment and punishment module is communicatively connected to the multimodal false information identification module and is used to trigger dynamic adjustment of the credit score of the information publisher user based on the false information identification result; this module has preset credit score calculation rules and reward and punishment strategies. The blockchain storage module is communicatively connected to the credit assessment and punishment module, and is used to utilize the immutability of blockchain technology to distribute and store user credit score change records and key data hash values that serve as the basis for identifying false information. The user interaction and management module provides users with an information submission interface, a credit score query interface, and an appeal feedback channel, and provides administrators with an interface for community monitoring, credit adjustment review, and system configuration.
2. The multimodal false information identification and credit punishment system according to claim 1, characterized in that: The multimodal fake information identification module includes: a text processing submodule, used to preprocess the input text information and extract deep semantic features using a pre-trained natural language processing model BERT to generate text feature vectors; an image processing submodule, used to preprocess the input image information and extract visual features using a pre-trained convolutional neural network model VGG to generate image feature vectors; a feature fusion submodule, which uses an attention-based fusion network to weight and fuse the text feature vectors and image feature vectors to generate a comprehensive feature vector; and a classification output submodule, used to input the comprehensive feature vector into a classifier and output the probability that the information to be identified is fake or real information and the final identification result.
3. The multimodal false information identification and credit punishment system according to claim 1, characterized in that: The pre-set credit score calculation rules in the credit assessment and punishment module include: basic behavior score rules, which quantify points for users' basic behaviors such as posting information and participating in interactions; quality contribution score rules, which award additional points based on content quality and community contribution; and violation punishment score rules, which deduct credit points from users confirmed to have posted false information according to preset standards based on the output results of the multimodal false information identification module. The reward and punishment strategy includes: linking user credit scores to community rights, with users with high scores enjoying priority participation and redemption rights, while users with low scores are subject to functional restrictions and permission constraints.
4. The multimodal false information identification and credit punishment system according to claim 1, characterized in that: The blockchain storage module is specifically used to: form a data block by recording each credit score change operation, the hash value of the associated false information content, and the operation timestamp; and encrypt the data block through a consensus mechanism and append it to the existing blockchain to form a traceable and tamper-proof credit operation log.
5. A multimodal false information identification and credit punishment system according to any one of claims 1 to 4, characterized in that: The system also includes: an information tracing module, used to trace the source and dissemination path of false information after it is identified, and to record the associated user identifiers; and a credit assessment and punishment module, which, based on the tracing results, implements differentiated credit score punishment measures against the original publishers of false information and those who knowingly spread false information on a large scale.
6. A method for multimodal misinformation identification and credit punishment, applied to a multimodal misinformation identification and credit punishment system as described in any one of claims 1-5, characterized in that, The method includes: Step S1: Receiving multimodal information to be identified submitted by the user through the user interaction and management module; Step S2: The multimodal misinformation identification module extracts the text features and image features of the information, performs fusion analysis, and generates an identification result; Step S3: The credit assessment and punishment module, based on the identification result, calls the preset points rules to calculate and generate a credit point adjustment scheme for the corresponding user; Step S4: The credit point adjustment scheme and key data hash value are stored in the blockchain storage module; Step S5: The user interaction and management module notifies the user of the points adjustment result and provides an appeal portal.
7. The method for multimodal false information identification and credit punishment according to claim 6, characterized in that: The specific steps of feature fusion analysis in step S2 include: Step S21: Encoding text information using a pre-trained BERT model to obtain text feature vectors; Step S22: Encoding image information using a pre-trained VGG model to obtain image feature vectors; Step S23: Inputting the text feature vectors and image feature vectors into an attention-based fusion layer to calculate the attention weights of each modality feature; Step S24: Weighting and summing the text and image features according to the attention weights to obtain the fused comprehensive feature vector; Step S25: Inputting the comprehensive feature vector into a classifier to obtain the final false information identification result.
8. The method for multimodal false information identification and credit punishment according to claim 6, characterized in that: After generating the credit score adjustment plan in step S3, the process before step S4 includes: Step S3a: For users who are first judged to have published false information and whose points have been deducted to a certain threshold, a manual review process is triggered; Step S3b: After the administrator reviews and confirms, the final score adjustment plan and the review record are sent to the blockchain storage module for evidence storage.