Multimodal false information detection method, device, equipment and storage medium based on bidirectional feature fusion and hybrid decision-making mechanism

Abstract

The present invention proposes a multimodal false information detection method, device, equipment and storage medium based on bidirectional feature fusion and hybrid decision-making mechanism. The method first performs refined preprocessing on multimodal data, including denoising, stop word removal and normalization of text data, and resolution filtering, deduplication and size unification of image data. Subsequently, the parameter-frozen RoBERTa model is used to extract word-level and sentence-level feature vectors from the text, while the VIT model is used to obtain image-level and word-level feature representations from the image. The features are mapped to dimensionality reduction using the Relu activation function and the fully connected layer to achieve a unified expression of the feature space. An innovative bidirectional feature fusion network is constructed to achieve deep interaction and fusion of text-visual features, generating multimodal features with strong expressive power. The present invention fully utilizes the unique advantages of unimodal information, explores the complementarity between modalities through deep feature fusion, and provides an efficient and robust solution for multimodal false information detection.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing in the field of information technology, and in particular to technical fields such as deep learning and multimodal feature fusion. Specifically, a multimodal false information detection method, device, electronic device and storage medium based on bidirectional feature fusion and hybrid decision-making mechanism are provided. Background Art

[0002] Online social media has rapidly evolved from its original pure text form to a combination of text and images. Early work on false information detection focused on analyzing pure text content, but the detection effect dropped significantly when dealing with false information that combines text and images. Multimodal false information detection emphasizes the use of text and image features as multimodal representations for false information detection. Jin et al. first attempted to use a local attention mechanism to fuse image, text, and social context features for multimodal false information detection. Wang et al. integrated text and image information through feature splicing, and then used a multi-task learning framework to simultaneously perform event classification and false information detection. Wu et al. stacked multiple joint attention layers for multimodal feature fusion. Although these methods effectively fuse multimodal features, they often face the problem of information loss during the fusion process, which may affect performance.

[0003] Other methods have explored cross-modal associations for false information detection. Chen et al. used VAE to compress image and text representations and measured cross-modal consistency based on their Kullback-Leibler (KL) divergence. Wei et al. proposed a two-stage network to establish cross-modal associations by training two unimodal networks through contrastive learning, and then fine-tuned the network for false information detection. However, these methods often rely on a single perspective or decision mechanism, which may limit their ability to capture the complexity of false information. The key issue is how to effectively integrate multiple perspectives and decision strategies to enhance the robustness and accuracy of false information detection.

[0004] In order to solve the problems existing in the above methods, the present invention provides a multimodal false information detection method, device, electronic device and storage medium based on bidirectional feature fusion and hybrid decision-making mechanism. Summary of the Invention

[0005] The present invention aims to provide a multimodal false information detection method, apparatus, device, and storage medium based on a bidirectional feature fusion and hybrid decision-making mechanism. The method acquires a multimodal dataset containing text and images, performs denoising, stop word removal, and normalization on the text, and performs resolution filtering, deduplication, and normalization preprocessing on the image. The preprocessed text data is fed into a RoBERTa pre-trained model with frozen parameters to extract word-level and sentence-level feature vectors, which are then subjected to dimensionality reduction to obtain a final text feature representation. Simultaneously, the image data is normalized and converted into a tensor format before being fed into a VIT pre-trained model with frozen parameters to obtain image-level and word-level feature vectors. The word-level features of the text and image are fed into a bidirectional feature fusion network to generate a fused multimodal feature vector. Finally, multiple classifiers are constructed for false information detection, including text, image, and multimodal detectors. The outputs of each classifier are integrated through a weighted fusion method, and a weighted voting and information entropy-based selection strategy are used to make the final decision. This innovative method effectively improves the overall performance of multimodal false information detection and exhibits broad application prospects.

[0006] The multimodal false information detection method based on bidirectional feature fusion and hybrid decision-making mechanism described in the present invention is carried out according to the following steps:

[0007] a. Obtain a multimodal false information dataset containing text and images, perform denoising, stop word removal, and normalization preprocessing on the text in the dataset, and perform resolution filtering, deduplication, and normalization preprocessing on the image data;

[0008] b. Input the text data preprocessed in step a into a preset RoBERTa pre-trained model. The parameters of the RoBERTa pre-trained model are frozen to obtain text word-level feature vectors and text sentence-level feature vectors. The text sentence-level feature vectors are obtained by performing a pooling operation on all word-level vectors.

[0009] c. Normalize the preprocessed image data in step a using the Normalize method and convert it into a tensor format. Input the tensor into a preset VIT pre-trained model. The parameters of the VIT pre-trained model are frozen to obtain image-level feature vectors and image word-unit-level feature vectors.

[0010] d. The text word-level feature vector and text sentence-level feature vector in step b, and the image word-level feature vector and image-level feature vector in step c are sequentially subjected to dimensionality reduction processing through the ReLU activation function and the fully connected layer. For the image word-level and text word-level feature vectors, the feature vector of each word is separately reduced in dimension to obtain a P-dimensional image word-level representation and text word-level representation. For the text sentence-level and image-level feature vectors, the dimensionality reduction is directly performed to obtain a P-dimensional sentence-level representation and image-level representation;

[0011] e. Construct a bidirectional feature fusion network, input the text word-level feature vector and the image word-level feature vector in step d into the bidirectional feature fusion network to obtain a fused multimodal feature vector;

[0012] f. Construct text, image, and multimodal false information classifiers, and input the text sentence-level feature vector, image-level feature vector, and multimodal feature vector obtained in step d, respectively, into the constructed text false information detector, image false information detector, and multimodal false information detector;

[0013] g. Design a multimodal false information detection algorithm based on a hybrid decision-making mechanism, which includes: combining the outputs of multiple classifiers through a weighted fusion method, and incorporating weighted voting and information entropy-based selection strategies to make the final decision output.

[0014] The detection device in the multimodal false information detection method based on bidirectional feature fusion and hybrid decision-making mechanism is composed of a data preprocessing module, a text feature extraction module, an image feature extraction module, a bidirectional feature fusion module and a hybrid decision-making module; wherein the data preprocessing module is responsible for segmenting and preprocessing the multimodal data to ensure data quality and format consistency for subsequent analysis;

[0015] Text feature extraction module: used to extract word-level and sentence-level feature vectors of text using the pre-trained RoBERTa model, and obtain text representation through feature dimensionality reduction;

[0016] Image feature extraction module: used to extract image features of the image using the pre-trained VIT model, wherein the image features include image-level feature vectors and image word-level feature vectors;

[0017] Bidirectional feature fusion module: used to build a bidirectional feature fusion network to achieve bidirectional interaction and fusion of text word-level features and image word-level features to generate multimodal feature representation;

[0018] A false information detection module is configured to input the text word-level feature representation, the image word-level feature representation, and the multimodal feature representation into a false information detector to obtain a detection result;

[0019] Hybrid decision module: This module integrates the outputs from different feature extractors and generates the final false information detection results through weighted fusion and decision strategies.

[0020] The detection device, data preprocessing module, includes:

[0021] The dataset splitting unit is used to divide the input multimodal dataset into training set, validation set and test set according to the preset ratio;

[0022] A text preprocessing unit, used to perform noise removal, stop word filtering, and text normalization processing on text data;

[0023] Image preprocessing unit, used to implement image resolution threshold filtering, redundant data removal and pixel normalization processing;

[0024] Data tensor conversion unit: used to convert the preprocessed image data into a standard tensor representation through the L2 norm normalization method;

[0025] The text feature extraction module includes:

[0026] The text feature extraction unit uses the parameter-frozen RoBERTa pre-trained model to encode the text, obtain the text word-level feature vector and the text sentence-level feature vector, and obtain the final text feature representation through feature dimensionality reduction;

[0027] The image feature extraction module includes:

[0028] The image feature extraction unit uses a parameter-frozen VIT image encoder to extract image feature vectors and word-unit-level image feature vectors;

[0029] The bidirectional feature fusion module includes:

[0030] Cross-modal feature alignment unit: receives text word-level feature vectors and image word-level feature vectors to achieve feature space alignment;

[0031] Bidirectional feature fusion unit, which implements interactive modeling of text-image features based on a bidirectional attention mechanism and generates cross-modal fusion feature representations;

[0032] The false information detection module includes:

[0033] Text false information detection unit: This unit recognizes false information in text modalities based on sentence-level features.

[0034] Image false information detector unit: realizes image modality false information recognition based on image-level features; multimodal false information detection unit: realizes multimodal false information recognition based on multimodal fusion features;

[0035] The hybrid decision module includes:

[0036] Ensemble learning unit: Adopts adaptive weight fusion algorithm to integrate the prediction results of multiple classifiers;

[0037] Decision optimization unit: Combines the weighted majority voting mechanism and the confidence assessment strategy based on information entropy to output optimized false information detection results.

[0038] An electronic device, comprising:

[0039] The device includes at least one processor; at least one GPU computing card; and a memory communicatively connected to the processor; wherein: the memory stores instructions executed by at least one processor, and the instructions are executed by at least one processor or at least one GPU computing card to enable at least one processor or at least one GPU computing card to execute the described method.

[0040] A non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the method.

[0041] The present invention provides a multimodal false information detection method, device, electronic device, and storage medium based on a bidirectional feature fusion and hybrid decision-making mechanism, wherein:

[0042] Extracting text feature representations and image feature representations of text and images in a target dataset; wherein the text feature representation includes: inputting the preprocessed text into a RoBERTa pre-trained model to obtain word-level feature vectors and sentence-level feature vectors, wherein the RoBERTa model parameters are frozen; and performing dimensionality reduction processing on the word-level feature vectors and sentence-level feature vectors in sequence through a ReLU activation function and a fully connected layer, ultimately obtaining a P-dimensional text feature representation;

[0043] The image feature representation and extraction process are as follows: the pre-processed image data is normalized by the Normalize method and converted into a tensor format, and then input into a preset VIT pre-trained model. The VIT model parameters are frozen to obtain image-level feature vectors and word-level feature vectors;

[0044] The text feature representation and the image feature representation are subjected to feature fusion to obtain a multimodal feature representation; the text word-level feature vector and the image word-level feature vector are input into the constructed bidirectional feature fusion network to obtain a fused multimodal feature vector;

[0045] Construct text, image and multimodal false information classifiers; input the text features, image features and multimodal features obtained in the steps into the text false information detector, image false information detector and multimodal false information detector respectively;

[0046] A multimodal false information detection algorithm based on a hybrid decision-making mechanism is designed; the algorithm combines the outputs of multiple classifiers through a weighted fusion method, and incorporates weighted voting and an information entropy-based selection strategy to make the final decision output.

[0047] The accuracy of false information detection is improved through the multimodal false information detection method, device, electronic device and storage medium based on bidirectional feature fusion and hybrid decision-making mechanism described in the present invention.

[0048] A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute any one of the methods described in the embodiments of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A schematic structural diagram of the detection method provided by the present invention;

[0050] Figure 2 A flow chart of the detection method provided by the present invention;

[0051] Figure 3 A flowchart of the method for extracting text features provided by the present invention;

[0052] Figure 4 A flowchart of the method for extracting image features provided by the present invention;

[0053] Figure 5 Flowchart of the bidirectional feature fusion method provided by the present invention;

[0054] Figure 6 A structural diagram of the bidirectional feature fusion method provided by the present invention;

[0055] Figure 7 A structural diagram of the hybrid decision network provided by the present invention;

[0056] Figure 8 A schematic diagram of the structure of the detection method device provided by the present invention;

[0057] Figure 9 1 is a block diagram of an electronic device of the present invention. DETAILED DESCRIPTION

[0058] To more clearly illustrate the technical solutions of the embodiments of the present invention, the present invention is further described in detail below with reference to the accompanying drawings. Various details of the embodiments of the present application are included to facilitate understanding, and they should be considered as merely exemplary. Therefore, it should be appreciated by those skilled in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0059] Example

[0060] The multimodal false information detection method based on bidirectional feature fusion and hybrid decision-making mechanism described in the present invention is carried out according to the following steps:

[0061] a. Obtain a multimodal false information dataset containing text and images, perform denoising on the text in the dataset, remove special characters and HTML tags, remove stop words, and perform normalization preprocessing on the text, perform resolution filtering on the image data, set a minimum resolution, perform deduplication based on an image hashing algorithm, and perform normalization resizing to 224x224 pixels, and perform JPEG preprocessing on the image data;

[0062] b. Input the preprocessed text data in step a into a preset RoBERTa pre-trained model. The parameters of the RoBERTa pre-trained model are frozen to obtain word-level feature vectors, where each word corresponds to a vector, and sentence-level feature vectors, where the entire sentence corresponds to a vector. The sentence-level feature vectors can be obtained by performing a pooling operation on all word-level vectors, such as average pooling or maximum pooling.

[0063] c. Normalize the image data preprocessed in step a using the Normalize method and convert it into a tensor format. Input the tensor into a preset VIT pre-trained model. The parameters of the VIT pre-trained model are frozen to obtain image-level feature vectors and word-level feature vectors.

[0064] d. The text word-level feature vector and the text sentence-level feature vector in step b, and the image word-level feature vector and the image-level feature vector in step c are sequentially subjected to dimensionality reduction processing through the Relu activation function and the fully connected layer. For the image word-level and text word-level feature vectors, the feature vector of each word is separately reduced in dimension to obtain a P-dimensional image word-level representation and a text word-level representation. For the text sentence-level and image-level feature vectors, the dimensionality reduction is directly performed to obtain a P-dimensional sentence-level representation and an image-level representation; e. A bidirectional feature fusion network is constructed, and the text word-level feature vector and the image word-level feature vector in step d are input into the bidirectional feature fusion network to obtain a fused multimodal feature vector;

[0065] f. Construct text, image, and multimodal false information classifiers, and input the text sentence-level feature vector, image-level feature vector, and multimodal feature vector obtained in step d, respectively, into the constructed text false information detector, image false information detector, and multimodal false information detector;

[0066] g. Design a multimodal false information detection algorithm based on a hybrid decision-making mechanism, which includes: combining the outputs of multiple classifiers through a weighted fusion method, and incorporating weighted voting and information entropy-based selection strategies to make the final decision output.

[0067] A detection device involved in a multimodal false information detection method based on bidirectional feature fusion and hybrid decision-making mechanism, the device is composed of a data preprocessing module, a text feature extraction module, an image feature extraction module, a bidirectional feature fusion module, a false information detection module, and a hybrid decision-making module; wherein:

[0068] Data preprocessing module: responsible for segmenting and preprocessing multimodal data to ensure data quality and format consistency for subsequent analysis;

[0069] Text feature extraction module: This module uses the parameter-frozen RoBERTa model to extract text features and generates representations that can be used for false information detection through feature dimensionality reduction.

[0070] Image feature extraction module: uses the VIT image encoder to extract image features to ensure that the image information can effectively reflect the authenticity of the content;

[0071] Bidirectional feature fusion module: Through cross-modal feature alignment and bidirectional attention mechanism, it promotes deep interaction between text and image features and generates more informative fused feature representations;

[0072] False information detection module: This module combines text and image features to identify false information, including single-modal and multi-modal integrated detection, to improve detection accuracy and robustness.

[0073] Hybrid Decision Module: This module uses an ensemble learning approach to combine the predictions of different models, using a weighted voting mechanism and confidence evaluation to optimize the final false information detection decision.

[0074] A detection device involved in a multimodal false information detection method based on bidirectional feature fusion and hybrid decision-making mechanism, the data preprocessing module includes

[0075] A text preprocessing unit, used to perform noise removal, stop word filtering, and text normalization processing on text data;

[0076] Image preprocessing unit, used to implement image resolution threshold filtering, redundant data removal and pixel normalization processing;

[0077] Data tensor conversion unit: used to convert the preprocessed image data into a standard tensor representation through the L2 norm normalization method;

[0078] The text feature extraction module includes:

[0079] The text feature extraction unit uses the parameter-frozen RoBERTa pre-trained model to encode the text, obtain word-level feature vectors and sentence-level feature vectors, and obtain the final text feature representation through feature dimensionality reduction. The image feature extraction module includes:

[0080] The image feature extraction unit uses a parameter-frozen VIT image encoder to extract image feature vectors and word-unit-level image feature vectors;

[0081] The bidirectional feature fusion module includes:

[0082] Cross-modal feature alignment unit: receives text word-level feature vectors and image word-level feature vectors to achieve feature space alignment;

[0083] Bidirectional feature fusion unit, which implements interactive modeling of text-image features based on a bidirectional attention mechanism and generates cross-modal fusion feature representations;

[0084] The false information detection module includes:

[0085] Text false information detection unit: This unit recognizes false information in text modalities based on sentence-level features.

[0086] Image false information detector unit: It realizes image modality false information recognition based on image level features;

[0087] Multimodal false information detection unit: Based on the bidirectional feature fusion network, multimodal fusion features are obtained to realize multimodal false information identification;

[0088] The hybrid decision module includes:

[0089] Ensemble learning unit: Adopts adaptive weight fusion algorithm to integrate the prediction results of multiple classifiers;

[0090] Decision optimization unit: Combines the weighted majority voting mechanism and the confidence assessment strategy based on information entropy to output optimized false information detection results.

[0091] An electronic device, comprising:

[0092] The device includes at least one processor; at least one GPU computing card; and a memory in communication with the processor; wherein: the memory stores instructions executed by at least one processor, and the instructions are executed by at least one processor or at least one GPU computing card, so that the at least one processor or the at least one GPU computing card can perform any one of the methods in the embodiments of the present application;

[0093] A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute any one of the methods described in the embodiments of the present application;

[0094] Figure 1 It is a structural diagram of the detection method of the present invention; Figure 2 is a flow chart of the detection method of the present invention; the method of the present invention is applicable to the case of identifying false information in the case of multimodal data; the method is performed by a multimodal false information detection device based on bidirectional feature fusion and hybrid decision-making mechanism, and the device is implemented by software and / or hardware;

[0095] Figure 2 A multimodal false information detection method based on bidirectional feature fusion and hybrid decision-making mechanism is provided, including:

[0096] Obtain a multimodal false information dataset containing text and images, perform denoising (removing special characters and HTML tags), stop word removal, and normalization preprocessing on the text in the dataset, and perform resolution filtering (setting a minimum resolution), deduplication (based on an image hashing algorithm), and normalization (resize to 224x224 pixels and unify the format to JPEG) on the image data;

[0097] The preprocessed text data is input into a preset RoBERTa pre-trained model. The parameters of the RoBERTa pre-trained model are frozen to obtain word-level feature vectors (one vector for each word) and sentence-level feature vectors (one vector for the entire sentence). The sentence-level feature vectors can be obtained by performing a pooling operation (e.g., average pooling or maximum pooling) on ​​all word-level vectors.

[0098] In the embodiment, the specific method of extracting word level and text sentence level features can be found in Figure 3 , specifically including:

[0099] For each text, it is encoded into a word sequence, and a special marker [CLS] is added at the beginning of the sequence and a special marker [SEP] is added at the end. The word sequence is input into the RoBERTa pre-trained model and vectorized. Each word corresponds to a vector, which is the word-level feature of the text. The sentence-level feature of the text is obtained by pooling all the word-level vectors.

[0100] For example, the input text is "This is a great product. However, it's also incredibly expensive. #FakeNews#Scam", with a special tag [CLS] added at the beginning and a special tag [SEP] added at the end, [CLS]This is a great product. However, it's also incredibly expensive. #FakeNews#Scam[SEP], and the RoBERTa pre-trained model is input to obtain word-level and text sentence-level features;

[0101] The preprocessed image data is normalized using the Normalize method and converted into a tensor format. The tensor is input into a preset VIT pre-training model. The parameters of the VIT pre-training model are frozen to obtain image-level feature vectors and word-level feature vectors.

[0102] In the embodiment, the specific method of extracting image features is shown in Figure 4 , specifically including:

[0103] The preprocessed image is input into the pre-trained VIT model. Similar to the text model, the image will be divided into different blocks, each block represents a word. The image is input into the model in word sequence and vectorized. Each word corresponds to an image word-level feature, which is obtained by pooling all the image word-level features.

[0104] For example, input the image corresponding to the text This is a great product. However, it's also incredibly expensive. #FakeNews#Scam, and use the transforms method of torchvision to encode it into a word sequence, and input it into the VIT model to obtain image-level features and image word-level features;

[0105] Constructing a bidirectional feature fusion network, inputting the text word-level feature vector and the image word-level feature vector into the bidirectional feature fusion network to obtain a fused multimodal feature vector;

[0106] In the embodiment, the specific method of text features and image features based on the bidirectional feature fusion network is shown in Figure 5 , specifically including:

[0107] Text word-level features token , Image word level features Image token , obtain the text-image similarity matrix Matrix through the similarity calculation function similarity , where each element of Matrix n,m Represents the similarity between the nth text word and the mth image tag; the network uses a bidirectional feature fusion method to identify the text word associated with each image word and the image word relationship associated with each text word; a softmax operation is performed on the similarity matrix to obtain the attention weights α and β of the text and image words; the weighted vector v is calculated based on the attention weights i and h i , these two vectors capture the most important features of their respective modalities; finally, the representation Q is constructed using the original text embedding and weighted features. iT , and extract multimodal features through BERT pooling to obtain the final output R m ;

[0108] For example, the word-level features of the text "This is a great product. However, it's also incredibly expensive. #FakeNews#Scam" and the word-level features of the image are effectively fused through a method based on a bidirectional feature fusion network, thereby improving the feature representation and interaction capabilities of text and image data;

[0109] We comprehensively evaluate input information from different perspectives and construct three types of false information classifiers: text-based, image-based, and multimodal. We use a hybrid decision network to weightedly fuse the outputs of multiple classifiers and adopt weighted voting and entropy-based selection strategies for decision making.

[0110] In the embodiment, the text sentence level features, sentence level image features and multimodal features are respectively input into three false information detectors, and a hybrid decision network is used to make decisions. Figure 7 , specifically including:

[0111] In order to comprehensively evaluate the input information from different perspectives, three false information detectors are constructed: text-based O t , based on image O i and multimodal false information detector O m ;

[0112] Weighted fusion assigns predefined weights to the output of each classifier and calculates the fusion output, F = w t ·O t +w i ·O i +w m ·O m To integrate the information of each classifier; the decision threshold calculates the positive probability F[i,1] of each sample; if the probability is within the preset threshold range θ low and θ high If the fusion output F[i] is between , further refined decision is made; otherwise, the fusion output F[i] is directly used as the final prediction; if refinement is required, the weighted vote Vote of each classifier is calculated. j =w t ·O ^ t,j +w i ·O ^ i,j +w m ·O ^ m,j , if the maximum number of votes meets: max(Vote j )>θ vote , then class j is selected as the final prediction; if no class obtains a majority vote, the entropy of each classifier output is calculated H(O) = -∑p(x)logp(x), and the classifier with the smallest entropy is selected for the final prediction O ^ min =arg min{H(O t ),H(O i ),H(O m )} to improve the confidence of the decision; the final prediction is based on the output of the classifier with majority voting or minimum entropy to ensure the accuracy of the decision; finally, the binary cross entropy loss function is used to calculate the loss of each detector and derive the final comprehensive loss L final =L multimodal +L text +L image ;

[0113] For example, the word-level and sentence-level (image-level) text features and image features of the text "This is a great product. However, it's also incredibly expensive. #FakeNews#Scam" and the corresponding image, as well as the multimodal features obtained by the bidirectional feature fusion network, are input into three fake information detectors respectively, and three detection results are output: the text classifier outputs: t=[0.55,0.45] (positive probability is 0.55, negative probability is 0.45), image classifier output O i =[0.65,0.35] (positive probability is 0.65, negative probability is 0.35), multimodal classifier output O m =[0.45,0.55] (positive probability is 0.45, negative probability is 0.55); set the predefined weight to: w t =0.3, w v =0.2, w m =0.5, and the fusion F=w is obtained by weighted fusion t ·O t +w i ·O i +w m ·O m =0.3·0.55+0.2·0.65+0.5·0.55=0.165+0.13+0.275=0.570. Set the threshold to θ low = 0.4 and θ high =0.7, since the positive probability F[1] = 0.570 in θ low and θ high (0.4<0.545<0.7), so further refined decision-making is needed; calculate the weighted vote for each classifier: Vote j =w t ·O ^ t,j +w i ·O ^ i,j +w m ·O ^ m,j , for the positive class: Vote1 = 0.3·1+0.2·1+0.5·0 = 0.3+0.2+0 = 0.5, for the negative class: Vote0 = 0.3·0+0.2·0+0.5·1 = 0+0+0.5 = 0.5. Assume θ vote = 0.5, check whether the maximum number of votes meets the condition max(Vote j )>θ vote , by calculating max(Vote j )=0.5, which does not meet the conditions, so it is necessary to calculate the entropy of each detector H(O)=-∑p(x)logp(x);

[0114] For text detector: H(O t )=-[0.55log(0.55)+0.45log(0.45)]≈0.991,

[0115] Image detector H(O i)=-[0.65log(0.65)+0.35log(0.35)]≈0.952, multimodal detector H(O m )=-[0.45log(0.45)+0.55log(0.55)]≈0.991;

[0116] The detector with the smallest entropy is selected for the final prediction O ^ min =arg min{H(O t ),H(O i ),H(O m )}=H(O i ), the final prediction is the output of the image detector, which is the negative class (0.35);

[0117] Figure 8 This is a schematic diagram of the structure of a multimodal false information detection device based on a bidirectional feature fusion and hybrid decision-making mechanism of the present invention, comprising: a data preprocessing module, a text feature extraction module, an image feature extraction module, a bidirectional feature fusion module, a false information detection module, and a hybrid decision-making module, wherein:

[0118] Data preprocessing module: responsible for preprocessing multimodal data to ensure data quality and format consistency for subsequent analysis;

[0119] Text feature extraction module: This module uses the parameter-frozen RoBERTa model to extract text features and generates text representations that can be used for false information detection through feature dimensionality reduction.

[0120] Image feature extraction module: responsible for applying the VIT image encoder to extract image features to ensure that the image information can effectively reflect the authenticity of the content;

[0121] Bidirectional feature fusion module: Through cross-modal feature alignment and bidirectional attention mechanism, it promotes deep interaction between text and image features and generates more informative fused feature representations;

[0122] False information detection module: This module combines text and image features to identify false information, including single-modal and multi-modal integrated detection, to improve detection accuracy and robustness.

[0123] Hybrid Decision Module: This module uses an ensemble learning approach to combine the predictions of different models, using a weighted voting mechanism and confidence evaluation to optimize the final false information detection decision.

[0124] Furthermore, the data preprocessing module includes:

[0125] A text preprocessing unit, used to perform noise removal, stop word filtering, and text normalization processing on text data;

[0126] Image preprocessing unit, used to implement image resolution threshold filtering, redundant data removal and pixel normalization processing;

[0127] Data tensor conversion unit: used to convert the preprocessed image data into a standard tensor representation through the L2 norm normalization method;

[0128] Furthermore, the text feature extraction module includes:

[0129] The text feature extraction unit uses the parameter-frozen RoBERTa pre-trained model to encode the text, obtain word-level feature vectors and sentence-level feature vectors, and obtain the final text feature representation through feature dimensionality reduction;

[0130] Furthermore, the image feature extraction module includes:

[0131] The image feature extraction unit uses a parameter-frozen VIT image encoder to extract image feature vectors and word-unit-level image feature vectors;

[0132] Furthermore, the bidirectional feature fusion module includes:

[0133] Cross-modal feature alignment unit: receives text word-level feature vectors and image word-level feature vectors to achieve feature space alignment; bidirectional feature fusion unit, based on the bidirectional attention mechanism, implements interactive modeling of text-image features and generates cross-modal fusion feature representation;

[0134] Furthermore, the false information detection module includes:

[0135] Text false information detection unit: This unit recognizes false information in text modalities based on sentence-level features.

[0136] Image false information detector unit: It realizes image modality false information recognition based on image level features;

[0137] Multimodal false information detection unit: Based on the bidirectional feature fusion network, multimodal fusion features are obtained to realize multimodal false information identification;

[0138] Furthermore, the hybrid decision module includes:

[0139] Ensemble learning unit: Adopts adaptive weight fusion algorithm to integrate the prediction results of multiple classifiers;

[0140] Decision optimization unit: Combines the weighted majority voting mechanism and the confidence assessment strategy based on information entropy to output optimized false information detection results.

[0141] The present invention provides an electronic device and a readable storage medium;

[0142] like Figure 9 The figure shows a block diagram of an electronic device of the present invention. The electronic device refers to various modern electronic digital computers, including personal computers, portable computers, and various server devices. The components and their interconnections and functions shown in this article are only examples.

[0143] like Figure 9 As shown, the electronic device includes: one or more multi-core processors, one or more GPU computing cards, and memory. In order to enable the electronic devices to interact, it should also include: input devices and output devices. Various devices are interconnected and communicated through a bus;

[0144] The memory is a non-transitory computer-readable storage medium provided in this application, wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor or the at least one GPU computing card to enable the at least one processor or the at least one GPU computing card to perform any one of the methods in the embodiments of this application;

[0145] Input devices provide and receive user input into electronic devices. These include keyboards that generate numeric or character information and mice used to control devices and generate other key signals. Output devices provide feedback to users about electronic devices and include displays that print execution results or processes.

[0146] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the disclosure. This application is intended to cover any variations, uses, or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art not disclosed herein. It should be understood that the present invention is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and variations may be made without departing from the scope thereof.

Claims

1. A multimodal false information detection method based on bidirectional feature fusion and hybrid decision-making mechanism, characterized in that: Follow these steps: a. Obtain a multimodal false information dataset containing text and images, perform denoising, stop word removal, and normalization preprocessing on the text in the dataset, and perform resolution filtering, deduplication, and normalization preprocessing on the image data; b. Input the text data preprocessed in step a into a preset RoBERTa pre-trained model. The parameters of the RoBERTa pre-trained model are frozen to obtain text word-level feature vectors and text sentence-level feature vectors. The text sentence-level feature vectors are obtained by performing a pooling operation on all word-level vectors. c. Normalize the preprocessed image data in step a using the Normalize method and convert it into a tensor format. Input the tensor into a preset VIT pre-trained model. The parameters of the VIT pre-trained model are frozen to obtain image-level feature vectors and image word-unit-level feature vectors. d. The text word-level feature vector and text sentence-level feature vector in step b, and the image word-level feature vector and image-level feature vector in step c are sequentially subjected to dimensionality reduction processing through the ReLU activation function and the fully connected layer. For the image word-level and text word-level feature vectors, the feature vector of each word is separately reduced in dimension to obtain a P-dimensional image word-level representation and text word-level representation. For the text sentence-level and image-level feature vectors, the dimensionality reduction is directly performed to obtain a P-dimensional sentence-level representation and image-level representation; e. Construct a bidirectional feature fusion network, input the text word-level feature vector and the image word-level feature vector in step d into the bidirectional feature fusion network to obtain a fused multimodal feature vector; f. Construct text, image, and multimodal false information classifiers, and input the text sentence-level feature vector, image-level feature vector, and multimodal feature vector obtained in step d, respectively, into the constructed text false information detector, image false information detector, and multimodal false information detector; g. Design a multimodal false information detection algorithm based on a hybrid decision-making mechanism, which includes: combining the outputs of multiple classifiers through a weighted fusion method, and incorporating weighted voting and information entropy-based selection strategies to make the final decision output.

2. A detection device in the multimodal false information detection method based on bidirectional feature fusion and hybrid decision-making mechanism according to claim 1, characterized in that: The device is composed of a data preprocessing module, a text feature extraction module, an image feature extraction module, a bidirectional feature fusion module and a hybrid decision module; in Data preprocessing module: responsible for segmenting and preprocessing multimodal data to ensure data quality and format consistency for subsequent analysis; Text feature extraction module: used to extract word-level and sentence-level feature vectors of text using the pre-trained RoBERTa model, and obtain text representation through feature dimensionality reduction; Image feature extraction module: used to extract image features of the image using the pre-trained VIT model, wherein the image features include image-level feature vectors and image word-level feature vectors; Bidirectional feature fusion module: used to build a bidirectional feature fusion network to achieve bidirectional interaction and fusion of text word-level features and image word-level features to generate multimodal feature representation; A false information detection module is configured to input the text word-level feature representation, the image word-level feature representation, and the multimodal feature representation into a false information detector to obtain a detection result; Hybrid decision module: This module integrates the outputs from different feature extractors and generates the final false information detection results through weighted fusion and decision strategies.

3. The detection device according to claim 2, characterized in that Data preprocessing module, including: The dataset splitting unit is used to divide the input multimodal dataset into training set, validation set and test set according to the preset ratio; A text preprocessing unit, used to perform noise removal, stop word filtering, and text normalization processing on text data; Image preprocessing unit, used to implement image resolution threshold filtering, redundant data removal and pixel normalization processing; Data tensor conversion unit: used to convert the preprocessed image data into a standard tensor representation through the L2 norm normalization method; The text feature extraction module includes: The text feature extraction unit uses the parameter-frozen RoBERTa pre-trained model to encode the text, obtain the text word-level feature vector and the text sentence-level feature vector, and obtain the final text feature representation through feature dimensionality reduction; The image feature extraction module includes: The image feature extraction unit uses a parameter-frozen VIT image encoder to extract image feature vectors and word-unit-level image feature vectors; The bidirectional feature fusion module includes: Cross-modal feature alignment unit: receives text word-level feature vectors and image word-level feature vectors to achieve feature space alignment; Bidirectional feature fusion unit, which implements interactive modeling of text-image features based on a bidirectional attention mechanism and generates cross-modal fusion feature representations; The false information detection module includes: Text false information detection unit: This unit recognizes false information in text modalities based on sentence-level features. Image false information detector unit: It realizes image modality false information recognition based on image level features; Multimodal false information detection unit: realizes multimodal false information identification based on multimodal fusion features; The hybrid decision module includes: Ensemble learning unit: Adopts adaptive weight fusion algorithm to integrate the prediction results of multiple classifiers; Decision optimization unit: Combines the weighted majority voting mechanism and the confidence assessment strategy based on information entropy to output optimized false information detection results.

4. An electronic device, characterized in that: include: The device includes at least one processor; At least one GPU computing card; And a memory communicatively connected to the processor; wherein: the memory stores instructions executed by at least one processor, and the instructions are executed by at least one processor or at least one GPU computing card to enable at least one processor or at least one GPU computing card to execute the method described in claim 1.

5. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method described in claim 1.

Images