Multimodal false information detection method, device and equipment based on bidirectional feature fusion and hybrid decision mechanism, and storage medium

Abstract

The invention provides a multimodal false information detection method and device based on bidirectional feature fusion and a hybrid decision mechanism, equipment and a storage medium. The method comprises the following steps: firstly, carrying out refined preprocessing on multi-modal data, including denoising, stop word removal and standardization of text data, and resolution filtering, duplicate removal and size unification processing of image data; then, lexical-level and sentence-level feature vectors are extracted from the text by using a parameter-frozen RoBERTa model, and image-level and lexical-level feature representations are obtained from the image by using a VIT model; dimensionality reduction mapping is carried out on the features through a Relu activation function and a full connection layer, and unified expression of a feature space is achieved. A bidirectional feature fusion network is innovatively constructed, deep interaction and fusion of text-visual features are realized, and multi-modal features with high expression ability are generated. According to the method, unique advantages of single-mode information are fully utilized, complementarity between modes is mined through deep feature fusion, and an efficient and stable solution is provided for multi-mode false information detection.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing in the field of information technology, and in particular to technical fields such as deep learning and multimodal feature fusion. The present invention specifically provides a multimodal false information detection method, device, electronic device and storage medium based on bidirectional feature fusion and hybrid decision-making mechanism. Background Art

[0003] Online social media has rapidly evolved from its original pure text form to a combination of text and images. Early work on false information detection focused on analyzing pure text content, but the detection effect dropped significantly when dealing with false information that combines text and images. Multimodal false information detection emphasizes the use of text and image features as multimodal representations for false information detection. Jin et al. first attempted to use a local attention mechanism to fuse image, text, and social context features for multimodal false information detection. Wang et al. integrated text and image information through feature concatenation, and then used a multi-task learning framework to simultaneously perform event classification and false information detection. Wu et al. stacked multiple joint attention layers for multimodal feature fusion. Although these methods effectively fuse multimodal features, they often face the problem of information loss during the fusion process, which may affect performance.

[0004] Other methods have explored cross-modal associations for false information detection. Chen et al. used VAE to compress image and text representations and measured cross-modal consistency based on their Kullback-Leibler (KL) divergence. Wei et al. proposed a two-stage network to establish cross-modal associations by training two unimodal networks through contrastive learning, and then fine-tuned the network for false information detection. However, these methods often rely on a single perspective or decision mechanism, which may limit their ability to capture the complexity of false information. The key issue is how to effectively integrate multiple perspectives and decision strategies to enhance the robustness and accuracy of false information detection.

[0005] In order to solve the problems existing in the above method, the present invention provides a multimodal false information detection method, device, electronic device and storage medium based on bidirectional feature fusion and hybrid decision-making mechanism. Summary of the invention

[0006] The purpose of the present invention is to provide a multimodal false information detection method, device, equipment and storage medium based on bidirectional feature fusion and hybrid decision mechanism. The method obtains a multimodal data set containing text and images, and performs denoising, stop word removal and standardization on the text, and performs resolution filtering, deduplication and standardization preprocessing on the image; the preprocessed text data is input into the RoBERTa pre-training model with frozen parameters to extract word-level and sentence-level feature vectors, and performs dimensionality reduction processing to obtain the final text feature representation; at the same time, the image data is normalized and converted into a tensor format, and then input into the VIT pre-training model with frozen parameters to obtain image-level and word-level feature vectors; the word-level features of the text and image are input into the bidirectional feature fusion network to generate a fused multimodal feature vector; finally, multiple classifiers are constructed for false information detection, including text, image and multimodal detectors, and the outputs of each classifier are integrated by a weighted fusion method, and a weighted voting and information entropy-based selection strategy are used to make the final decision; this innovative method effectively improves the comprehensive performance of multimodal false information detection and shows a wide range of application prospects.

[0007] The multimodal false information detection method based on bidirectional feature fusion and hybrid decision-making mechanism described in the present invention is carried out according to the following steps:

[0008] a. Obtain a multimodal false information dataset containing text and images, perform denoising, stop word removal and standardization preprocessing on the text in the dataset, and perform resolution filtering, deduplication and standardization preprocessing on the image data;

[0009] b. Input the text data preprocessed in step a into a preset RoBERTa pre-trained model, wherein the parameters of the RoBERTa pre-trained model are frozen to obtain a text word-level feature vector and a text sentence-level feature vector, wherein the text sentence-level feature vector is obtained by performing a pooling operation on all word-level vectors;

[0010] c. The image data in step a after preprocessing is normalized by the Normalize method and converted into a tensor format, and the tensor is input into a preset VIT pre-training model, the parameters of the VIT pre-training model are frozen, and the image-level feature vector and the image-word-level feature vector are obtained;

[0011] d. The text word-level feature vector and the text sentence-level feature vector in step b, and the image word-level feature vector and the image-level feature vector in step c are sequentially subjected to dimensionality reduction processing through the Relu activation function and the fully connected layer. For the image word-level and text word-level feature vectors, the feature vector of each word is respectively reduced in dimension to obtain a P-dimensional image word-level representation and a text word-level representation. For the text sentence-level and image-level feature vectors, the dimensionality reduction is directly performed to obtain a P-dimensional sentence-level representation and an image-level representation;

[0012] e. Construct a bidirectional feature fusion network, input the text word-level feature vector and the image word-level feature vector in step d into the bidirectional feature fusion network, and obtain a fused multimodal feature vector;

[0013] f. Construct text, image and multimodal false information classifiers, and input the text sentence-level feature vector obtained in step d, the image-level feature vector and the multimodal feature vector obtained in step e into the constructed text false information detector, image false information detector and multimodal false information detector respectively;

[0014] g. Design a multimodal false information detection algorithm based on a hybrid decision-making mechanism, which includes: combining the outputs of multiple classifiers through a weighted fusion method, and incorporating weighted voting and information entropy-based selection strategies to make the final decision output.

[0015] The detection device in the multimodal false information detection method based on bidirectional feature fusion and hybrid decision-making mechanism is composed of a data preprocessing module, a text feature extraction module, an image feature extraction module, a bidirectional feature fusion module and a hybrid decision-making module; wherein the data preprocessing module is responsible for segmenting and preprocessing the multimodal data to ensure data quality and format consistency for subsequent analysis;

[0016] Text feature extraction module: used to extract word-level and sentence-level feature vectors of text using the pre-trained RoBERTa model, and obtain text representation through feature dimensionality reduction;

[0017] Image feature extraction module: used to extract image features of an image using a pre-trained VIT model, wherein the image features include image-level feature vectors and image word-level feature vectors;

[0018] Bidirectional feature fusion module: used to build a bidirectional feature fusion network to achieve bidirectional interaction and fusion of text word-level features and image word-level features to generate multimodal feature representation;

[0019] A false information detection module: used to input the text word-level feature representation, the image word-level feature representation and the multimodal feature representation into a false information detector to obtain a detection result;

[0020] Hybrid decision module: This module integrates the outputs from different feature extractors and generates the final false information detection results through weighted fusion and decision strategies.

[0021] The detection device, data preprocessing module, comprises:

[0022] A data set segmentation unit, used to divide the input multimodal data set into a training set, a validation set, and a test set according to a preset ratio;

[0023] A text preprocessing unit, used to perform noise elimination, stop word filtering and text normalization processing on text data;

[0024] An image preprocessing unit, used to implement image resolution threshold filtering, redundant data removal and pixel normalization processing;

[0025] Data tensor conversion unit: used to convert the preprocessed image data into a standard tensor representation through the L2 norm normalization method;

[0026] The text feature extraction module comprises:

[0027] The text feature extraction unit uses the parameter-frozen RoBERTa pre-trained model to encode the text, obtain the text word-level feature vector and the text sentence-level feature vector, and obtain the final text feature representation through feature dimensionality reduction;

[0028] The image feature extraction module comprises:

[0029] An image feature extraction unit, which uses a parameter-frozen VIT image encoder to extract image feature vectors and word-level image feature vectors;

[0030] The bidirectional feature fusion module includes:

[0031] Cross-modal feature alignment unit: receives text word-level feature vectors and image word-level feature vectors to achieve feature space alignment;

[0032] Bidirectional feature fusion unit, which realizes interactive modeling of text-image features based on the bidirectional attention mechanism and generates cross-modal fusion feature representation;

[0033] The false information detection module comprises:

[0034] Text false information detection unit: recognizes false information in text modality based on text sentence-level features;

[0035] Image false information detector unit: realizes image modality false information recognition based on image level features; Multimodal false information detection unit: realizes multimodal false information recognition based on multimodal fusion features;

[0036] The hybrid decision module comprises:

[0037] Ensemble learning unit: Adopts adaptive weight fusion algorithm to integrate the prediction results of multiple classifiers;

[0038] Decision optimization unit: Combines the weighted majority voting mechanism and the confidence assessment strategy based on information entropy to output optimized false information detection results.

[0039] An electronic device, comprising:

[0040] The device includes at least one processor; at least one GPU computing card; and a memory connected to the processor in communication; wherein: the memory stores instructions executed by at least one processor, and the instructions are executed by at least one processor or at least one GPU computing card to enable at least one processor or at least one GPU computing card to execute the method described in claims 1-3.

[0041] A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the method described in claims 1-3.

[0042] The present invention discloses a multimodal false information detection method, device, electronic device and storage medium based on bidirectional feature fusion and hybrid decision-making mechanism, wherein:

[0043] Extracting text feature representation and image feature representation of text and image in the target data set; wherein the text feature representation includes: inputting the preprocessed text into the RoBERTa pre-trained model to obtain word-level feature vectors and sentence-level feature vectors, and the RoBERTa model parameters are frozen; performing dimensionality reduction processing on the word-level feature vectors and sentence-level feature vectors in turn through the ReLU activation function and the fully connected layer, and finally obtaining a P-dimensional text feature representation;

[0044] The image feature representation,extraction process is: the pre-processed image data is normalized by the Normalize method and converted into a tensor format, and input into a preset VIT pre-trained model, wherein the VIT model parameters are frozen to obtain an image-level feature vector and a word-level feature vector;

[0045] The text feature representation and the image feature representation are feature-fused to obtain a multimodal feature representation; the text word-unit-level feature vector and the image word-unit-level feature vector are input into the constructed bidirectional feature fusion network to obtain a fused multimodal feature vector;

[0046] Construct text, image and multimodal false information classifiers; input the text features, image features and multimodal features obtained in the steps into the text false information detector, image false information detector and multimodal false information detector respectively;

[0047] A multimodal false information detection algorithm based on a hybrid decision-making mechanism is designed; the algorithm combines the outputs of multiple classifiers through a weighted fusion method, and incorporates weighted voting and a selection strategy based on information entropy to make the final decision output.

[0048] The accuracy of false information detection is improved by the multimodal false information detection method, device, electronic device and storage medium based on bidirectional feature fusion and hybrid decision-making mechanism described in the present invention.

[0049] A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute any one of the methods described in the embodiments of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 A schematic diagram of the structure of the detection method provided by the present invention;

[0051] Figure 2 A flow chart of the detection method provided by the present invention;

[0052] Figure 3 A flowchart of the method for extracting text features provided by the present invention;

[0053] Figure 4 A flow chart of the method for extracting image features provided by the present invention;

[0054] Figure 5 A flow chart of the bidirectional feature fusion method provided by the present invention;

[0055] Figure 6 A structural diagram of the bidirectional feature fusion method provided by the present invention;

[0056] Figure 7 A structural diagram of the hybrid decision network provided by the present invention;

[0057] Figure 8 A schematic diagram of the structure of the detection method device provided by the present invention;

[0058] Fig. 9 A block diagram of an electronic device of the present invention. DETAILED DESCRIPTION

[0059] In order to more clearly illustrate the technical solution of the embodiment of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings. Various details of the embodiments of the present application are included to facilitate understanding, and they should be considered as merely exemplary. Therefore, it should be appreciated by those of ordinary skill in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0060] Example

[0061] The multimodal false information detection method based on bidirectional feature fusion and hybrid decision-making mechanism described in the present invention is carried out according to the following steps:

[0062] a. Obtain a multimodal false information dataset containing text and images, denoise the text in the dataset, remove special characters and HTML tags, remove stop words and perform standardization preprocessing, perform resolution filtering on the image data, set the minimum resolution, remove duplicates based on the image hash algorithm and standardize the size to 224x224 pixels, and unify the format into JPEG preprocessing;

[0063] b. Input the preprocessed text data in step a into a preset RoBERTa pre-trained model, wherein the parameters of the RoBERTa pre-trained model are frozen to obtain a word-level feature vector, wherein each word-unit corresponds to a vector and a sentence-level feature vector, wherein the entire sentence corresponds to a vector. The sentence-level feature vector can be obtained by performing a pooling operation on all word-unit vectors, such as average pooling or maximum pooling;

[0064] c. The image data preprocessed in step a is normalized by the Normalize method and converted into a tensor format, and the tensor is input into a preset VIT pre-training model, the parameters of the VIT pre-training model are frozen, and an image-level feature vector and a word-level feature vector are obtained;

[0065] d. The text word-level feature vector and the text sentence-level feature vector in step b, and the image word-level feature vector and the image-level feature vector in step c are sequentially subjected to dimensionality reduction processing through the Relu activation function and the fully connected layer. For the image word-level and text word-level feature vectors, the feature vector of each word is respectively reduced in dimension to obtain a P-dimensional image word-level representation and a text word-level representation. For the text sentence-level and image-level feature vectors, the dimensionality is directly reduced to obtain a P-dimensional sentence-level representation and an image-level representation; e. A bidirectional feature fusion network is constructed, and the text word-level feature vector and the image word-level feature vector in step d are input into the bidirectional feature fusion network to obtain a fused multimodal feature vector;

[0066] f. Construct text, image and multimodal false information classifiers, and input the text sentence-level feature vector obtained in step d, the image-level feature vector and the multimodal feature vector obtained in step e into the constructed text false information detector, image false information detector and multimodal false information detector respectively;

[0067] g. Design a multimodal false information detection algorithm based on a hybrid decision-making mechanism, which includes: combining the outputs of multiple classifiers through a weighted fusion method, and incorporating weighted voting and information entropy-based selection strategies to make the final decision output.

[0068] A detection device involved in a multimodal false information detection method based on a bidirectional feature fusion and hybrid decision-making mechanism, the device is composed of a data preprocessing module, a text feature extraction module, an image feature extraction module, a bidirectional feature fusion module, a false information detection module, and a hybrid decision-making module; wherein:

[0069] Data preprocessing module: responsible for segmenting and preprocessing multimodal data to ensure data quality and format consistency for subsequent analysis;

[0070] Text feature extraction module: uses the parameter-frozen RoBERTa model to extract text features and generates representations that can be used for false information detection through feature dimensionality reduction;

[0071] Image feature extraction module: Use VIT image encoder to extract image features to ensure that image information can effectively reflect the authenticity of the content;

[0072] Bidirectional feature fusion module: Through cross-modal feature alignment and bidirectional attention mechanism, it promotes the deep interaction between text and image features and generates more informative fused feature representation;

[0073] False information detection module: combines text and image features to implement false information identification, including single-modal and multi-modal comprehensive detection to improve detection accuracy and robustness;

[0074] Hybrid decision module: It uses ensemble learning methods to combine the prediction results of different models, and optimizes the final false information detection decision through weighted voting mechanism and confidence evaluation;

[0075] A detection device involved in a multimodal false information detection method based on bidirectional feature fusion and hybrid decision-making mechanism, the data preprocessing module includes

[0076] A text preprocessing unit, used to perform noise elimination, stop word filtering and text normalization processing on text data;

[0077] An image preprocessing unit, used to implement image resolution threshold filtering, redundant data removal and pixel normalization processing;

[0078] Data tensor conversion unit: used to convert the preprocessed image data into a standard tensor representation through the L2 norm normalization method;

[0079] The text feature extraction module comprises:

[0080] The text feature extraction unit uses the parameter-frozen RoBERTa pre-trained model to encode the text, obtain the word-level feature vector and the sentence-level feature vector, and obtain the final text feature representation through feature dimensionality reduction; the image feature extraction module includes:

[0081] An image feature extraction unit, which uses a parameter-frozen VIT image encoder to extract image feature vectors and word-level image feature vectors;

[0082] The bidirectional feature fusion module includes:

[0083] Cross-modal feature alignment unit: receives text word-level feature vectors and image word-level feature vectors to achieve feature space alignment;

[0084] Bidirectional feature fusion unit, which realizes interactive modeling of text-image features based on the bidirectional attention mechanism and generates cross-modal fusion feature representation;

[0085] The false information detection module comprises:

[0086] Text false information detection unit: recognizes false information in text modality based on text sentence-level features;

[0087] Image false information detector unit: image modality false information recognition based on image level features;

[0088] Multimodal false information detection unit: Based on the bidirectional feature fusion network, multimodal fusion features are obtained to realize multimodal false information identification;

[0089] The hybrid decision module comprises:

[0090] Ensemble learning unit: Adopts adaptive weight fusion algorithm to integrate the prediction results of multiple classifiers;

[0091] Decision optimization unit: Combines the weighted majority voting mechanism and the confidence assessment strategy based on information entropy to output optimized false information detection results.

[0092] An electronic device, comprising:

[0093] The device includes at least one processor; at least one GPU computing card; and a memory connected to the processor in communication; wherein: the memory stores instructions executed by at least one processor, and the instructions are executed by at least one processor or at least one GPU computing card, so that the at least one processor or the at least one GPU computing card can execute any one of the methods in the embodiments of the present application;

[0094] A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute any one of the methods described in the embodiments of the present application;

[0095] Figure 1 It is a structural diagram of the detection method of the present invention; Figure 2 is a flow chart of the detection method of the present invention; the method of the present invention is applicable to the case of identifying false information in the case of multimodal data; the method is performed by a multimodal false information detection device based on a bidirectional feature fusion and hybrid decision mechanism, and the device is implemented by software and / or hardware;

[0096] Figure 2 A multimodal false information detection method based on bidirectional feature fusion and hybrid decision-making mechanism is provided, comprising:

[0097] Obtain a multimodal false information dataset containing text and images, perform denoising (remove special characters and HTML tags), stop word removal and standardization preprocessing on the text in the dataset, and perform resolution filtering (set minimum resolution), deduplication (based on image hashing algorithm) and standardization (size adjustment to 224x224 pixels, unified format to JPEG) preprocessing on the image data;

[0098] Input the preprocessed text data into a preset RoBERTa pre-trained model, the parameters of which are frozen to obtain a word-level feature vector (one vector for each word-unit) and a sentence-level feature vector (one vector for the entire sentence). The text sentence-level feature vector can be obtained by performing a pooling operation (e.g., average pooling or maximum pooling) on ​​all word-unit vectors;

[0099] In the embodiment, the specific method of extracting word level and text sentence level features is shown in Figure 3 , specifically including:

[0100] For each text, encode it into a word-unit sequence, add a special marker [CLS] at the beginning of the sequence, and add a special marker [SEP] at the end. Input the word-unit sequence into the RoBERTa pre-trained model and output the vector. Each word-unit corresponds to a vector, i.e., the word-unit level feature of the text. The sentence-level feature of the text is obtained by pooling all the word-unit vectors.

[0101] For example, the input text is This is a great product. However, it's also incredibly expensive. #FakeNews#Scam, a special tag [CLS] is added at the beginning, and a special tag [SEP] is added at the end, [CLS] This is a great product. However, it's also incredibly expensive. #FakeNews#Scam[SEP], and the RoBERTa pre-trained model is input to obtain word-level and text sentence-level features;

[0102] The preprocessed image data is normalized by the Normalize method and converted into a tensor format, the tensor is input into a preset VIT pre-trained model, the parameters of the VIT pre-trained model are frozen, and an image-level feature vector and a word-level feature vector are obtained;

[0103] In the embodiment, the specific method of extracting image features is shown in Figure 4 , specifically including:

[0104] The preprocessed image is input into the pre-trained VIT model. Similar to the text model, the image will be divided into different blocks, each block represents a word unit, and the image is input into the model in word unit sequence for vector output; each word unit corresponds to an image word unit level feature, and the image level feature is obtained by pooling all image word unit level features;

[0105] For example, an image corresponding to the text This is a great product. However, it's also incredibly expensive. #FakeNews#Scam is input, and encoded into a word sequence using the transforms method of torchvision, and input into the VIT model to obtain image-level features and image word-level features;

[0106] Constructing a bidirectional feature fusion network, inputting the text word-level feature vector and the image word-level feature vector into the bidirectional feature fusion network to obtain a fused multimodal feature vector;

[0107] In the embodiment, the specific method of text features and image features based on the bidirectional feature fusion network is shown in Figure 5 , specifically including:

[0108] The word-level features Text token , Image word level features Image token , get the text-image similarity matrix Matrix through the similarity calculation function similarity , where each element of Matrix n,m represents the similarity between the nth text word and the mth image tag; the network adopts a bidirectional feature fusion method to identify the text word associated with each image word and the image word relationship associated with each text word; a softmax operation is performed on the similarity matrix to obtain the attention weights α and β of the text and image words; a weighted vector v is calculated based on the attention weights i and h i , these two vectors capture the most important features of their respective modalities; finally, the original text embedding and weighted features are used to construct the representation Q iT , and extract multimodal features through BERT pooling to obtain the final output R m ;

[0109] For example, the word-level features of the text This is a great product. However, it's also incredibly expensive. #FakeNews #Scam and the word-level features of the image are effectively fused through a method based on a bidirectional feature fusion network, thereby improving the feature representation and interaction capabilities of text and image data;

[0110] We comprehensively evaluate input information from different perspectives and construct three false information classifiers: text-based, image-based, and multimodal false information classifiers. We use a hybrid decision network to weightedly fuse the outputs of multiple classifiers and adopt weighted voting and entropy-based selection strategies to make decisions.

[0111] In the embodiment, the text sentence level features, sentence level image features and multimodal features are respectively input into three false information detectors, and a hybrid decision network is used for decision making. Figure 7 , specifically including:

[0112] In order to comprehensively evaluate the input information from different perspectives, three false information detectors are constructed: text-based t , based on image O i Based on multimodal false information detector O m ;

[0113] Weighted fusion assigns a predefined weight to the output of each classifier and calculates the fusion output, F = w t ·O t +w i ·O i +w m ·O m To integrate the information of each classifier; the decision threshold calculates the positive class probability F[i,1] of each sample; if the probability is within the preset threshold range θ low and θ high If the fusion output F[i] is between , further refined decision is made; otherwise, the fusion output F[i] is directly used as the final prediction; if refinement is required, the weighted vote Vote of each classifier is calculated. j =w t ·O^ t,j +w i ·O^ i,j +w m ·O^ m,j , if the maximum number of votes meets: max(Vote j )>θ vote , then class j is selected as the final prediction; if no class obtains the majority vote, the entropy of each classifier output is calculated H(O) = -∑p(x)logp(x), and the classifier with the smallest entropy is selected for the final prediction O^ min = arg min{H(O t ),H(O i ),H(O m )} to improve the confidence of the decision; the final prediction is based on the output of the classifier with majority voting or minimum entropy to ensure the accuracy of the decision; finally, the binary cross entropy loss function is used to calculate the loss of each detector and derive the final comprehensive loss L final =L multimodal +L text +L image ;

[0114] For example, the word-level and sentence-level (image-level) text features and image features of the text This is a great product. However, it's also incredibly expensive. #FakeNews #Scam and the corresponding image, as well as the multimodal features obtained by the bidirectional feature fusion network, are respectively input into three false information detectors, and three detection results are output: the text classifier outputs: t =[0.55,0.45] (positive probability is 0.55, negative probability is 0.45), image classifier output O i =[0.65,0.35] (positive probability is 0.65, negative probability is 0.35), multimodal classifier output O m =[0.45,0.55] (positive probability is 0.45, negative probability is 0.55); set the predefined weight to: w t =0.3, w v =0.2, w m = 0.5, and the fusion F = w is obtained through weighted fusion t ·O t +w i ·O i +w m ·O m =0.3·0.55+0.2·0.65+0.5·0.55=0.165+0.13+0.275=0.570. Set the threshold value to θ low = 0.4 and θ high = 0.7, since the positive probability F[1] = 0.570 in θ low and θ high (0.4 < 0.545 < 0.7), so further refined decision-making is required; calculate the weighted vote for each classifier: Vote j =w t ·O^ t,j +w i ·O^ i,j +w m ·O^ m,j , for the positive class: Vote 1 =0.3·1+0.2·1+0.5·0=0.3+0.2+0=0.5, for negative class: Vote 0 =0.3·0+0.2·0+0.5·1=0+0+0.5=0.5. Assume θ vote = 0.5, check whether the maximum number of votes meets the condition max(Vote j )>θ vote , by calculating max(Vote j)=0.5, which does not meet the condition, so it is necessary to calculate the entropy of each detector H(O)=-∑p(x)logp(x);

[0115] For text detector: H(O t )=-[0.55log(0.55)+0.45log(0.45)]≈0.991,

[0116] Image detector H(O i )=-[0.65log(0.65)+0.35log(0.35)]≈0.952, multimodal detector H(O m )=-[0.45log(0.45)+0.55log(0.55)]≈0.991;

[0117] Select the detector with the smallest entropy for the final prediction O^ min = arg min{H(O t ),H(O i ),H(O m )}=H(O i ), the final prediction is the output of the image detector, i.e. the negative class (0.35);

[0118] Figure 8 The present invention is a structural schematic diagram of a multimodal false information detection device based on a bidirectional feature fusion and hybrid decision-making mechanism, comprising: a data preprocessing module, a text feature extraction module, an image feature extraction module, a bidirectional feature fusion module, a false information detection module and a hybrid decision-making module, wherein:

[0119] Data preprocessing module: responsible for preprocessing multimodal data to ensure data quality and format consistency for subsequent analysis;

[0120] Text feature extraction module: uses the parameter-frozen RoBERTa model to extract text features and generates text representations that can be used for false information detection through feature dimensionality reduction;

[0121] Image feature extraction module: responsible for applying the VIT image encoder to extract image features to ensure that the image information can effectively reflect the authenticity of the content;

[0122] Bidirectional feature fusion module: Through cross-modal feature alignment and bidirectional attention mechanism, it promotes the deep interaction between text and image features and generates more informative fused feature representation;

[0123] False information detection module: combines text and image features to implement false information identification, including single-modal and multi-modal comprehensive detection to improve detection accuracy and robustness;

[0124] Hybrid decision module: It uses ensemble learning methods to combine the prediction results of different models, and optimizes the final false information detection decision through weighted voting mechanism and confidence evaluation;

[0125] Furthermore, the data preprocessing module includes:

[0126] A text preprocessing unit, used to perform noise elimination, stop word filtering and text normalization processing on text data;

[0127] An image preprocessing unit, used to implement image resolution threshold filtering, redundant data removal and pixel normalization processing;

[0128] Data tensor conversion unit: used to convert the preprocessed image data into a standard tensor representation through the L2 norm normalization method;

[0129] Furthermore, the text feature extraction module includes:

[0130] The text feature extraction unit uses the parameter-frozen RoBERTa pre-trained model to encode the text, obtain the word-level feature vector and sentence-level feature vector, and obtain the final text feature representation through feature dimensionality reduction;

[0131] Furthermore, the image feature extraction module includes:

[0132] An image feature extraction unit, which uses a parameter-frozen VIT image encoder to extract image feature vectors and word-level image feature vectors;

[0133] Furthermore, the bidirectional feature fusion module includes:

[0134] Cross-modal feature alignment unit: receives text word-level feature vectors and image word-level feature vectors to achieve feature space alignment; bidirectional feature fusion unit, based on the bidirectional attention mechanism, realizes interactive modeling of text-image features and generates cross-modal fusion feature representation;

[0135] Furthermore, the false information detection module includes:

[0136] Text false information detection unit: recognizes false information in text modality based on text sentence-level features;

[0137] Image false information detector unit: image modality false information recognition based on image level features;

[0138] Multimodal false information detection unit: Based on the bidirectional feature fusion network, multimodal fusion features are obtained to realize multimodal false information identification;

[0139] Furthermore, the hybrid decision module includes:

[0140] Ensemble learning unit: Adopts adaptive weight fusion algorithm to integrate the prediction results of multiple classifiers;

[0141] Decision optimization unit: Combines the weighted majority voting mechanism and the confidence assessment strategy based on information entropy to output optimized false information detection results.

[0142] The present invention provides an electronic device and a readable storage medium;

[0143] like Fig. 9 As shown, it is a block diagram of the electronic device of the present invention. The electronic device refers to various modern electronic digital computers, including personal computers, portable computers, and various server devices. The components and their interconnections and functions shown in this article are only for examples.

[0144] like Fig. 9 As shown, the electronic device includes: one or more multi-core processors, one or more GPU computing cards, and a memory. In order to make the electronic devices interactive, it should also include: an input device and an output device. Various devices are interconnected and communicated through a bus;

[0145] The memory is a non-transitory computer-readable storage medium provided in the present application, wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor or the at least one GPU computing card, so that the at least one processor or the at least one GPU computing card can perform any one of the methods in the embodiments of the present application;

[0146] Input devices provide and receive control signals input by users into electronic devices, including keyboards that generate digital or character information and mice used to control devices to generate other key signals. Output devices provide feedback information from user electronic devices, including displays that print execution results or processes.

[0147] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practice disclosure. This application is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art that are not disclosed in the present invention. It should be understood that the present invention is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from the scope thereof.

Claims

1. A multimodal false information detection method based on bidirectional feature fusion and hybrid decision-making mechanism, characterized in that: Follow these steps: a. Obtain a multimodal false information dataset containing text and images, perform denoising, stop word removal and standardization preprocessing on the text in the dataset, and perform resolution filtering, deduplication and standardization preprocessing on the image data; b. Input the text data preprocessed in step a into a preset RoBERTa pre-trained model, wherein the parameters of the RoBERTa pre-trained model are frozen to obtain a text word-level feature vector and a text sentence-level feature vector, wherein the text sentence-level feature vector is obtained by performing a pooling operation on all word-level vectors; c. The image data in step a after preprocessing is normalized by the Normalize method and converted into a tensor format, and the tensor is input into a preset VIT pre-training model, the parameters of the VIT pre-training model are frozen, and the image-level feature vector and the image-word-level feature vector are obtained; d. The text word-level feature vector and the text sentence-level feature vector in step b, and the image word-level feature vector and the image-level feature vector in step c are sequentially subjected to dimensionality reduction processing through the Relu activation function and the fully connected layer. For the image word-level and text word-level feature vectors, the feature vector of each word is respectively reduced in dimension to obtain a P-dimensional image word-level representation and a text word-level representation. For the text sentence-level and image-level feature vectors, the dimensionality reduction is directly performed to obtain a P-dimensional sentence-level representation and an image-level representation; e. Construct a bidirectional feature fusion network, input the text word-level feature vector and the image word-level feature vector in step d into the bidirectional feature fusion network, and obtain a fused multimodal feature vector; f. Construct text, image and multimodal false information classifiers, and input the text sentence-level feature vector obtained in step d, the image-level feature vector and the multimodal feature vector obtained in step e into the constructed text false information detector, image false information detector and multimodal false information detector respectively; g. Design a multimodal false information detection algorithm based on a hybrid decision-making mechanism, which includes: combining the outputs of multiple classifiers through a weighted fusion method, and incorporating weighted voting and information entropy-based selection strategies to make the final decision output.

2. A detection device in the multimodal false information detection method based on bidirectional feature fusion and hybrid decision-making mechanism according to claim 1, characterized in that: The device is composed of a data preprocessing module, a text feature extraction module, an image feature extraction module, a bidirectional feature fusion module and a hybrid decision module; in Data preprocessing module: responsible for segmenting and preprocessing multimodal data to ensure data quality and format consistency for subsequent analysis; Text feature extraction module: used to extract word-level and sentence-level feature vectors of text using the pre-trained RoBERTa model, and obtain text representation through feature dimensionality reduction; Image feature extraction module: used to extract image features of an image using a pre-trained VIT model, wherein the image features include image-level feature vectors and image word-level feature vectors; Bidirectional feature fusion module: used to build a bidirectional feature fusion network to achieve bidirectional interaction and fusion of text word-level features and image word-level features to generate multimodal feature representation; A false information detection module: used to input the text word-level feature representation, the image word-level feature representation and the multimodal feature representation into a false information detector to obtain a detection result; Hybrid decision module: This module integrates the outputs from different feature extractors and generates the final false information detection results through weighted fusion and decision strategies.

3. The detection device according to claim 2, characterized in that: Data preprocessing module, including: A data set segmentation unit, used to divide the input multimodal data set into a training set, a validation set, and a test set according to a preset ratio; A text preprocessing unit, used to perform noise elimination, stop word filtering and text normalization processing on text data; An image preprocessing unit, used to implement image resolution threshold filtering, redundant data removal and pixel normalization processing; Data tensor conversion unit: used to convert the preprocessed image data into a standard tensor representation through the L2 norm normalization method; The text feature extraction module comprises: The text feature extraction unit uses the parameter-frozen RoBERTa pre-trained model to encode the text, obtain the text word-level feature vector and the text sentence-level feature vector, and obtain the final text feature representation through feature dimensionality reduction; The image feature extraction module comprises: An image feature extraction unit, which uses a parameter-frozen VIT image encoder to extract image feature vectors and word-level image feature vectors; The bidirectional feature fusion module includes: Cross-modal feature alignment unit: receives text word-level feature vectors and image word-level feature vectors to achieve feature space alignment; Bidirectional feature fusion unit, which realizes interactive modeling of text-image features based on the bidirectional attention mechanism and generates cross-modal fusion feature representation; The false information detection module comprises: Text false information detection unit: recognizes false information in text modality based on text sentence-level features; Image false information detector unit: image modality false information recognition based on image level features; Multimodal false information detection unit: multimodal false information recognition based on multimodal fusion features; The hybrid decision module comprises: Ensemble learning unit: Adopts adaptive weight fusion algorithm to integrate the prediction results of multiple classifiers; Decision optimization unit: Combines the weighted majority voting mechanism and the confidence assessment strategy based on information entropy to output optimized false information detection results.

4. An electronic device, characterized in that: include: The device comprises at least one processor; At least one GPU computing card; And a memory communicatively connected to the processor; wherein: the memory stores instructions executed by at least one processor, and the instructions are executed by at least one processor or at least one GPU computing card, so that at least one processor or at least one GPU computing card can execute the method described in claims 1-3.

5. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method described in claims 1-3.

Images