A quality-guided progressive fusion multi-modal rumor detection method

Abstract

The application relates to the technical field of artificial intelligence, in particular to a quality-guided gradual fusion multi-modal rumor detection method, which comprises the following steps: firstly, extracting features of each mode; secondly, converting each mode into a scene graph based on an LLM and performing knowledge enhancement; thirdly, calculating the quality score of each mode according to the structure and entity semantics of the scene graph, and extracting scene graph features of each mode using an SGAT; on this basis, fusing the scene graph features and original features obtained in the first two steps, inputting the fused features and the quality score into a core mode-guided gradual fusion process, and calculating the dynamic quality score between each layer of the gradual fusion; secondly, fusing the final update result of the core mode and the final result of the secondary mode to obtain the final fusion feature of the event; finally, inputting the obtained final fusion feature into a classifier; meanwhile, using an explanation module to make certain explanation on the prediction made by the classifier in combination with part of the intermediate results in the quality score calculation.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to a quality-guided progressive fusion multimodal rumor detection method. Background Technology

[0002] With the rapid development of the internet, online information has exploded, including numerous rumors that could have serious consequences, such as impacting social stability. Manual methods for dealing with rumors are far from sufficient to meet real-world needs, making the development of automated rumor detection methods crucial. Automated rumor detection has evolved from unimodal to multimodal approaches, from content-based to context-based methods, and from focusing on classification methods to emphasizing the robustness of models in real-world scenarios. Early rumor detection relied heavily on unimodal text analysis, incorporating supplementary features such as post comments, propagation structures, and user characteristics on top of words. Current research consensus advocates integrating multimodal information such as text, images, and videos to effectively improve detection results. The core challenge lies in finding more effective feature fusion methods. Simultaneously, researchers in rumor detection are focusing on solving practical application challenges, such as handling modality loss and deepfakes, while also prioritizing rapid response and early warning in the early stages of rumor emergence.

[0003] However, despite the rapid development of rumor detection research in recent years, some issues remain unaddressed. For example, the composition of post information is diverse; the relationship between images and text is no longer simply a matter of images and text or text and images. Images can contain substantial amounts of text, and sometimes the text within an image is the main message. Therefore, it is necessary to focus on the fine-grained characteristics of posts and the logical connections between various modalities. The quality of various modalities, such as images, text, text within images, and external knowledge graphs, varies, and simply treating them as equally important and merging them is no longer sufficient to address the complex relationships between modalities. Furthermore, rumors spread rapidly once they emerge, and detecting them after they have caused significant social impact is no longer practical. Therefore, it is necessary to focus on the content of the post itself for initial detection. Summary of the Invention

[0004] The purpose of this invention is to provide a quality-guided progressive fusion multimodal rumor detection method that can effectively focus on the fine-grained features of posts, pay attention to the logical connections between different modalities of posts, solve the problem of complex quality inconsistencies between modalities, and focus on content detection in the early stages of post appearance, thereby solving the problems mentioned in the background art.

[0005] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0006] A quality-guided progressive fusion multimodal rumor detection method, comprising:

[0007] S100. Extracting Features from Each Modality: The input dataset contains four modalities: event text, event image, external knowledge text, and external knowledge image. The external text and external knowledge image were obtained by searching and reverse searching for event-related information in a browser, respectively. Next, Paddleocr was used to extract the text from the input event image and external knowledge image, treating them as independent modalities. Then, BERT+LSTM and Swin Transformer were used to extract features from the four text modalities and the two image modalities, respectively, resulting in features for six modalities.

[0008] S200, Scene Graph Generation and Analysis: An LLM-based scene graph generation method is used, with external knowledge linking performed when calling the large model to obtain an information-enhanced scene graph. Next, quality scores are calculated based on the scene graph's structural features and semantic features between entities in different modalities. Finally, SGAT is used to encode the generated scene graph to obtain scene graph features. This section ultimately yields the scene graph features and quality scores for each modality.

[0009] S300. Progressive fusion guided by core modalities and incorporating dynamic quality scores: First, the scene graph features obtained from the scene graph generation and analysis steps are combined with the original features obtained from the modal feature extraction steps to obtain fused features. Then, core modalities are selected based on the quality scores obtained from the scene graph generation and analysis steps. In each layer of progressive fusion, a Transformer is applied to the core modality, and core modality-guided cross-attention is performed on all secondary modalities other than the core modality. Simultaneously, a dynamic quality score update mechanism exists between each layer. The dynamic quality score is related to the updated core modality obtained at that layer, the current secondary modality, and the initial quality score. Finally, the final update result of the core modality is fused with the final result of the secondary modalities to obtain a comprehensive feature of the event, which is the final feature of the event.

[0010] S400, Classification and Interpretation: The enhanced features are mapped to the class space, and finally, the output is probabilistically converted using Softmax. To demonstrate the interpretability of this method, a simple interpretation module is designed here, using intermediate calculation results obtained from part of the scene graph generation and analysis steps to provide some explanation for the choice of classifier.

[0011] Preferably, in S100, the PaddleOCR model is used to extract text from event images and external knowledge images, and BERT+LSTM is used to extract features from all text modalities, and SwinTransformer is used to extract features from all image modalities, resulting in six modal features of an event.

[0012] Preferably, S200 includes:

[0013] Using BLIP2-FlanT5-xl, all six modalities of the event are processed into text, and the API is called to extract entities, relationships between entities, and supplementary descriptions of entities from the text to obtain a knowledge-enhanced scene graph.

[0014] Based on the scene graph, the internal information richness, internal modality consistency, and external knowledge consistency are calculated respectively, and the quality score is further calculated to obtain the scene graph features and quality score for each modality;

[0015] Specifically, it includes:

[0016] S201, Calculate the richness of internal information :

[0017] Get the number of unique nodes in the scene graph for each modality. The relationship between scene diagrams and diversity Calculate graph density This indicates that the number of relationships is greater than the number of nodes; therefore, it is based on the number of unique nodes. Relationship diversity And graph density Obtain the richness of internal information :

[0018] ;

[0019] in, , and This indicates the number of unique nodes. Relationship diversity And graph density All metrics are standardized to between 0 and 1;

[0020] S202, Calculate internal consistency :

[0021] The six modalities were divided into two groups: three modalities within the event and three modalities within the external knowledge, to evaluate the degree of mutual support between each modality within the group;

[0022] First, for each modality within each of the three modalities, extract all entities from its scene graph and form a set: Divide all entities of all modalities into a set to be computed, A, and a set to be compared, then... As the set to be computed, the remaining modes within the group Merge as comparison set B;

[0023] For all n entities in set A: Calculate the comparison set for each entity in A. Find the minimum Manhattan distance for all entities in A, where the minimum represents the distance of that entity to its "most similar partner" in another modality; then average the minimum values ​​for all entities in A to obtain the internal consistency of the current modality. ;

[0024] Specifically, calculation The specific formula is as follows:

[0025] First, convert all entities into vectors and divide them into a set to be computed and a set to be compared:

[0026] ;

[0027] ;

[0028] in, , This represents the entity vector formed by the entity after being encoded by BERT;

[0029] Secondly, calculation entity vectors in Minimum Manhattan distance of the entity set:

[0030] ;

[0031] in, Represents the set to be calculated entity vectors in Indicates all In the entity vector, d represents the Manhattan distance;

[0032] Secondly, all The internal consistency score of a mode is obtained by averaging the minimum Manhattan distances corresponding to the entities in the middle. : ;

[0033] By analogy, the internal consistency scores of the remaining five modes are obtained: ;

[0034] S203, Calculate External Consistency :

[0035] The principle for calculating external consistency is similar to that for calculating internal consistency, but this calculation is performed between groups: The six modalities are still divided into two groups (referred to as the internal group and the external group): three internal modalities of the event (Event_text, Event_image, Event_ocr) and three external knowledge modalities (External_text, External_image, External_ocr). However, the division of the set to be calculated, A, and the comparison set, B, is different.

[0036] If we want to calculate the external consistency of a certain modality within an internal group, then we divide the set to be calculated, A, into a comparison set, B: , ;

[0037] If we want to calculate the external consistency of a certain modality in the external group, then we need to divide the set to be calculated, A, into a comparison set, B: , ;

[0038] Next, calculate the external consistency of A and B. The process is the same as the calculation of IC in S202;

[0039] S204. Calculate the total mass fraction:

[0040] First, the three indicators R, IC, and EC are mapped to a uniform order of magnitude using the min-max standardization method; second, the standardized values ​​are... The original mass fraction of mode m was calculated. : ;in, ;

[0041] Secondly, Mapped to the range [0,1], using a modified softmax normalization. Normalization yields the final quality score : ;

[0042] in, This indicates an adjustable parameter. The larger the value, the sharper the softmax, and the greater the difference in quality scores between different modes. It can strengthen the optimal terms and weaken the low-quality modes. The smaller the value, the smoother the softmax, which can increase the model's exploration opportunities and weaken the differences;

[0043] Then, for each event obtained in the first step, a scene graph with six modalities is generated, and the quality scores of the six modalities are obtained. ;

[0044] Finally, the SGAT model was used to extract features from the scene images to obtain the features of each scene image: ;

[0045] Where i represents the i-th event.

[0046] Preferably, S300 includes: a Core Modality Refinement Unit (CFRU) and a Quality Weighted Secondary Modality Fusion Unit (QWMFU); the specific process is as follows: acquiring scene graph features and features after splicing the original features; after aligning the six input modalities, using the event text and the event OCR text extracted from the event image as the core modality, and the others as secondary modalities; through multi-layer fusion of K layers of Core Modality Refinement Units (CFRU) and K+1 layers of Quality Weighted Secondary Modality Fusion Units (QWMFU), obtaining the jointly enhanced modality of the six modalities. Ultimately, the fused features of the six modalities were obtained. Specifically, it includes:

[0047] S301. Select the core modality and use the corresponding quality scores for both the event text and the OCR text extracted from the event image. , The remaining modes are treated as independent secondary modes and weighted fusion.

[0048] S302. In the Core Modality Refinement Unit (CFRU), the CFRU encodes the core modality features layer by layer through K layers of Transformer. Each layer can capture feature information at different levels, thereby obtaining richer feature representations and integrating the core modalities selected in the previous step. As input to CFRU Next, inter-layer updates of the core modalities are performed:

[0049] ;

[0050] in, This represents the update result of the k-th layer. This represents the update result of the previous layer; here, k is synchronized with k in QWMFU; after performing K-layer calculations, we obtain... ;

[0051] S303. In the Quality-Weighted Secondary Modal Fusion Unit (QWMFU), all secondary modes except the core mode are used as inputs to the QWMFU, defined as follows: ,in, The quality-weighted secondary modal fusion unit has a total of K+1 layers;

[0052] For all secondary modalities, each modality corresponds to a cross-attention guided by the primary modality; where, for the current layer k, the features of the primary modality from the previous layer are used. As a query, the secondary modality The cross-attention function, using both Key and Value inputs, is formulated as follows:

[0053] ;

[0054] Secondly, calculate the dynamic interlayer quality fraction. Assuming the current layer is the k-th layer, the input to the dynamic quality score calculation module is the core mode of that layer. and all secondary modes of this layer ,in The output is the dynamic interlayer quality score of the k-th layer. ;

[0055] Specifically, the core and secondary modes of the input in this layer are first pooled to obtain the overall representation:

[0056] ;

[0057] ;

[0058] Secondly, a quality weight logit transformation is performed to map the quality weights to a space more suitable for MLP:

[0059] ;

[0060] in, Represent a minimum value To prevent numerical instability, the high-quality mode is calculated using this formula. The closer it is to 1, the more its The further the value is than 0. Conversely, low-quality modes... Much less than 0.

[0061] Next, compute the input to the Gate MLP:

[0062] ;

[0063] in, This represents the input to the Gate network. This indicates splicing based on the last dimension;

[0064] Secondly, confirm the Gate network, represented as: ;in, Indicates the number of hidden units;

[0065] Specifically: First, a first layer of linear and ReLU operations is performed to increase nonlinearity, enabling the network to capture complex relationships. ;

[0066] Perform Dropout to prevent overfitting: ;

[0067] Perform a second layer of linear and Sigmoid: ;in, This indicates the Gate weights of the modal corresponding to the final output;

[0068] The dynamic quality score of each secondary mode is obtained. And the enhancement features of each secondary modality Then, all secondary modalities are weighted and fused into a weighted representation of the secondary modalities. :

[0069] ;

[0070] Secondly, the secondary modal features of the j-th layer are fused. With the QWMFU output of the previous layer Merge and update to obtain the input to the next layer. :

[0071] in It is a two-layer feedforward network. It is a learnable parameter;

[0072] After performing k+1 level calculations, the last level is obtained. ;

[0073] S304, the previously obtained and The final output of this module is obtained by fusion. : ;

[0074] and the final output Feed it into the classifier.

[0075] Preferably, S400 includes:

[0076] After performing feature fusion and obtaining the final fused features Then, a standard multilayer perceptron is used for the final rumor classification; this classifier is based on the results obtained in the previous step. As input, it is mapped to three categories: rumor / non-rumor / uncertain.

[0077] Meanwhile, to demonstrate the interpretability of this method, a simple interpretation module based on quality grading calculation is designed in the final output. The module takes the quality score, information richness R, internal consistency score IC, and external consistency score EC and fills them into a template to provide some interpretation of the model's classification results.

[0078] Compared with the prior art, the beneficial effects achieved by the present invention are:

[0079] This invention amplifies the role of core modalities to some extent through progressive fusion guided by core modalities, enabling the model to place greater emphasis on important modalities in multimodal rumor detection tasks. Simultaneously, by using quality scores, the model focuses more on high-quality modalities through prior quality scores and subsequent dynamic quality score calculations, without losing low-quality modalities. Finally, by constructing and analyzing scene graphs, the model can focus on the logical relationships between events, and the method of calling a large model to generate scene graphs and adding external knowledge enhances the model's ability to extract logical features. Attached Figure Description

[0080] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0081] Figure 1 This is a flowchart of a quality-guided progressive fusion multimodal rumor detection method according to the present invention;

[0082] Figure 2 This is a comparison chart of the rumor detection performance of this invention with three other models on the MR2 dataset. Detailed Implementation

[0083] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0084] Please see Figure 1-2 The present invention provides the following technical solution:

[0085] Example 1: A quality-guided progressive fusion multimodal rumor detection method of the present invention includes: first, extracting features for each modality; then, transforming each modality into a scene graph based on LLM and performing knowledge augmentation; next, calculating the quality score of each modality based on the structure and entity semantics of the scene graph, and using SGAT to extract scene graph features for each modality; based on this, fusing the scene graph features obtained in the first two steps with the original features, and inputting them along with the quality score into the core modality-guided progressive fusion process, while simultaneously calculating the dynamic quality score between each progressive fusion layer. Finally, fusing the final update result of the core modality with the final result of the secondary modality to obtain the final fused feature of the event; finally, inputting the final fused feature obtained in the previous step into the classifier. Simultaneously, an interpretation module is used to interpret the predictions made by the classifier based on some intermediate results in the quality score calculation. Specific implementation details are as follows:

[0086] Step 1: Feature Extraction. Given four parts of the original dataset (event text, event image, external knowledge text, and external knowledge image), feature extraction is performed using BERT+LSTM for the text modality. For the image modality, OCR based Paddleocr is used for feature extraction, followed by image feature extraction based on Swin Transformer. The extracted OCR text is then used for feature extraction using RoBERT. The final output is the features for all six modalities. , where i represents event i.

[0087] Step 2, Scene Graph Generation and Analysis: First, use BLIP2-FlanT5-xl to convert all image modalities into text descriptions, processing all six modalities into text. Then, call the API to extract entities and relationships from the text, and perform external knowledge linking to obtain a knowledge-enhanced scene graph. This yields a scene graph for each event with six modalities. , where i represents the i-th event.

[0088] Next, the quality score is calculated using a scene graph, taking into account three aspects: internal information richness, consistency with the modality of the event itself, and consistency with external knowledge.

[0089] Preferably, for information-rich R: First, calculate the number of unique nodes N in the scene graph for each modality. The more unique nodes, the richer the information. Next, calculate the relation diversity R_{types} of the scene graph, which is the number of unique relation types. Higher relation diversity indicates a more detailed representation of relations in that modality. Then, calculate the graph density E, which is the ratio of the number of relations to the number of nodes. Higher graph density indicates a more compact scene graph structure and higher modal quality. This yields the number of unique nodes for each modality. Relationship diversity , graph density Finally, the overall information richness is calculated: first, all indicators are standardized to between 0 and 1, and then the average of the three indicators is taken as the internal information richness.

[0090] Internal Consistency IC: The six modalities are divided into two groups: three internal modalities I (Event_text, Event_image, Event_ocr) and three external knowledge modalities E (External_text, External_image, External_ocr). This module's internal consistency IC calculation is used to evaluate the mutual support of each modality within the group. First, for each modality in each of the three modalities in group I and group E, all entities in its scene graph are extracted and formed into a set. , Next, the obtained entity sets of all modalities are divided into a set A to be calculated and a comparison set B. Then, the minimum distance from each entity in A to all entities in B is calculated. Finally, the minimum distances corresponding to all entities in A are averaged to obtain the IC of the modality corresponding to set A.

[0091] Taking the calculation of the internal consistency of Event_text as an example: As the set to be computed, A, Merge them into a comparison set B. For all n entities in set A:

[0092] ;

[0093] Calculate the comparison set for each entity in A. The Manhattan distances of all entities in A are calculated, and the minimum value is found. This minimum value represents the distance of the entity to its "most similar partner" in another modality. The internal consistency score of the current modality (Event_text) is obtained by averaging the minimum values ​​for all entities in A. Thus, each entity in A corresponds to a distance, and the average of these distances is calculated. The larger this value, the farther the entities in the modality being evaluated are from other modalities, and the worse the internal consistency. The specific formula is as follows:

[0094] First, all entities are converted into vectors, and then the set to be computed and the set to be compared are divided, where v and u represent the entity vectors formed by the BERT encoding of the entities:

[0095] ;

[0096] ;

[0097] Next calculation entity vectors in Minimum Manhattan distance of the entity set:

[0098] ;

[0099] in, The set to be computed is A, and u refers to... All entity vectors in the vector, where d refers to the Manhattan distance.

[0100] Next, all The internal consistency score of the modality (Event_text) is obtained by averaging the minimum Manhattan distances corresponding to the entities in the middle. : ;

[0101] By analogy, the internal consistency scores of the remaining five modes can be obtained: ;

[0102] External Consistency (EC): The principle for calculating external consistency is similar to that for calculating internal consistency, but this calculation is performed between groups. Here, the six modalities are still divided into two groups (referred to as the internal group and the external group): three internal event modalities (Event_text, Event_image, Event_ocr) and three external knowledge modalities (External_text, External_image, External_ocr). However, the division of the set to be calculated (A) and the comparison set (B) is different.

[0103] Specifically, if we are calculating the external consistency of a mode within an internal group, let set A be the selected mode, and set B be another external group within the selected mode's group. If we are calculating the external consistency of a mode within an external group, let set A be the selected mode, and set B be another internal group within the selected mode's group. Similarly, in this part, a larger EC value indicates worse external consistency.

[0104] After obtaining the above three values ​​for each mode, the total mass fraction of each mode is calculated:

[0105] (1) Map the three indicators R, IC, and EC to a unified order of magnitude:

[0106] The R, IC, and EC metrics are used to model and quantify certain properties, which inevitably presents challenges in data alignment. Here, we use a more comprehensive min-max normalization method to map all metrics to a uniform range [0,1], and then use clip(y,0,1) to prune them, preventing extreme values ​​from having an excessive impact.

[0107] ;

[0108] Where x represents the three indicators R, IC, and EC. Represents the mapped , , , This represents the 1st percentile of each of the three indicators. This represents the 99th percentile of each of the three indicators. This percentile is calculated based on the percentiles of all R, IC, and EC values ​​obtained from all samples in the training set using the above method.

[0109] (2) Standardize The summation yields the original mass fraction of mode m. ( Next, for each event i, calculate the raw quality fraction for each of its six modes. : .

[0110] (3) Mapping to the range [0,1] yields the final quality score. Here, a modified softmax normalization method is used. Normalization yields : .

[0111] in, It is an adjustable parameter. The larger the value, the sharper the softmax, and the greater the difference in quality scores between different modes. It can strengthen the optimal terms and weaken the low-quality modes. The smaller the value, the smoother the softmax, which can increase the model's exploration opportunities and weaken the differences.

[0112] At this point, each event obtained in the first step yields a scene graph with six modalities, resulting in a quality score for each modality. }

[0113] Step 3: In the progressive fusion guided by the core modality combined with dynamic quality scores, it is divided into two parts: the Core Modality Refinement Unit (CFRU) and the Quality Weighted Secondary Modality Fusion Unit (QWMFU). The process is roughly as follows: the original features of each modality and the scene graph features of each modality are concatenated and used as the input for this part. After aligning the six input modalities, the modality with the highest quality weight is selected as the core modality, and the others are selected as secondary modalities. Through multi-layer fusion of K layers of CFRU and K+1 layers of QWMFU, the jointly enhanced modality of the six modalities is obtained. Ultimately, the fused features of the six modalities were obtained. .include:

[0114] (1) First, perform modality alignment: for input features (The original features obtained by the feature extraction module and the scene graph features obtained by the scene graph generation and processing module are combined and fused together, where...) The original time step of the input feature sequence. The input features (original dimension) are projected and positionally encoded. Then, the numerical distribution is stabilized through layer normalization. Attention weights between the input features and a set of learnable query vectors are calculated, and alignment weights are generated after numerical stabilization. Finally, based on these weights, the original sequence is weighted and fused into a feature sequence of the target length, and normalized again. The final output is then given. Specifically:

[0115] Using a predefined linear layer, the concatenated features are projected onto a uniform dimension. :

[0116] ;

[0117] Next, a normalization layer is performed: .

[0118] (2) Next, the core modality is selected. In this method, the event text and the OCR text extracted from the event image are selected using their corresponding quality scores. , The weighted fusion is used as the core mode Z, and the remaining modes are used as independent secondary modes.

[0119] (3) Next, progressive fusion guided by core modality is performed. The features used are the aligned features obtained in the first step of modality alignment.

[0120] In the Core Modality Refinement Unit (CFRU), the CFRU encodes the core modality features layer by layer through K layers of Transformer. Each layer captures feature information at different levels, thereby obtaining richer and more discriminative feature representations, which then transform the core modalities selected in the previous step. As input to CFRU Next, perform inter-layer updates of the core modalities:

[0121] .

[0122] in, This represents the update result of the k-th layer. This represents the update result of the previous layer; here, k is synchronized with k in QWMFU; after performing K-layer calculations, we obtain... .

[0123] (3) In the quality-weighted secondary modal fusion unit, all secondary modes except the core mode are used as inputs to the QWMFU, which is named here. , The quality-weighted secondary modality fusion unit consists of K+1 layers. For all secondary modalities, each modality corresponds to a cross-attention mechanism guided by the primary modality. Specifically, for the current layer k, the features of the primary modality from the previous layer are used. As a query, the secondary modality The cross-attention function, using both Key and Value inputs, is formulated as follows:

[0124] .

[0125] Next, the dynamic interlaminar quality fraction is calculated. Assuming the current layer is the k-th layer, the input to the dynamic quality score calculation module is the core mode of that layer. All secondary modes of this layer , in The output is the dynamic interlayer quality score of the k-th layer. .

[0126] Specifically, the core and secondary modes of the input in this layer are first pooled to obtain the overall representation:

[0127] ;

[0128] ;

[0129] Next, we perform a quality weight logit transformation, mapping the quality weights to a space more suitable for MLP:

[0130] ;

[0131] in, It is a minimum value To prevent numerical instability, this formula is used to calculate the high-quality mode. The closer it is to 1, the more its The further the value is than 0. Conversely, low-quality modes... Much less than 0.

[0132] Next, we compute the input to the Gate MLP:

[0133] ;

[0134] in, This represents the input to the Gate network. This indicates splicing based on the last dimension;

[0135] Next is the Gate network, which can be roughly represented as follows: ;

[0136] in, Indicates the number of hidden units;

[0137] Specifically: First, perform a first layer of linear and ReLU operations to increase nonlinearity, enabling the network to capture complex relationships. ;

[0138] Next, we perform the second layer of linear and Sigmoid: ;

[0139] in, This refers to the Gate weights of the modal corresponding to the final output.

[0140] The dynamic quality score of each secondary mode is obtained. And the enhancement features of each secondary modality Then, all secondary modalities are weighted and fused into a weighted representation of the secondary modalities. :

[0141] ;

[0142] Next, we will fuse the secondary modal features of the j-th layer. With the QWMFU output of the previous layer Merge and update to obtain the input to the next layer. :

[0143] ;

[0144] in It is a two-layer feedforward network. It is a learnable parameter.

[0145] After performing k+1 level calculations, the last level is obtained. .

[0146] (4) Finally, the results obtained from the above steps and The final output of this module is obtained by fusion. The data is then fed into a classifier using the following formula:

[0147] .

[0148] This completes the final output of the module. .

[0149] Step 4: Classification and Explanation:

[0150] After performing feature fusion and obtaining the final fused features Then, a standard multilayer perceptron is used for the final rumor classification. This classifier is based on the results obtained in the previous step. As input, it is mapped to three categories: rumor / non-rumor / uncertain. The loss function used for training includes cross-entropy loss and L2 regularization:

[0151] ;

[0152] ;

[0153] ;

[0154] Meanwhile, to demonstrate the interpretability of this method, a simple interpretation module based on quality grading calculation is designed in the final output. This module takes the quality score, information richness R, internal consistency score IC, and external consistency score EC and fills them into a template to provide some interpretation of the model's classification results. For example: "Event 1969 is identified as a non-rumor real event. The label is: non-rumor. The main basis is: ['event_text', 'event_ocr_text', 'external_ocr_text'] provides high-quality information. The auxiliary basis is: information richness: 0.22, internal consistency score: 0.48, external consistency score: 0.35."

[0155] Meanwhile, the model employs the AdamW optimizer and cosine annealing learning rate scheduling. To prevent overfitting, a regularization term is added to the loss function, and Dropout and early stopping are used during training.

[0156] like Figure 2 As shown, in order to more comprehensively demonstrate the effect of the present invention in rumor detection, the present invention was compared with three types of models on the MR2 dataset;

[0157] The comparison results show that our model achieves good performance on most metrics. In the rumor category, our model outperforms the best comparison model by 4.2% on the Pre metric, 0.8% on the Rec metric, and 2.5% on the F1 score. In the unverified category, our model outperforms the best comparison model by 1% on the Pre metric, 2.1% on the Rec metric, and 1.6% on the F1 score. Overall, our model demonstrates good performance. These results can be attributed to three factors. First, the progressive fusion guided by core modalities amplifies the role of core modalities to a certain extent, making the model pay more attention to important modalities in the multimodal rumor detection task. Second, the use of quality scores allows the model to focus more on high-quality modalities through prior quality scores and subsequent dynamic quality score calculations, without losing low-quality modalities. Finally, the construction and analysis of scene graphs allow the model to focus on the logical relationships between events, while the method of calling large models to generate scene graphs and adding external knowledge also enhances the model's ability to extract logical features.

[0158] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A quality-guided progressive fusion multimodal rumor detection method, characterized in that: The method includes: S100. Obtain the four modalities of the original dataset: event text, event image, external knowledge text, and external knowledge image; extract the text from the event image and the external knowledge image respectively and treat them as two independent modalities, then extract the features of the six modalities respectively; S200, an LLM-based scene graph generation method, performs external knowledge linking when calling the large model to obtain an information-enhanced scene graph; calculates quality scores based on scene graph structural features and semantic features between scene graph entities of different modalities, and uses SGAT to encode the generated scene graph to obtain scene graph features, thus obtaining scene graph features and quality scores for each modality. S300. The scene graph features of each modality are concatenated with the original features, and then the core modality is selected. Through progressive fusion guided by multi-layer core modalities, and combined with the dynamic quality score mechanism, the modal features of all events are fused to obtain the final feature representation. S400: The enhanced features are mapped to the category space, and finally the output is probabilistically converted using softmax; at the same time, a simple interpretable module is designed.

2. The quality-guided progressive fusion multimodal rumor detection method as described in claim 1, characterized in that, In S100, the PaddleOCR model is used to extract text from event images and external knowledge images, and BERT+LSTM is used to extract features from all text modalities. SwinTransformer is used to extract features from all image modalities, resulting in six modal features for an event.

3. The quality-guided progressive fusion multimodal rumor detection method as described in claim 1, characterized in that, The S200 includes: Using BLIP2-FlanT5-xl, all six modalities of the event are processed into text, and the API is called to extract entities, relationships between entities, and supplementary descriptions of entities from the text to obtain a knowledge-enhanced scene graph. Based on the scene graph, the internal information richness, internal modality consistency, and external knowledge consistency are calculated respectively, and the quality score is further calculated to obtain the scene graph features and quality score of each modality.

4. The quality-guided progressive fusion multimodal rumor detection method as described in claim 3, characterized in that, The process involves calculating internal information richness, internal modality consistency, and external knowledge consistency based on the scene graph, and further calculating quality scores to obtain the scene graph features and quality scores for each modality, including: S201, Calculate the richness of internal information : Get the number of unique nodes in the scene graph for each modality. The relationship between scene diagrams and diversity Calculate graph density This indicates that the number of relationships is greater than the number of nodes; therefore, it is based on the number of unique nodes. Relationship diversity And graph density Obtain the richness of internal information : ； in, , and This indicates the number of unique nodes. Relationship diversity And graph density All metrics are standardized to between 0 and 1; S202, Calculate internal consistency : The six modalities were divided into two groups: three modalities within the event and three modalities within the external knowledge, to evaluate the degree of mutual support between each modality within the group; First, for each modality within each of the three modalities, extract all entities from its scene graph and form a set: Divide all entities of all modalities into a set to be computed, A, and a set to be compared, then... As the set to be computed, the remaining modes within the group Merge as comparison set B; For all n entities in set A: Calculate the comparison set for each entity in A. Find the minimum Manhattan distance for all entities in A, where the minimum represents the distance of the entity to its "most similar partner" in another modality; then average the minimum values ​​for all entities in A to obtain the internal consistency of the current modality. ; Specifically, calculation The specific formula is as follows: First, convert all entities into vectors and divide them into a set to be computed and a set to be compared: ； ； in, , This represents the entity vector formed by the entity after being encoded by BERT; Secondly, calculation entity vectors in Minimum Manhattan distance of the entity set: ； in, Represents the set to be calculated entity vectors in Indicates all In the entity vector, d represents the Manhattan distance; Secondly, all The internal consistency score of a mode is obtained by averaging the minimum Manhattan distances corresponding to the entities in the middle. : ; By analogy, the internal consistency scores of the remaining five modes are obtained: ; S203, Calculate External Consistency : If we want to calculate the external consistency of a certain modality within an internal group, then we divide the set to be calculated, A, into a comparison set, B: , ; If we want to calculate the external consistency of a certain modality in the external group, then we need to divide the set to be calculated, A, into a comparison set, B: , ; Next, calculate the external consistency of A and B. The process is the same as the calculation of IC in S202; S204. Calculate the total mass fraction: First, the three indicators R, IC, and EC are mapped to a uniform order of magnitude using the min-max standardization method; second, the standardized values ​​are... The original mass fraction of mode m was calculated. : ;in, ; Secondly, Mapping to the range [0,1] yields the final quality score. : ; in, Indicates an adjustable parameter; Then, for each event obtained in the first step, a scene graph with six modalities is generated, and the quality scores of the six modalities are obtained. ; Finally, the SGAT model was used to extract features from the scene images to obtain the features of each scene image: ; Where i represents the i-th event.

5. The quality-guided progressive fusion multimodal rumor detection method as described in claim 1, characterized in that, S300 includes: acquiring scene graph features and features obtained by splicing the original features; aligning the six input modalities, using the event text and the event OCR text extracted from the event image as the core modalities, and the others as secondary modalities; and obtaining the jointly enhanced modalities of the six modalities through multi-layer fusion of a K-layer core modality refinement unit CFRU and a K+1-layer quality-weighted secondary modality fusion unit QWMFU. Ultimately, the fused features of the six modalities were obtained. Specifically, it includes: S301. Select the core modality and use the corresponding quality scores for both the event text and the OCR text extracted from the event image. , The remaining modes are treated as independent secondary modes and weighted fusion. S302. In the Core Modality Refinement Unit (CFRU), the CFRU encodes the core modality features layer by layer through K layers of Transformer, obtaining richer feature representations, and then uses the core modalities selected in the previous step... As input to CFRU Next, inter-layer updates of the core modalities are performed: ； in, This represents the update result of the k-th layer. This represents the update result of the previous layer; here, k is synchronized with k in QWMFU; after performing K-layer calculations, we obtain... ; S303. In the Quality-Weighted Secondary Modal Fusion Unit (QWMFU), all secondary modes except the core mode are used as inputs to the QWMFU, defined as follows: ,in, The quality-weighted secondary modal fusion unit has a total of K+1 layers; For all secondary modalities, each modality corresponds to a cross-attention guided by the primary modality; where, for the current layer k, the features of the primary modality from the previous layer are used. As a query, the secondary modality The cross-attention function, using both Key and Value inputs, is formulated as follows: ； Secondly, calculate the dynamic interlayer quality fraction. Assuming the current layer is the k-th layer, the input to the dynamic quality score calculation module is the core mode of that layer. and all secondary modes of this layer ,in The output is the dynamic interlayer quality score of the k-th layer. ; Specifically, the core and secondary modes of the input in this layer are first pooled to obtain the overall representation: ； ； Secondly, a quality weight logit transformation is performed to map the quality weights to a space more suitable for MLP: ； in, Represent a minimum value ; Next, compute the input to the Gate MLP: ； in, This represents the input to the Gate network. This indicates splicing based on the last dimension; Secondly, confirm the Gate network, represented as: ;in, Indicates the number of hidden units; Specifically: First, a first layer of linear and ReLU operations is performed to increase nonlinearity, enabling the network to capture complex relationships. ; Perform Dropout to prevent overfitting: ; Perform a second layer of linear and Sigmoid: ;in, This indicates the Gate weights of the modal corresponding to the final output; The dynamic quality score of each secondary mode is obtained. And the enhancement features of each secondary modality Then, all secondary modalities are weighted and fused into a weighted representation of the secondary modalities. : ； Secondly, the secondary modal features of the j-th layer are fused. With the QWMFU output of the previous layer Merge and update to obtain the input to the next layer. : in It is a two-layer feedforward network. It is a learnable parameter; After performing k+1 level calculations, the last level is obtained. ; S304, the previously obtained and The final output of this module is obtained by fusion. : ; and the final output Feed it into the classifier.

6. The quality-guided progressive fusion multimodal rumor detection method as described in claim 1, characterized in that, The S400 includes: After performing feature fusion and obtaining the final fused features Then, a standard multilayer perceptron is used for the final rumor classification; this classifier is based on the results obtained in the previous step. As input, it is mapped to three categories: rumor / non-rumor / uncertain; Meanwhile, a simple interpretation module based on quality grading calculation is designed for the final output; the module takes the quality score, information richness R, internal consistency score IC, and external consistency score EC and fills a template to provide some interpretation of the model's classification results.

Images