A multimodal network rumor detection method based on large language model feature enhancement

Abstract

The present invention provides a multimodal network rumor detection method based on large language model feature enhancement, which belongs to the field of language processing technology; it includes acquiring text modal data and image modal data; performing enhancement processing on the text modal data and the image modal data; combining the enhanced text with a predefined prompt template to obtain input prompt enhanced text; converting the image modal data into a visual feature representation; obtaining an emotion-enhanced text feature representation, a fused visual feature representation and a fused text feature representation based on the enhanced text; optimizing the fused visual feature representation and the text feature representation to generate a visual integrated feature vector and a text integrated feature vector; weighting the visual integrated feature vector and the text integrated feature vector to generate a visual cross-modal feature and a text cross-modal feature; inputting the visual cross-modal feature and the text cross-modal feature into a fully connected layer together with the text information and the visual information balanced by an adjustable weight coefficient, and outputting a prediction result.

Description

Technical Field

[0001] The present invention belongs to the technical field of language processing and relates to a multimodal network rumor detection method based on large language model feature enhancement. Background Art

[0002] With the rise of large language models and cued learning techniques, their application in multimodal network rumor detection has become a hot research frontier. Early network rumor detection methods mainly relied on manually formulated rules, dictionary matching or simple machine learning models (such as naive Bayes and support vector machines). These methods are difficult to effectively capture the complex characteristics of rumors when faced with massive, diverse and dynamically changing content in social networks. In recent years, the development of deep learning technology has enabled neural network-based models to better extract complex semantics and feature expressions from data, and to a certain extent improved the performance of rumor detection.

[0003] However, relying solely on a single modality of information (such as text or images) for rumor detection has inherent limitations. It is easy to be difficult to improve detection accuracy due to information inconsistencies and information islands between modalities, as well as the lack of effective cross-modal interaction mechanisms. Therefore, multimodal information fusion becomes the key to improving rumor detection results.

[0004] Although the current multimodal rumor detection methods have made some progress, they still have many shortcomings. On the one hand, traditional data enhancement methods such as back translation and random insertion often fail to capture the subtle semantic changes and deep semantic structures of rumor texts. The generated data variants are obviously insufficient in semantic richness and diversity, and it is difficult to fully cover the complex forms of rumor expression in the real world. On the other hand, most of the existing multimodal feature fusion methods use simple splicing or weighted fusion, which fails to deeply explore the intrinsic correlation and complementary information between the features of each modality, resulting in low feature fusion efficiency. In addition, although sentiment analysis plays an important role in rumor detection, existing methods often fail to effectively integrate sentiment information with other modal information, resulting in the weakening of the value of sentiment features and failure to give full play to their auxiliary role in rumor detection.

[0005] In summary, existing technologies have obvious technical bottlenecks in dealing with the problem of multimodal network rumor detection, and it is difficult to meet the growing needs of social network information security. Therefore, it is urgent to develop a new method that can make full use of large language models and prompt learning technology to generate semantically rich and diverse text variants, and more effectively integrate text, visual and emotional features through optimized cross-modal fusion mechanisms. This method will generate diverse semantically enhanced texts by introducing a large language model, integrate text, image and emotional features by using an optimized cross-modal fusion mechanism, and combine noise robustness strategies and model parameter optimization to achieve higher detection accuracy and robustness, thereby providing a more accurate and efficient solution for social network rumor detection. Summary of the invention

[0006] The present invention proposes a multimodal social network rumor detection method based on large language model data enhancement and prompt learning, aiming to improve the accuracy of network rumor detection by enhancing the diversity and richness of text data.

[0007] The present invention provides a multimodal network rumor detection method based on large language model feature enhancement, comprising the following steps:

[0008] S1. Obtain the text modality data of news information in the dataset to be tested and image modality data ;

[0009] S2. Text modal data and image modality data Perform enhancement processing to obtain the final enhanced text ;

[0010] S3. In enhanced text Based on the Combine with predefined prompt templates to create input prompts to enhance text ;

[0011] S4. Use the visual encoder of the CLIP model to convert the image modality data in each article in the dataset to be detected Convert to visual feature representation ;

[0012] And, based on enhanced text Get text feature representation and sentiment-enhanced text feature representation ;

[0013] S5. Fusion of visual feature representation via self-attention and multi-head attention mechanisms and sentiment-enhanced text feature representation , and obtain the fused visual feature representation And the fused text feature representation ;

[0014] Optimizing the fused visual feature representation using residual connections And the fused text feature representation , generate visual integrated feature vector and text integrated feature vector ;

[0015] Combined cosine similarity to visual integrated feature vector and text integrated feature vector Weighted to generate the final visual cross-modal features and text cross-modal features ;

[0016] Visual cross-modal features , text cross-modal features The text information and visual information balanced with adjustable weight coefficients are input into the fully connected layer, and the prediction result is finally output.

[0017] Furthermore, the specific method of using the pre-trained large language model to perform data enhancement processing on the data in the test data set is:

[0018] Write a script to send a request to the Big Language Model API in a non-streaming call mode, and use prompt words to guide the model to generate text variants and image description text that meet specific needs;

[0019] The prompt words are divided into first-category prompt words, second-category prompt words and third-category prompt words.

[0020] in:

[0021] The first type of prompt words guide the large language model to analyze text modal data Perform data enhancement to obtain enhanced text generated by the first type of prompt words ;

[0022] The second type of prompt words encourages the model to redescribe the text modality data. Perform text diversification to obtain enhanced text generated by the second type of prompt words ;

[0023] The third type of prompt words guide the large language model to image modality data Describe in detail and get the enhanced text generated by the third type of prompt ;

[0024] The enhanced text generated by the first type of prompt words , enhanced text generated by the second type of prompt words and enhanced text generated by the third type of prompts Merge to get a semantically rich and diverse enhanced text .

[0025] Furthermore, the specific process of creating the prompt enhanced text is as follows:

[0026] S3.1, prompt-enhanced text generation;

[0027] Enhanced text after large language model enhancement , integrated with predefined prompt templates to form prompt enhanced text ;

[0028] S3.2, model processing and probability distribution output;

[0029] Use RoBERTa-base model to enhance text Processing is performed to obtain the probability distribution of possible fill values ​​at the mask mark position .

[0030] Furthermore, we obtain the visual feature representation , text feature representation and sentiment-enhanced text feature representation The specific process is as follows:

[0031] Vision encoder via CLIP model The image modality data in each article Converted into visual feature representation ;

[0032] Text encoder via CLIP model Will enhance the text Convert to text feature representation ;

[0033] Apply Sentiment Analysis Tool (SAT) to extract enhanced text The emotional polarity and subjectivity in the text are normalized to the interval [0,1] to obtain the normalized emotional element; the normalized emotional element is then multiplied with the text feature representation to obtain the emotionally enhanced text feature representation. .

[0034] Furthermore, the specific process of converting the image modality data in each article in the dataset to be detected into visual feature representation is as follows:

[0035] S4.1, visual feature representation and text feature extraction;

[0036] Vision encoder via CLIP model The image modality data in each article Converted into visual feature representation ;

[0037] Through the text encoder in the CLIP model Will enhance the text Convert to text feature representation ;

[0038] S4.2, obtain sentiment-enhanced text feature representation;

[0039] ①. Enhance the text Extract the sentiment polarity of the text through sentiment analysis tools and subjectivity ;

[0040] ②, Emotional polarity Normalize it so that its value range is scaled to the interval [0,1];

[0041] ③. Combined with normalized sentiment polarity and subjectivity , calculate the sentiment element ;

[0042] ④、Normalized emotional elements With text feature representation Multiply them together to get the sentiment-enhanced text features .

[0043] Furthermore, the fused visual feature representation is obtained And the fused text feature representation The specific process is as follows:

[0044] Using the self-attention mechanism, the query vector , key vector Sum value vector Combine to calculate the attention weight of each head ;

[0045] Use multi-head attention mechanism to represent visual features and sentiment-enhanced text feature representation Interact to generate fused visual feature representation And the fused text feature representation ; Among them, the output of the multi-head attention mechanism is obtained by concatenating the outputs of multiple attention heads.

[0046] Further, generate visual integrated feature vector and text integrated feature vector The specific process is as follows:

[0047] Using the residual connection strategy, the fused visual features are represented And the fused text feature representation Added to the visual feature representation and sentiment-enhanced text feature representation To obtain visual integration features and text integration features ;

[0048] Integrate visual features and text integration features Input into the fusion framework to generate a visual integrated feature vector and text integrated feature vector ; Among them, the fusion framework mixes linear transformation, batch normalization, Activation and Dropout layers.

[0049] Furthermore, the final visual cross-modal features are generated and text cross-modal features The specific process is as follows:

[0050] Using cosine similarity to measure visual feature representation and text feature representation Similarity ;

[0051] Then the cosine similarity is passed through Activate and generate weights; by using weights to integrate the visual feature vectors and text integrated feature vector Weighted to generate the final visual cross-modal features and text cross-modal features .

[0052] Furthermore, the visual cross-modal features and text cross-modal features The specific process of inputting the fully connected layer together with the text and visual information balanced by the adjustable weight coefficient and finally outputting the prediction result is as follows:

[0053] The probability distribution is transformed through residual connection Respectively with visual cross-modal features , text cross-modal features Combined with, further optimize the prompt selection in the continuous embedding space; at the same time, an adjustable weight coefficient is introduced , to balance the importance of text and visual information in the final decision process, and obtain the composite feature vector ;in, ;

[0054] The composite feature vector Input to the fully connected layer for further processing to generate classification probability distribution , and finally determined that <mask>” Mark the most appropriate filler word;

[0055] The cross entropy loss function is used to optimize the parameters of the large language model so that the large language model can more accurately predict the correct filler words given the input, thereby indirectly achieving the result output of rumor detection classification, that is, the final output prediction result.

[0056] Compared with the prior art, the present invention has the following beneficial effects:

[0057] The present invention provides a multimodal network rumor detection method based on large language model feature enhancement, which fully utilizes the multimodal feature extraction capability of the CLIP model by combining the large language model with the prompt learning technology, and realizes the efficient fusion of cross-modal information through sentiment analysis and attention mechanism, thereby providing an efficient and accurate solution for social network rumor detection, and has good practicality and promotion value.

[0058] In addition to the above-described purposes, features and advantages, the present invention has other purposes, features and advantages. The present invention will be further described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The drawings constituting a part of this application are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0060] Figure 1 It is a flow chart of a multimodal network rumor detection method based on large language model feature enhancement in an embodiment of the present invention. DETAILED DESCRIPTION

[0061] In order to make the above-mentioned purposes, features and advantages of the present invention more clear and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings. It should be noted that the drawings of the present invention are all simplified and use non-precise proportions, which are only used to conveniently and clearly assist in explaining the implementation of the present invention; the "number" mentioned in the present invention is not limited to the specific number in the examples in the accompanying drawings; the directions or positional relationships indicated by "front", "middle", "back", "left", "right", "up", "down", "top", "bottom", "middle", etc. mentioned in the present invention are based on the directions or positional relationships shown in the drawings of the present invention, and do not indicate or imply that the devices or components referred to must have a specific direction, nor can they be understood as limitations on the present invention. Example

[0062] See also Figure 1 As shown, the present invention provides a multimodal network rumor detection method based on large language model feature enhancement, comprising the following steps:

[0063] S1. Obtain the text modality data of news information in the dataset to be tested and image modality data .

[0064] Text modal data The expression is as follows:

[0065] ;

[0066] in, is the number of words in each article, ;

[0067] Image modality data The expression is as follows:

[0068] ;

[0069] in, is the total number of images for each text, ;

[0070] Preferably, in this embodiment, the data set to be detected is preferably set to a multimodal rumor data set to be detected that is publicly released from an online platform.

[0071] S2. Enhance the data in the dataset to be tested to obtain the final enhanced text .

[0072] In order to increase the diversity and richness of text modality data, a pre-trained large language model is used to perform data enhancement processing on the data in the test data set. Preferably, in this embodiment, the pre-trained large language model is preferably any one of the existing models such as GPT-4 and Tongyi Qianwen.

[0073] Furthermore, the specific method of using the pre-trained large language model to perform data enhancement processing on the data in the test data set is:

[0074] By writing a script to send a request to the large language model API in a non-streaming call mode, the prompt words are used to guide the model to generate text variants and image description texts that meet specific needs. Preferably, the prompt words are divided into the first type of prompt words, the second type of prompt words and the third type of prompt words. The first type of prompt words refers to the text enhancement method based on regularization. This type of prompt words directly operates the original text on the basis of following specific specifications to generate new enhanced text. These operations usually include back translation, random insertion, random deletion and synonym replacement. For example, through the back translation operation, the original sentence "I like eating apples" can be translated into English "I like eating apples", and then translated back into Chinese may become "I love eating apples", obtaining different expressions while keeping the semantics unchanged. Random insertion is to add some related words to the original sentence, such as changing "I like apples" to "I like apples very much". Random deletion deletes some words, such as changing "I like to eat fresh apples" to "I like apples". Synonym replacement replaces the words in the sentence with synonyms, such as changing "This question is very important" to "This question is very critical". These prompts guide the large language model to adjust vocabulary and sentence structure through predefined rules to expand the diversity of data while maintaining semantic consistency.

[0075] The second type of prompts focuses on diversifying text expressions through redescription. This type of prompts leverages the generative power of large language models to encourage the model to rephrase or adjust sentence structures in different ways. For example, when rephrasing, the original sentence "I like apples" can be changed to "Apples are one of my favorite fruits" to convey the same meaning in different ways of expression. When changing the tone or style, the formal sentence "Please reply to this email as soon as possible" can be changed to the informal expression "Can you reply to my email soon?". In addition, the amount of information can be increased by expanding the description, such as changing "I like apples" to "I especially like to eat sweet, freshly picked apples because they are very delicious"; or by simplifying the description and removing redundant information, "Because apples taste good, I especially like it" is simplified to "I like apples." Unlike the first type of prompts, the second type of prompts is more flexible, emphasizing natural generation and diversity of expressions, helping to increase the richness of the text.

[0076] The third type of prompt words is designed to guide the large language model to generate image description text suitable for the visual modality. By combining specific contextual information, style requirements or semantic features, the model can generate image description text that meets the intended purpose and provide support for subsequent image generation, data annotation or multimodal tasks. The image description text generated by the third type of prompt words usually has clear descriptiveness, structured semantic information and good language fluency. For example, when describing an image in conjunction with a news topic, the model can construct an information-rich text based on the visual elements in the image (such as characters, scenes, events, etc.), which can not only convey the core content of the image, but also highlight the situational background and emotional atmosphere related to the image.

[0077] in:

[0078] The first type of prompt words guide the large language model to analyze text modal data under specific specifications. Perform data enhancement to obtain enhanced text generated by the first type of prompt words Specifically, the enhanced text generated by the first type of prompt words is specifically manifested as specifying the large language model to perform data enhancement by back-translation, random insertion, random deletion, etc. This type of prompt word guides the model to generate new text by changing the vocabulary or sentence structure while maintaining the semantics of the original text. Furthermore, the first type of prompt word can be "Please use synonyms to replace the words in the following sentence."

[0079] Enhanced text generated by the first type of prompt words The expression is as follows:

[0080] ;

[0081] in, It is a language model enhancement method based on the first type of prompt words;

[0082] The second type of prompt words encourages the model to redescribe the text modality data. Realize diversified text expression and obtain enhanced text generated by the second type of prompt words . Further, the second type of prompt words may be "Please restate the following in a different way".

[0083] Enhanced text generated by the second type of prompt words The expression is as follows:

[0084] ;

[0085] in, A language model enhancement method based on the second type of prompt words;

[0086] The third type of prompt words guide the large language model to analyze image modality data in specific scenarios. Describe it in detail to get the enhanced text generated by the third type of prompt words ;

[0087] Enhanced text generated by the third type of prompt words The expression is as follows:

[0088] ;

[0089] in, A language model enhancement method based on the third type of prompt words;

[0090] Enhanced text generated by the first type of prompt words , enhanced text generated by the second type of prompt words And the enhanced text generated by the third type of prompt words Merge to get a semantically rich and diverse enhanced text .

[0091] Enhanced Text The expression is as follows:

[0092] .

[0093] S3. In enhanced text Based on this, the prompt learning technology is used to combine the enhanced text with the predefined prompt template to create the input prompt enhanced text.

[0094] In enhanced text Based on this, the prompt learning technology is used to combine the enhanced text with the predefined prompt template to create the prompt enhanced text.

[0095] Specifically, the prompt template includes a trainable marker and a " <mask>"Placeholder," <mask>” placeholder is used to indicate the words related to the model’s prediction of news authenticity;

[0096] Use the pre-trained language model RoBERTa-base to process the prompt augmented text. The pre-trained language model RoBERTa-base can output " <mask>The probability distribution of possible words marked by ” helps to get an initial assessment of the authenticity of the text.

[0097] Furthermore, the specific process of creating the prompt enhanced text is as follows:

[0098] S3.1, prompt-enhanced text generation;

[0099] Enhanced text after large language model enhancement , integrated with predefined prompt templates to form prompt enhanced text The prompt template consists of a trainable marker and a " <mask>” placeholders are used to guide the model in predicting missing words in a sentence.

[0100] Prompt Template The expression is as follows:

[0101] ;

[0102] in, and are all trainable prompt vectors, That is, the placeholder that the large language model needs to predict.

[0103] Tip Enhanced Text The expression is as follows:

[0104] ;

[0105] S3.2, model processing and probability distribution output;

[0106] Use RoBERTa-base model to enhance text RoBERTa-base is a pre-trained masked language model that predicts the most likely fill value of the mask tag and outputs the probability distribution of the possible fill value for each corresponding mask position. This probability distribution provides a basis for evaluating the authenticity of each word, helping the model to detect rumors more accurately.

[0107] The probability distribution of possible fill values ​​for each mask position The expression is as follows:

[0108] ;

[0109] in, For pre-trained masked language models.

[0110] S4. Use the visual encoder of the CLIP model to convert the image modality data in each article in the dataset to be detected Convert to visual feature representation ;

[0111] And, based on enhanced text Get text feature representation and sentiment-enhanced text feature representation .

[0112] Furthermore, the specific process of converting the image modality data in each article in the dataset to be detected into visual feature representation is as follows:

[0113] S4.1, visual feature representation and text feature extraction;

[0114] Vision encoder via CLIP model The image modality data in each article Converted into visual feature representation ;

[0115] Visual feature representation The expression is as follows:

[0116] ;

[0117] Through the text encoder in the CLIP model Will enhance the text Convert to text feature representation ;

[0118] Text feature representation The expression is as follows:

[0119] ;

[0120] S4.2, obtain sentiment-enhanced text feature representation;

[0121] ①. Enhance the text Extract the sentiment polarity of the text through sentiment analysis tools (such as TextBlob) and subjectivity , emotional polarity Reflects the sentiment of the text (positive or negative sentiment), and its value range is [-1,1], where -1 means very negative and 1 means very positive. It is used to measure the subjectivity of the text. Its value range is [0,1], where 0 means completely objective and 1 means very subjective.

[0122] Emotional Polarity , Subjectivity The expression is as follows:

[0123] ;

[0124] ②, Emotional polarity Normalize it to scale its value range to the [0,1] interval for unified processing. Normalized sentiment polarity The expression is as follows:

[0125] ;

[0126] in, is the minimum value of sentiment polarity (-1), is the maximum value of sentiment polarity (1);

[0127] Normalized sentiment polarity The range of the expression is [0,1] as follows:

[0128]

[0129] ③. Combined with normalized sentiment polarity and subjectivity , calculate the sentiment element ; In order to avoid the disappearance of subsequent features due to zero-valued sentiment elements, the sentiment elements Adjust to at least 0.1.

[0130] Emotional elements The expression is as follows:

[0131] ;

[0132] ④、Normalized emotional elements With text feature representation Multiply them together to get the sentiment-enhanced text features ;

[0133] Normalized emotional elements The expression is as follows:

[0134] ;

[0135] in, is the minimum value of the emotional element, is the maximum value of the emotional element;

[0136] Sentiment-enhanced text features The expression is as follows:

[0137] .

[0138] S5. Fusion of visual feature representation via self-attention and multi-head attention mechanisms and sentiment-enhanced text feature representation , and obtain the fused visual feature representation And the fused text feature representation ;

[0139] Optimizing the fused visual feature representation using residual connections And the fused text feature representation , generate visual integrated feature vector and text integrated feature vector ;

[0140] Combined cosine similarity to visual integrated feature vector and text integrated feature vector Weighted to generate the final visual cross-modal features and text cross-modal features ;

[0141] Visual cross-modal features and text cross-modal features The text and visual information balanced with adjustable weight coefficients are input into the fully connected layer, and the prediction result is finally output.

[0142] It specifically includes the following steps:

[0143] S5.1, Modal interaction under multi-head attention mechanism;

[0144] ① Use the self-attention mechanism to convert the query vector , key vector Sum value vector Combine to calculate the attention weight of each head ;

[0145] Attention weights for each head The expression is as follows:

[0146] ;

[0147] in, Used to normalize the attention scores to ensure that the sum of the weights is 1; It is a scaling factor to avoid gradient explosion caused by too large inner product.

[0148] ② Use multi-head attention mechanism to represent visual features and sentiment-enhanced text feature representation Interact to generate fused visual feature representation And the fused text feature representation ; Among them, the output of the multi-head attention mechanism is obtained by concatenating the outputs of multiple attention heads.

[0149] Fusion text feature representation The expression is as follows:

[0150] ;

[0151] Fusion of visual features The expression is as follows:

[0152] ;

[0153] in, It is the implementation of the multi-head attention mechanism; It means concatenating the outputs of multiple attention heads, using a power of 2 value (1, 2, 4, 8) as The number of and sentiment-enhanced text feature representation Dimension.

[0154] S5.2, residual connection and fusion framework;

[0155] ① Use the residual connection strategy to represent the fused visual features And the fused text feature representation Added to the visual feature representation and sentiment-enhanced text feature representation To obtain visual integration features and text integration features .

[0156] Visual integration features The expression is as follows:

[0157] ;

[0158] Text Integration Features The expression is as follows:

[0159] ;

[0160] ②Integrate visual features and text integration features Input into the fusion framework of the large language model to generate a visual integrated feature vector and text integrated feature vector ; Among them, the fusion framework mixes linear transformation, batch normalization, Activation and Dropout layers.

[0161] Visual integrated feature vector The expression is as follows:

[0162] ;

[0163] Text Ensemble Feature Vector The expression is as follows:

[0164] ;

[0165] in, is the linear transformation layer, is the batch normalization layer, is the dropout layer.

[0166] S5.3. In order to reduce noise and enhance the synergy between modalities, cosine similarity is used to measure the relationship between the visual features. and text feature representation Similarity .

[0167] Similarity The expression is as follows:

[0168] ;

[0169] Then the cosine similarity is passed through Activate and generate weights. By using weights, the visual integrated feature vector and text integrated feature vector Weighted to generate the final visual cross-modal features and text cross-modal features .

[0170] Visual cross-modal features The expression is as follows:

[0171] ;

[0172] Text cross-modal features The expression is as follows:

[0173] ;

[0174] S5.4, prompt selection optimization and classification decision;

[0175] ①, through the residual connection, the probability distribution Respectively with visual cross-modal features , text cross-modal features Combined with, further optimize the prompt selection in the continuous embedding space. At the same time, an adjustable weight coefficient is introduced , to balance the importance of text and visual information in the final decision process, and obtain the composite feature vector .

[0176] Composite eigenvector The expression is as follows:

[0177] ;

[0178] ②、Compound feature vector The input is further processed into the fully connected layer (FC) to generate the classification probability distribution , finally determined" <mask>” Mark the most appropriate filler word.

[0179] Classification probability distribution The expression is as follows:

[0180] ;

[0181] ;

[0182] in, is a set of categories (news is true or false), is the embedding of the true label, is the embedding of the predicted label word, is a fully connected layer.

[0183] ③、The cross entropy loss Minimizes to:

[0184] ;

[0185] ④. Use the cross-entropy loss function to optimize the large language model parameters. This loss function measures the difference between the probability distribution predicted by the large language model and the true label distribution, so that the optimized large language model can more accurately predict the correct filler word given the input. This filler word is usually a word indicating the authenticity of the news, such as "true" or "false", thereby indirectly achieving the output of rumor detection classification results.

[0186] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.< / mask> < / mask> < / mask> < / mask> < / mask> < / mask>

Claims

1. A multimodal network rumor detection method based on large language model feature enhancement, characterized in that: The following steps are involved: S1. Obtain the text modality data of news information in the dataset to be tested and image modality data ; S2. Text modal data and image modality data Perform enhancement processing to obtain the final enhanced text ; S3. In enhanced text Based on the Combine with predefined prompt templates to create input prompts to enhance text ; S4. Use the visual encoder of the CLIP model to convert the image modality data in each article in the dataset to be detected Convert to visual feature representation ; And, based on enhanced text Get text feature representation and sentiment-enhanced text feature representation ; S5. Fusion of visual feature representation via self-attention and multi-head attention mechanisms and sentiment-enhanced text feature representation , and obtain the fused visual feature representation And the fused text feature representation ; Optimizing the fused visual feature representation using residual connections And the fused text feature representation , generate visual integrated feature vector and text integrated feature vector ; Combined cosine similarity to visual integrated feature vector and text integrated feature vector Weighted to generate the final visual cross-modal features and text cross-modal features ; Visual cross-modal features and text cross-modal features The text information and visual information balanced with adjustable weight coefficients are input into the fully connected layer, and the prediction result is finally output.

2. According to claim 1, the multimodal network rumor detection method based on large language model feature enhancement is characterized in that: The specific method of using the pre-trained large language model to perform data enhancement processing on the data in the test dataset is: Write a script to send a request to the Big Language Model API in a non-streaming call mode, and use prompt words to guide the model to generate text variants and image description text that meet specific needs; The prompt words are divided into first-category prompt words, second-category prompt words and third-category prompt words. in: The first type of prompt words guide the large language model to analyze text modal data Perform data enhancement to obtain enhanced text generated by the first type of prompt words ; The second type of prompt words encourages the model to redescribe the text modality data. Perform text diversification to obtain enhanced text generated by the second type of prompt words ; The third type of prompt words guide the large language model to image modality data Describe it in detail to get the enhanced text generated by the third type of prompt words ; The enhanced text generated by the first type of prompt words , enhanced text generated by the second type of prompt words And the enhanced text generated by the third type of prompt words Merge to get a semantically rich and diverse enhanced text .

3. The multimodal network rumor detection method based on large language model feature enhancement according to claim 2 is characterized in that: The specific process of creating prompt enhanced text is as follows: S3.1, prompt-enhanced text generation; Enhanced text after large language model enhancement , integrated with predefined prompt templates to form prompt enhanced text ; S3.2, model processing and probability distribution output; Use RoBERTa-base model to enhance text Processing is performed to obtain the probability distribution of possible fill values ​​at the mask mark position .

4. The multimodal network rumor detection method based on large language model feature enhancement according to claim 3 is characterized in that: Get visual feature representation , text feature representation and sentiment-enhanced text feature representation The specific process is as follows: Vision encoder via CLIP model The image modality data in each article Converted into visual feature representation ; Text encoder via CLIP model Will enhance the text Convert to text feature representation ; Apply sentiment analysis tools to extract enhanced text The emotional polarity and subjectivity in the text are normalized to the interval [0,1] to obtain the normalized emotional element; the normalized emotional element is then multiplied with the text feature representation to obtain the emotionally enhanced text feature representation. .

5. The multimodal network rumor detection method based on large language model feature enhancement according to claim 3 is characterized in that: The specific process of converting the image modality data in each article in the dataset to be tested into visual feature representation is as follows: S4.1, visual feature representation and text feature extraction; Vision encoder via CLIP model The image modality data in each article Converted into visual feature representation ; Through the text encoder in the CLIP model Will enhance the text Convert to text feature representation ; S4.2, obtain sentiment-enhanced text feature representation; ①. Enhance the text Extract the sentiment polarity of the text through sentiment analysis tools and subjectivity ; ②Emotional polarity Normalize it so that its value range is scaled to the interval [0,1]; ③. Combined with normalized sentiment polarity and subjectivity , calculate the sentiment element ; ④、Normalized emotional elements With text feature representation Multiply them together to get the sentiment-enhanced text features .

6. The multimodal network rumor detection method based on large language model feature enhancement according to claim 4 or 5, characterized in that: Get the fused visual feature representation And the fused text feature representation The specific process is as follows: Using the self-attention mechanism, the query vector , key vector Sum value vector Combine to calculate the attention weight of each head ; Use multi-head attention mechanism to represent visual features and sentiment-enhanced text feature representation Interact to generate fused visual feature representation And the fused text feature representation ; Among them, the output of the multi-head attention mechanism is obtained by concatenating the outputs of multiple attention heads.

7. The multimodal network rumor detection method based on large language model feature enhancement according to claim 6 is characterized in that: Generate visual ensemble feature vector and text integrated feature vector The specific process is as follows: Using the residual connection strategy, the fused visual features are represented And the fused text feature representation Added to the visual feature representation and sentiment-enhanced text feature representation To obtain visual integration features and text integration features ; Integrate visual features and text integration features Input into the fusion framework to generate a visual integrated feature vector and text integrated feature vector ; Among them, the fusion framework mixes linear transformation, batch normalization, Activation and Dropout layers.

8. The multimodal network rumor detection method based on large language model feature enhancement according to claim 7 is characterized in that: Generate the final visual cross-modal features and text cross-modal features The specific process is as follows: Using cosine similarity to measure visual feature representation and text feature representation Similarity ; Then the cosine similarity is passed Activation, generating weights; By using weights to integrate the visual feature vectors and text integrated feature vector Weighted to generate the final visual cross-modal features and text cross-modal features .

9. The multimodal network rumor detection method based on large language model feature enhancement according to claim 8 is characterized in that: Visual cross-modal features and text cross-modal features The specific process of inputting the fully connected layer together with the text and visual information balanced by the adjustable weight coefficient and finally outputting the prediction result is as follows: The probability distribution is transformed through residual connection Respectively with visual cross-modal features , text cross-modal features Combined with, further optimize the prompt selection in the continuous embedding space; at the same time, an adjustable weight coefficient is introduced , to balance the importance of text and visual information in the final decision process, and obtain the composite feature vector ;in, ; The composite feature vector Input to the fully connected layer for further processing to generate classification probability distribution , finally determined" <mask> ” Mark the most appropriate filler word;< / mask> The cross entropy loss function is used to optimize the parameters of the large language model so that the large language model can more accurately predict the correct filler words given the input, thereby indirectly achieving the result output of rumor detection classification, that is, the final output prediction result.

Images