Rumor and evidence detection method based on language model and multi-modal fusion

Abstract

The invention provides a rumor and evidence detection method based on a language model and multi-modal fusion. The rumor and evidence detection method comprises the steps that a to-be-verified text material and a to-be-verified image material are acquired; respectively carrying out feature extraction on the text material and the image material to obtain a text feature vector and an image feature vector; performing feature interaction fusion on the text feature vector and the image feature vector by adopting a multi-head self-attention mechanism to obtain a unified feature vector; inputting the unified feature vector into a fully connected layer classifier, and outputting rumor identification probability distribution; and determining rumor evidence based on the rumor identification probability distribution. The rumor identification efficiency and accuracy are improved through the deep learning technology. And meanwhile, the method has the advantages of high automation degree, reduction of manual intervention, capability of comprehensively understanding material contents and the like. In practical application, the method can be widely applied to rumor identification and evidence locking tasks in scenes such as social media and news websites.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a rumor and evidence detection method based on language model and multimodal fusion. Background Art

[0002] In the current digital information age, the popularity of the Internet and social media has made the speed and breadth of information dissemination reach an unprecedented level. However, this convenient way of information dissemination has also provided a breeding ground for the spread of rumors, and the speed and scope of its spread have increased exponentially, which has had a profound impact on personal privacy, public opinion and even social stability. Although traditional rumor identification methods, such as manual review and user reporting, can curb the spread of rumors to a certain extent, they are inefficient and highly dependent on human subjective judgment, and can no longer meet the needs of rapid identification and response to rumors in the modern social media environment.

[0003] In this context, the development of automated rumor identification and evidence locking technology based on artificial intelligence is particularly important. This technology can use big data analysis and machine learning algorithms to quickly and accurately identify rumors from massive amounts of social media information and automatically lock relevant evidence, greatly improving the efficiency and accuracy of rumor response. It can not only block the spread of rumors in a timely manner and reduce the negative impact of rumors on individuals and society, but also provide strong evidence support for relevant departments to facilitate subsequent accountability and handling. Therefore, the automated rumor identification and evidence locking method based on artificial intelligence has important significance and value in maintaining network information security and protecting the public interest. Summary of the invention

[0004] The present invention provides a rumor and evidence detection method based on language model and multimodal fusion, so as to solve the defects existing in the prior art.

[0005] In a first aspect, the present invention provides a rumor and evidence detection method based on a language model and multimodal fusion, comprising: Obtain textual and image materials to be verified; Extracting features from the text material and the image material respectively to obtain a text feature vector and an image feature vector; A multi-head self-attention mechanism is used to interactively fuse the text feature vector and the image feature vector to obtain a unified feature vector; Input the unified feature vector into a fully connected layer classifier, and output a rumor identification probability distribution; Based on the rumor identification probability distribution, rumor evidence is determined.

[0006] According to a rumor and evidence detection method based on language model and multimodal fusion provided by the present invention, text materials and image materials to be verified are obtained, including: Collect text titles and images to be verified from multiple channels, as well as associated text evidence materials and image evidence materials.

[0007] According to a rumor and evidence detection method based on language model and multimodal fusion provided by the present invention, feature extraction is performed on the text material and the image material respectively to obtain a text feature vector and an image feature vector, including: Extracting features of the text title and the text evidence material based on the Ernie-m model to obtain title features and text evidence features, and obtaining the text feature vector from the title features and the text evidence features; Based on the pre-trained ResNet model, feature extraction is performed on the image and the image evidence material to obtain image features and image evidence features, and the image feature vector is obtained from the image features and the image evidence features.

[0008] According to a rumor and evidence detection method based on language model and multimodal fusion provided by the present invention, a multi-head self-attention mechanism is used to perform feature interactive fusion on the text feature vector and the image feature vector to obtain a unified feature vector, including: Calculating attention weights of the text material and the image material; Based on the attention weight, the text feature vector and the image feature vector that have passed the multi-head self-attention mechanism are concatenated with the title feature and the image feature to form the unified feature vector.

[0009] According to a rumor and evidence detection method based on language model and multimodal fusion provided by the present invention, the unified feature vector is input into a fully connected layer classifier, and a rumor identification probability distribution is output, including: The unified feature vector is input into the fully connected layer classifier, and the softmax function is used as the activation function of the fully connected layer classifier to output the rumor identification probability distribution corresponding to multiple categories.

[0010] According to a rumor and evidence detection method based on language model and multimodal fusion provided by the present invention, based on the rumor identification probability distribution, determining rumor evidence includes: Determining a plurality of rumor evidence probability distribution intervals, wherein the plurality of rumor evidence probability distribution intervals correspond to the probability of the rumor being true; The text material and the image material corresponding to the rumor identification probability distribution and located within a specified rumor evidence probability distribution interval are extracted as the rumor evidence.

[0011] In a second aspect, the present invention further provides a rumor and evidence detection system based on language model and multimodal fusion, comprising: An acquisition module, used to acquire text materials and image materials to be verified; An extraction module, used to extract features from the text material and the image material respectively to obtain a text feature vector and an image feature vector; A fusion module, used for performing feature interactive fusion on the text feature vector and the image feature vector by using a multi-head self-attention mechanism to obtain a unified feature vector; A classification module, used for inputting the unified feature vector into a fully connected layer classifier and outputting a rumor identification probability distribution; A determination module is used to determine rumor evidence based on the rumor identification probability distribution.

[0012] In a third aspect, the present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the rumor and evidence detection method based on language model and multimodal fusion as described in any one of the above is implemented.

[0013] In a fourth aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a rumor and evidence detection method based on a language model and multimodal fusion as described in any one of the above.

[0014] In a fifth aspect, the present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned rumor and evidence detection methods based on language model and multimodal fusion.

[0015] The rumor and evidence detection method based on language model and multimodal fusion provided by the present invention combines language model and multimodal data, and improves the efficiency and accuracy of rumor identification through deep learning technology. At the same time, the present invention also has the advantages of high degree of automation, reduced manual intervention, and more comprehensive understanding of the content of the material. In practical applications, the present invention can be widely used in rumor identification and evidence locking tasks in scenarios such as social media and news websites. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 This is one of the flow charts of the rumor and evidence detection method based on language model and multimodal fusion provided by the present invention; Figure 2 This is the second flow chart of the rumor and evidence detection method based on language model and multimodal fusion provided by the present invention; Figure 3 It is a structural schematic diagram of a rumor and evidence detection system based on a language model and multimodal fusion provided by the present invention; Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] In view of the problems existing in the prior art, the present invention applies multimodal detection technology, combines language models and multimodal data, and improves the efficiency and accuracy of rumor identification through deep learning technology. Multimodal detection plays a vital role in the automated rumor identification and evidence locking technology based on artificial intelligence. Since the information in social media often exists in multiple modes such as text, images, and videos, a single text detection or image processing technology can no longer meet the comprehensive rumor detection needs. Multimodal detection can comprehensively analyze multiple information such as text, images, and videos, and consider the authenticity of information from multiple dimensions, so as to more accurately identify rumors. For example, in some cases, text information may be ambiguous, but combined with relevant image or video evidence, the authenticity of the information can be quickly determined. Therefore, multimodal detection has an irreplaceable position in the automated rumor identification and evidence locking method, and is a key link to ensure the accuracy and efficiency of the technology.

[0020] Figure 1 This is one of the flow charts of the rumor and evidence detection method based on language model and multimodal fusion provided by an embodiment of the present invention, such as Figure 1 As shown, including: Step 100: Obtain text materials and image materials to be verified; Step 200: extracting features from the text material and the image material respectively to obtain a text feature vector and an image feature vector; Step 300: using a multi-head self-attention mechanism to perform feature interactive fusion on the text feature vector and the image feature vector to obtain a unified feature vector; Step 400: input the unified feature vector into a fully connected layer classifier, and output a rumor identification probability distribution; Step 500: Determine rumor evidence based on the rumor identification probability distribution.

[0021] The present invention combines language models and multimodal data, combines deep image feature extraction with multilingual text processing, and constructs a multimodal rumor identification system that can simultaneously process image and text data. This method not only enriches the data source for rumor identification, but also improves the system's ability to identify different types of rumors; cross-language text processing capability: by introducing the pre-trained Ernie-m multilingual model, the present invention realizes the synchronous processing of Chinese and English texts, broadens the language application scope of the rumor identification system, and enhances its practicality and internationalization level; innovative feature interaction mechanism: the present invention designs a unique feature interaction stage, and uses a multi-head self-attention mechanism to achieve deep interaction between titles and text evidence, and between images and image evidence. This interaction mechanism helps to explore the potential connections between data and improves the sensitivity and accuracy of rumor identification; efficient rumor identification and evidence locking process: the present invention effectively splices multimodal features through a fully connected layer, and uses a classifier to quickly and accurately identify rumors. This process design not only simplifies the operating steps of rumor detection, but also greatly improves the processing speed and accuracy, providing a powerful tool for timely response to online rumors.

[0022] Based on the above embodiment, step 100 includes: Collect text titles and images to be verified from multiple channels, as well as associated text evidence materials and image evidence materials.

[0023] Specifically, for rumor identification, we first need to collect and organize the titles and images that need to be verified, as well as the text and image evidence materials that support verification. These data can come from a variety of channels such as social media, news websites, forums, etc. In the process of collecting data, it is necessary to ensure the authenticity and reliability of the data, and perform necessary preprocessing on the data, such as cleaning and deduplication. For example, we can collect titles and images of suspected rumors that have been circulating recently from social media platforms such as Weibo and WeChat public accounts, and collect relevant text and image evidence materials.

[0024] The present invention can not only process single text or image data, but also simultaneously process multimedia data containing text and images, thereby more comprehensively capturing various manifestations of rumors.

[0025] Based on the above embodiment, step 200 includes: Extracting features of the text title and the text evidence material based on the Ernie-m model to obtain title features and text evidence features, and obtaining the text feature vector from the title features and the text evidence features; Based on the pre-trained ResNet model, feature extraction is performed on the image and the image evidence material to obtain image features and image evidence features, and the image feature vector is obtained from the image features and the image evidence features.

[0026] Specifically, Figure 2 As shown in the figure, in the feature extraction stage, the ResNet model is used to extract deep features for the input image data; for text data, the pre-trained Ernie-m multilingual model is used to simultaneously process Chinese and English texts to obtain text features. The processed text data includes the title description qCap or the description qImg of the image material to be verified, as well as the text evidence material features caps and image evidence material features imgs used to support verification.

[0027] It is understandable that for image data, the embodiment of the present invention uses a pre-trained ResNet model for feature extraction, and extracts deep features of the image through multiple residual learning modules. The model has a powerful image feature extraction capability and can effectively capture key information in the image. The feature vector of the image can be obtained through the ResNet model.

[0028] For text data, the embodiment of the present invention uses the Ernie-m model for feature extraction. Ernie-m is a Transformer-based multilingual model designed to solve cross-language natural language processing tasks, especially for small languages ​​with fewer resources. The model was proposed by Baidu Research Institute to improve the effect of cross-language understanding, especially in languages ​​with sparse data. Ernie-m uses a new training method and a back-translation mechanism to learn the semantic alignment relationship between languages ​​from a small number of parallel corpora, significantly improving the effects of a variety of typical cross-language tasks including cross-language natural language inference, machine question answering, named entity recognition, agreed paraphrase recognition, and cross-language retrieval. The model has been pre-trained on a large-scale corpus, can effectively capture the semantic information in the text, and output a vector representation of the text. The feature vector of the text can be obtained through the Ernie-m model.

[0029] By introducing the ResNet and Ernie-m models, the present invention optimizes the processing process of image and text data, bringing many beneficial effects, including improving the depth and accuracy of feature extraction, enhancing the ability to capture text semantic information, improving the consistency of cross-modal data processing, expanding the breadth of application scenarios, and improving the robustness and generalization ability of the system.

[0030] Based on the above embodiment, step 300 includes: Calculating attention weights of the text material and the image material; Based on the attention weight, the text feature vector and the image feature vector that have passed the multi-head self-attention mechanism are concatenated with the title feature and the image feature to form the unified feature vector.

[0031] Specifically, Figure 2 As shown, in the feature interaction stage, the embodiment of the present invention realizes the interaction between the title and text evidence materials, and the interaction between the image and the image evidence materials through the multi-head self-attention mechanism, so as to output the evidence features that are highly related to the title and image to be verified. By applying the multi-head self-attention mechanism, the attention weights between the title or image to be verified and the respective related evidence materials are calculated to strengthen the expression of important features. By weighted summation, the evidence features are fused according to the attention weights to generate the interactive features caps_feature and imgs_features for the title and image to be verified.

[0032] The image features and text features are concatenated to form a unified feature vector, which contains both image information and text information, providing rich feature information for subsequent rumor identification.

[0033] This invention significantly improves the performance of the model in dealing with complex verification tasks by introducing a multi-head self-attention mechanism and weighted feature fusion. The flexibility and accuracy of this method enable the model to better adapt to different data and task requirements, laying a solid foundation for subsequent verification work. At the same time, by strengthening the expression of important features and fusing multi-source information, the scheme also enhances the robustness and generalization ability of the model, making it more advantageous in practical applications.

[0034] Based on the above embodiment, step 400 includes: The unified feature vector is input into the fully connected layer classifier, and the softmax function is used as the activation function of the fully connected layer classifier to output the rumor identification probability distribution corresponding to multiple categories.

[0035] Specifically, after obtaining the unified feature vector, we input it into the fully connected layer classifier for rumor identification. This classifier can determine whether it is a rumor based on the input feature vector. Specifically, the softmax function can be used as the activation function of the classifier to output the probability distribution of each category. By comparing the probabilities of different categories, we can determine whether the input data is a rumor.

[0036] Based on the above embodiment, step 500 includes: Determining a plurality of rumor evidence probability distribution intervals, wherein the plurality of rumor evidence probability distribution intervals correspond to the probability of the rumor being true; The text material and the image material corresponding to the rumor identification probability distribution and located within a specified rumor evidence probability distribution interval are extracted as the rumor evidence.

[0037] Specifically, after judging the authenticity of the rumor based on the output of the classifier, we can further lock in the relevant evidence. Specifically, we can determine which evidence is related to the rumor based on the probability distribution of the classifier output and use it as locked evidence. For example, if the classifier determines that a certain title and image are rumors and gives a high probability value, then we can assume that the text and image evidence materials related to the title and image are evidence supporting the rumor. This evidence can be used for subsequent tasks such as rumor tracing and propagation path analysis.

[0038] The present invention comprehensively considers multimodal information and uses deep learning technology to refine features, which not only improves the accuracy of rumor identification but also realizes the rapid locking of evidence materials.

[0039] The rumor and evidence detection system based on language model and multimodal fusion provided by the present invention is described below. The rumor and evidence detection system based on language model and multimodal fusion described below and the rumor and evidence detection method based on language model and multimodal fusion described above can be referenced to each other.

[0040] Figure 3 is a structural diagram of a rumor and evidence detection system based on a language model and multimodal fusion provided by an embodiment of the present invention, such as Figure 3 As shown, it includes: an acquisition module 31, an extraction module 32, a fusion module 33, a classification module 34 and a determination module 35, wherein: The acquisition module 31 is used to obtain the text material and image material to be verified; the extraction module 32 is used to perform feature extraction on the text material and the image material respectively to obtain a text feature vector and an image feature vector; the fusion module 33 is used to use a multi-head self-attention mechanism to perform feature interactive fusion on the text feature vector and the image feature vector to obtain a unified feature vector; the classification module 34 is used to input the unified feature vector into a fully connected layer classifier and output a rumor identification probability distribution; the determination module 35 is used to determine the rumor evidence based on the rumor identification probability distribution.

[0041] Figure 4 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 4As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communication interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute a rumor evidence identification method based on multimodal fusion, the method comprising: obtaining text materials and image materials to be verified; performing feature extraction on the text materials and the image materials respectively to obtain a text feature vector and an image feature vector; performing feature interactive fusion on the text feature vector and the image feature vector using a multi-head self-attention mechanism to obtain a unified feature vector; inputting the unified feature vector into a fully connected layer classifier to output a rumor identification probability distribution; and determining rumor evidence based on the rumor identification probability distribution.

[0042] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0043] On the other hand, the present invention also provides a computer program product, which includes a computer program, and the computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the rumor and evidence detection method based on language model and multimodal fusion provided by the above methods, and the method includes: obtaining text materials and image materials to be verified; performing feature extraction on the text materials and the image materials respectively to obtain text feature vectors and image feature vectors; using a multi-head self-attention mechanism to perform feature interactive fusion on the text feature vector and the image feature vector to obtain a unified feature vector; inputting the unified feature vector into a fully connected layer classifier to output a rumor identification probability distribution; and determining rumor evidence based on the rumor identification probability distribution.

[0044] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it is implemented to execute the rumor and evidence detection method based on language model and multimodal fusion provided by the above-mentioned methods, the method comprising: obtaining text materials and image materials to be verified; performing feature extraction on the text materials and the image materials respectively to obtain text feature vectors and image feature vectors; using a multi-head self-attention mechanism to perform feature interactive fusion on the text feature vectors and the image feature vectors to obtain a unified feature vector; inputting the unified feature vector into a fully connected layer classifier to output a rumor identification probability distribution; and determining rumor evidence based on the rumor identification probability distribution.

[0045] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0046] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0047] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A rumor and evidence detection method based on language model and multimodal fusion, characterized in that: include: Obtain textual and image materials to be verified; Extracting features from the text material and the image material respectively to obtain a text feature vector and an image feature vector; A multi-head self-attention mechanism is used to interactively fuse the text feature vector and the image feature vector to obtain a unified feature vector; Input the unified feature vector into a fully connected layer classifier, and output a rumor identification probability distribution; Based on the rumor identification probability distribution, rumor evidence is determined.

2. The rumor and evidence detection method based on language model and multimodal fusion according to claim 1 is characterized in that: Obtain text and image materials to be verified, including: Collect text titles and images to be verified from multiple channels, as well as associated text evidence materials and image evidence materials.

3. The rumor and evidence detection method based on language model and multimodal fusion according to claim 2 is characterized in that: Feature extraction is performed on the text material and the image material respectively to obtain a text feature vector and an image feature vector, including: Extracting features of the text title and the text evidence material based on the Ernie-m model to obtain title features and text evidence features, and obtaining the text feature vector from the title features and the text evidence features; Based on the pre-trained ResNet model, feature extraction is performed on the image and the image evidence material to obtain image features and image evidence features, and the image feature vector is obtained from the image features and the image evidence features.

4. The rumor and evidence detection method based on language model and multimodal fusion according to claim 3 is characterized in that: A multi-head self-attention mechanism is used to interactively fuse the text feature vector and the image feature vector to obtain a unified feature vector, including: Calculating attention weights of the text material and the image material; Based on the attention weight, the text feature vector and the image feature vector that have passed the multi-head self-attention mechanism are concatenated with the title feature and the image feature to form the unified feature vector.

5. The rumor and evidence detection method based on language model and multimodal fusion according to claim 1 is characterized in that: The unified feature vector is input into the fully connected layer classifier, and the rumor identification probability distribution is output, including: The unified feature vector is input into the fully connected layer classifier, and the softmax function is used as the activation function of the fully connected layer classifier to output the rumor identification probability distribution corresponding to multiple categories.

6. The rumor and evidence detection method based on language model and multimodal fusion according to claim 1 is characterized in that: Determining rumor evidence based on the rumor identification probability distribution includes: Determining a plurality of rumor evidence probability distribution intervals, wherein the plurality of rumor evidence probability distribution intervals correspond to the probability of the rumor being true; The text material and the image material corresponding to the rumor identification probability distribution and located within a specified rumor evidence probability distribution interval are extracted as the rumor evidence.

7. A rumor and evidence detection system based on language model and multimodal fusion, characterized in that: include: An acquisition module, used to acquire text materials and image materials to be verified; An extraction module, used to extract features from the text material and the image material respectively to obtain a text feature vector and an image feature vector; A fusion module, used for performing feature interactive fusion on the text feature vector and the image feature vector by using a multi-head self-attention mechanism to obtain a unified feature vector; A classification module, used for inputting the unified feature vector into a fully connected layer classifier and outputting a rumor identification probability distribution; A determination module is used to determine rumor evidence based on the rumor identification probability distribution.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the rumor and evidence detection method based on language model and multimodal fusion as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the rumor and evidence detection method based on language model and multimodal fusion as described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the rumor and evidence detection method based on language model and multimodal fusion as described in any one of claims 1 to 6 is implemented.

Images