Rumor detection method and system based on multi-modal information in low-resource environment

Abstract

The invention discloses a rumor detection method and system based on multi-modal information in a low-resource environment. The method comprises the following steps: collecting multi-modal data of each post in a first field; performing feature extraction on each piece of modal data in the multi-modal data by using a corresponding preset model to obtain a modal feature of each piece of modal data; the modal features of all the modal data are fused by using a multi-modal attention mechanism to obtain multi-modal fusion features; constructing a propagation graph structure for all posts in the first field, and performing convolution on the propagation graph structure by using a graph convolution network to obtain an event-level structure representation representing a propagation relationship of each node in the propagation graph structure, and determining a rumor detection node of the post corresponding to each node by using the event-level structure representation of each node. According to the scheme provided by the invention, the rumor detection accuracy can be effectively improved in a low-resource scene.

Description

Technical Field

[0001] The present invention relates to the field of information processing technology, and in particular to a rumor detection method and system based on multimodal information in a low-resource environment. Background Art

[0002] Existing rumor detection systems are mainly based on single text processing and cannot effectively identify rumors containing multimodal information (such as images, videos, and audio). With the popularity of social media and multimedia content, the application of multimodal information fusion in rumor detection has become increasingly important.

[0003] There are some studies on multimodal rumor detection systems that combine text and image features and use transfer learning and contrastive learning to improve detection results. Specifically, they are as follows:

[0004] 1. Cross-modal contrastive learning: Studies have shown that the cross-modal contrastive learning framework performs well in multimodal rumor detection. This method improves the semantic consistency of image-text pairs by aligning image and text representations, and further optimizes the alignment accuracy through cross-modal consistency learning tasks.

[0005] 2. Multimodal Fusion Network: Some models use self-supervised learning loss to align source post representations in different views, thereby achieving better multimodal fusion. For example, the multimodal feature-enhanced attention network that uses an attention mechanism to combine text, visual, and social graph features has achieved good results in rumor detection.

[0006] 3. Weighted cross-modal feature aggregation: Another method is the weighted cross-modal feature aggregation network, which combines highly correlated features from different modalities and obtains a unified representation in the same space. This method uses adversarial training methods to enhance the robustness of the model.

[0007] Although the existing methods have achieved certain results in some aspects, they also have defects in some aspects, as follows:

[0008] The shortcomings of cross-modal contrastive learning: Single alignment mechanism: Existing cross-modal contrastive learning methods mainly rely on image-text alignment and lack the processing of video and audio modalities. This limits the effectiveness of the system in dealing with rumors containing complex multimedia content. Insufficient hard negative sample mining: Although this method uses contrastive learning, it lacks an effective hard negative sample mining strategy, which may lead to insufficient discrimination of the model for difficult-to-distinguish samples. Insufficient real-time performance: This method mainly focuses on the training and testing of static datasets, fails to effectively process real-time data streams, and cannot detect and respond to emerging rumors in a timely manner.

[0009] The defects of the multimodal feature enhanced attention network: Fixed modal weights: In this method, the weights of modal features are fixed during the training phase and cannot be dynamically adjusted to meet the needs of different situations. This may perform poorly when processing dynamically changing information. Lack of sentiment and psychological analysis: The system mainly focuses on the enhancement of modal features and ignores the extraction of sentiment and psychological features, which may play an important role in the spread of rumors. Insufficient modeling of propagation structure: Although the attention mechanism is used, it is still insufficient in modeling the propagation path and time dynamic features of social networks, and fails to fully utilize the information of the propagation network.

[0010] The defects of the weighted cross-modal feature aggregation network: Lack of consideration of the time dimension: This method does not take into account the time dimension of information propagation, but only performs weighted aggregation on static features, ignoring the temporal characteristics of rumor propagation. Insufficient domain adaptation: The system lacks domain adaptation technology and is difficult to cope with the needs of rumor detection in different fields (such as different languages ​​and cultural backgrounds). Lack of user feedback mechanism: Without the introduction of a user feedback mechanism, the system cannot self-adjust and optimize according to user feedback in actual applications, and has poor robustness.

[0011] In summary, existing technologies still face many challenges in multimodal information fusion, cross-modal alignment and real-time detection. Summary of the invention

[0012] In order to solve the technical problem that the existing rumor detection system has low accuracy and robustness in low-resource scenarios, the embodiments of the present invention provide a rumor detection method and system based on multimodal information in a low-resource environment.

[0013] The technical solution of the embodiment of the present invention is achieved as follows:

[0014] The embodiment of the present invention provides a rumor detection method based on multimodal information in a low-resource environment, the method comprising: collecting multimodal data of each post in a first domain, the multimodal data comprising text modal data, image modal data, video modal data and audio modal data; for each modal data in the multimodal data, using a corresponding preset model to perform feature extraction to obtain the modal features of each modal data; and using a multimodal attention mechanism to fuse the modal features of all modal data to obtain multimodal fusion features; constructing a propagation graph structure for all posts in the first domain, and using a graph convolutional network to convolve the propagation graph structure to obtain an event-level structural representation characterizing the propagation relationship of each node in the propagation graph structure, and using the event-level structural representation of each node to determine the rumor detection result of the post corresponding to each node; each node in the propagation graph structure is each post in the first domain; and the edges in the propagation graph structure are propagation paths between posts in the first domain.

[0015] In one embodiment, multimodal data of each post in the first field is collected, including: for each data source in the first field, determining the data format of the data collected by each data source; the data format includes text, image, video and audio; according to the data format of the data collected by each data source, using an interface corresponding to the data format for collection.

[0016] In one embodiment, after collecting data using an interface corresponding to the data format, the method further includes: verifying the collected data; the verification step includes at least one of the following: calculating a checksum for each collected data, and determining whether the checksum changes during the data transmission process; if so, determining that the data verification fails; checking whether the data is complete at a preset checkpoint; if the data is incomplete, determining that the data verification fails; verifying whether the format of the data is correct, if the format of the data is incorrect, determining that the data verification fails; verifying whether the data source of the data is a secure data source, if the data source of the data is not a secure data source, determining that the data verification fails; performing anomaly detection on the data using a preset algorithm, if the detection result is abnormal, determining that the data verification fails.

[0017] In one embodiment, for each modal data in the multimodal data, feature extraction is performed using a corresponding preset model to obtain the modal features of each modal data, including: for text modal data, semantic information vectors in the text modal data are extracted using a preset language model, and expression and emotion distribution vectors in the text modal data are extracted using a preset expression and emotion distribution knowledge base; the semantic information vectors and the expression and emotion distribution vectors are fused to obtain the modal features of the text modal data; the expression and emotion distribution knowledge base contains a mapping relationship between expression and emotion; for image modal data, feature extraction is performed using a preset convolutional neural network to obtain the modal features of the image modal data; for video modal data, feature extraction is performed using a preset three-dimensional convolutional neural network to obtain the modal features of the video modal data; for audio modal data, feature extraction is performed using a preset audio model to obtain the modal features of the audio modal data.

[0018] In one embodiment, the semantic information vector and the expression emotion distribution vector are fused to obtain the modal feature of the text modal data, including: fusing the semantic information vector and the expression emotion distribution vector using the following calculation formula:

[0019]

[0020] Among them, T is the modal feature of text modal data, G iis the expression emotion distribution vector, hi is the semantic information vector, G i is the expression emotion distribution vector g of all expressions in the text modal data sentence j The weighted average of T r is the data set, V soft(x) Is the expression X in the sentence i The soft label vector, h i =Encoder([cls]+X i +[sep]), [cls] and [sep] are special tags for the preset language model.

[0021] In one embodiment, a multimodal attention mechanism is used to fuse the modal features of all modal data to obtain a multimodal fusion feature, including: performing a linear transformation on the modal features of each modal data to obtain a query vector, a key vector, and a value vector of each modal data; based on the query vector, the key vector, and the value vector of each modal data, using a scaled dot product attention mechanism to calculate the attention score of each modal data; performing a weighted summation on the attention score of each modal data to obtain a multimodal fusion feature; wherein the weighted summation of the attention score of each modal data is performed using the following calculation formula:

[0022]

[0023] Among them, F is the multimodal fusion feature, and are the attention scores of text modality data, image modality data, video modality data, and audio modality data, respectively, T , α I , α V and α A are the weights of text modality data, image modality data, video modality data, and audio modality data, respectively, T +α I +α V +α A =1, d k is the dimension of the key vector, W Q , W K and W V is a linear transformation matrix, T, I, V, and A are the modal features of text modality data, image modality data, video modality data, and audio modality data, respectively.

[0024] In one embodiment, determining the rumor detection result of the post corresponding to each node using the event-level structure representation of each node includes: determining the rumor detection result of the post corresponding to each node using the event-level structure representation of each node by the following calculation formula:

[0025]

[0026] Among them, y is the rumor detection result of each post, is the event-level structure representation of the entire propagation thread, d (l) is the output dimension of the graph neural network, is the hidden representation declared at the (l-1)th layer, ∥ is the concatenation operation with the broadcast mechanism, H (l) is the node feature matrix of the lth layer.

[0027] In one embodiment, the method further includes: collecting multimodal data of each post in the second domain; using a domain adaptive contrast learning mechanism, aligning posts with the same authenticity in the first domain and the second domain, and separating posts with different authenticity, so as to detect rumors in posts in the second domain; wherein, during the alignment process, the contrast loss between the first domain and the second domain is:

[0028]

[0029] in, is the contrast loss between the first field and the second field, N t represents the number of samples in the second domain data, Represents the category label of sample i in the second field The number of samples of the same first field, and Represent the category labels of target sample i and source sample j respectively, N s represents the number of samples in the first domain data, represents the event-level feature representation of the i-th sample in the second domain, Represents the feature representation of samples in the first domain that have the same category as sample i in the second domain, represents the similarity between the second domain sample i and the first domain sample j, The feature representation of samples in the first domain that are not necessarily the same as sample i in the second domain, It represents the similarity between sample i in the second domain and all samples in the first domain.

[0031] An embodiment of the present invention also provides a rumor detection system based on multimodal information in a low-resource environment, comprising: a processor and a memory for storing a computer program that can be run on the processor; wherein, when the processor is used to run the computer program, it executes the steps of any one of the above methods.

[0032] An embodiment of the present invention further provides a storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above methods are implemented.

[0033] The rumor detection method and system based on multimodal information in a low-resource environment provided by the embodiment of the present invention collects multimodal data of each post in the first field, and the multimodal data includes text modal data, image modal data, video modal data and audio modal data; for each modal data in the multimodal data, the corresponding preset model is used to extract features to obtain the modal features of each modal data; and the modal features of all modal data are fused using the multimodal attention mechanism to obtain multimodal fusion features; a propagation graph structure is constructed for all posts in the first field, and the propagation graph structure is convoluted using a graph convolutional network to obtain an event-level structural representation that characterizes the propagation relationship of each node in the propagation graph structure, and the rumor detection result of the post corresponding to each node is determined using the event-level structural representation of each node; each node in the propagation graph structure is each post in the first field; the edge in the propagation graph structure is the propagation path between posts in the first field. The scheme provided by the present invention can effectively improve the accuracy of rumor detection in low-resource scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A schematic diagram of a process of a rumor detection method based on multimodal information in a low-resource environment according to an embodiment of the present invention;

[0035] Figure 2 This is a schematic diagram of the system flow of an embodiment of the present invention;

[0036] Figure 3 1 is a diagram showing the internal structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0037] The present invention will be described in further detail below with reference to the accompanying drawings and embodiments.

[0038] The embodiment of the present invention provides a rumor detection method based on multimodal information in a low-resource environment. Figure 1 As shown, the method includes:

[0039] Step 101: Collecting multimodal data of each post in a first domain, wherein the multimodal data includes text modal data, image modal data, video modal data, and audio modal data;

[0040] Step 102: for each modal data in the multimodal data, extract features using a corresponding preset model to obtain modal features of each modal data; and fuse the modal features of all modal data using a multimodal attention mechanism to obtain a multimodal fusion feature;

[0041] Step 103: Construct a propagation graph structure for all posts in the first field, and use a graph convolutional network to convolve the propagation graph structure to obtain an event-level structural representation that characterizes the propagation relationship of each node in the propagation graph structure, and use the event-level structural representation of each node to determine the rumor detection result of the post corresponding to each node; each node in the propagation graph structure is each post in the first field; the edges in the propagation graph structure are propagation paths between posts in the first field.

[0042] This embodiment mainly discusses the application of multimodal information fusion and graph neural network in rumor detection.

[0043] Specifically, the scheme of this embodiment can be applied to a rumor detection system, which may include a data acquisition module, a multimodal feature extraction module (including an expression and emotion analysis module), a multi-layer graph convolutional network module (including a domain adaptive contrastive learning module and a target contrastive learning module) and a rumor detection module.

[0044] See also Figure 2 , the specific process based on the above modules can be:

[0045] 1. Data acquisition module

[0046] The data collection module is responsible for collecting multimodal data from domestic social media and other online platforms, including text modal data, image modal data, video modal data, and audio modal data. Data source identification and management is the first step in data collection, ensuring that the system can identify and manage different data sources. The specific steps are as follows: Data source identification: Determine the main data sources, such as Weibo, WeChat, Douyin, Zhihu, Toutiao and other domestic social media platforms. Data source interface: Design a corresponding API interface for each data source to ensure that the system can communicate with the data source through the API. Data source metadata management: Establish a data source metadata management system to record the access frequency, data format, content type and other information of each data source.

[0047] Data collection: Use the API interface provided by the system to collect data, such as Weibo API and WeChat public platform API. For data sources that do not provide API, use web crawler technology to collect data and filter relevant content based on predefined keywords to improve collection efficiency.

[0048] That is, in one embodiment, for each data source in the first field, the data format of the data collected by each data source is determined; the data format includes text, images, video and audio; according to the data format of the data collected by each data source, the data is collected using an interface corresponding to the data format.

[0049] Design corresponding data collection strategies based on different data sources and content types to ensure the comprehensiveness and efficiency of data collection.

[0050] Specifically, text modality data can be collected in the following ways:

[0051] The python code for API calls is as follows:

[0052] import requests

[0053] api_key="YOUR_API_KEY"

[0054] url=f"https: / / api.weibo.com / 2 / statuses / public_timeline.json?access_token={api_key}"

[0055] response = requests.get(url)

[0056] statuses=response.json()['statuses']

[0057] #Print the collected Weibo content

[0058] for status in statuses:

[0059] print(status['text'])

[0060] To filter keywords, the python code is as follows:

[0061]

[0062] Image modality data can be collected in the following ways:

[0063] Use the image API provided by the platform to collect data, use web crawler technology to collect image content on web pages, convert and compress the collected images for subsequent processing.

[0064] The python code for the image crawler is as follows:

[0065]

[0066] from bs4 import BeautifulSoup

[0067] import os

[0068] def download_images(url,folder_path):

[0069] page = requests.get(url)

[0070] soup=BeautifulSoup(page.content,'html.parser')

[0071] img_tags=soup.find_all('img')

[0072] if not os.path.exists(folder_path):

[0073] os.makedirs(folder_path)

[0074] for img in img_tags:

[0075] img_url = img['src']

[0076] img_data=requests.get(img_url).content

[0077] img_name=os.path.join(folder_path,img_url.split(' / ')[-1])

[0078] with open(img_name,'wb')as handler:

[0079] handler.write(img_data)

[0080] download_images('https: / / example.cn','downloaded_images')

[0081] To process the image, the python code is as follows:

[0082] from PIL import Image

[0083] import os

[0084] def process_images(folder_path):

[0085] for filename in os.listdir(folder_path):

[0086] if filename.endswith(".jpg")or filename.endswith(".png"):

[0087] img=Image.open(os.path.join(folder_path,filename))

[0088] img = img.convert("RGB")

[0089] img = img.resize((256,256))

[0090] img.save(os.path.join(folder_path,filename))

[0091] process_images('downloaded_images')

[0092] For video modality data, it can be collected in the following ways:

[0093] Use the video APIs provided by platforms such as Douyin API and Kuaishou API to collect data, use web crawler technology to collect video links and metadata on web pages, transcode and edit videos, and extract key frames and audio content.

[0094] The Python code for calling the video API is as follows:

[0095] import requests

[0096] api_key="YOUR_API_KEY"

[0097] url=f"https: / / open.douyin.com / video / search / ?access_token={api_key}&keyword=example"

[0098] response = requests.get(url)

[0099]

[0100] The video is then processed, the python code is as follows:

[0101]

[0102] Audio modal data can be collected in the following ways:

[0103] Use the audio API provided by platforms such as Himalaya API to collect data, use web crawler technology to collect audio links and metadata from web pages, convert audio formats and perform noise reduction processing to improve audio quality.

[0104] The Python code for calling the audio API is as follows:

[0105]

[0106] Then process the audio, the python code is as follows:

[0107]

[0108] process_audio('example_audio.mp3','processed_audio.wav')

[0109] Through the algorithms and formulas described in detail above, this embodiment can efficiently collect multimodal data from domestic social media and other online platforms, and preprocess and store the data, providing a reliable data basis for subsequent multimodal feature extraction and rumor detection. During the data collection process, the data is preprocessed to ensure data quality and consistency. The data is cleaned to remove duplicate, invalid and noisy data to ensure data accuracy, and the data format is converted to convert data of different formats into a unified standard format for subsequent processing.

[0110] Data quality control is an important part of data collection, ensuring that the collected data is of high quality and reliability. Check whether the collected data is complete, including the integrity of text, images, videos and audio content.

[0111] Use data verification technology to check whether the collected data is accurate. The data verification technology is described in detail. The specific steps are as follows:

[0112] Data integrity check:

[0113] Checksum: Calculate a checksum (such as using SHA-256) for each data file or data block and compare it with the checksum of the original data. This can detect whether the data has been changed or damaged during the collection or storage process.

[0114] Data integrity check: Set integrity checkpoints for data acquisition and storage systems to periodically verify data integrity. For example, when storing image data, check whether each image can be loaded and displayed correctly.

[0115] Verification of data accuracy:

[0116] Data format verification: Ensure that the data complies with the predetermined format standards. For text data, check whether it complies with the grammatical rules; for images, verify whether it complies with the expected resolution and file type.

[0117] Data source verification: Verify that the data comes from a reliable source. For example, during the collection of image and video data, ensure that the collection equipment is working properly and that the metadata of the image or video (such as shooting time and location) is consistent with expectations.

[0118] Data consistency check (for specific scenarios):

[0119] Cross-modal consistency check: Although direct consistency check of data from different modalities may not always make sense, it can be verified in specific scenarios, for example, whether the key information in the text description can find corresponding features in related images or videos.

[0120] Automated matching tools: Use automated tools (such as image recognition and natural language processing models) to perform preliminary matching of data to check the relevance of text descriptions to image content. This can help detect potential errors or inconsistencies, but should not be used as the sole means of verification.

[0121] Data quality monitoring:

[0122] Anomaly detection: Use machine learning algorithms to detect anomalies in data and identify possible errors or irregular data. For example, detect abnormal characters or format errors in text data, or identify abnormal pixels in images.

[0123] Data review: Schedule regular manual reviews, especially when the data is large or critical. Manual reviews can ensure that the data meets quality standards and make any necessary corrections.

[0124] Data verification report:

[0125] Generate verification report: After data collection and processing, generate a detailed verification report, including verification and results, data format verification results, source verification records, etc. The report should clearly identify any problems found and their resolution status.

[0126] That is, in one embodiment, the collected data is verified; the verification step includes at least one of the following: calculating a checksum for each collected data, and determining whether the checksum changes during the data transmission process; if it changes, determining that the data verification fails; checking whether the data is complete at a preset checkpoint; if the data is incomplete, determining that the data verification fails; verifying whether the format of the data is correct, if the format of the data is incorrect, determining that the data verification fails; verifying whether the data source of the data is a secure data source, if the data source of the data is not a secure data source, determining that the data verification fails; using a preset algorithm to perform anomaly detection on the data, if the detection result is abnormal, determining that the data verification fails.

[0127] 2. Multimodal feature extraction module

[0128] The multimodal feature extraction module uses a pre-trained model to extract features from each modality and fuses them through a multimodal attention mechanism. The core of this module is to comprehensively utilize multimodal information such as text, images, videos, and audio, dynamically adjust the weights of different modalities, and enable the system to adaptively highlight the most relevant modal information in different situations. The feature extraction of each modality relies on a pre-trained model to obtain high-quality feature representation.

[0129] The text is encoded using a pre-trained language model (such as RoBERTa), wherein this embodiment also adds an Emoji sentiment analysis module (i.e., an expression sentiment analysis module) to analyze the expressions appearing in the text to obtain a fusion vector with more features, thereby improving the detection capability of the model.

[0130] Here, this embodiment adopts a soft label method to learn the direct relationship between expression Emoji and target emotion from training data, construct an Emoji emotion distribution knowledge base (i.e., expression emotion distribution knowledge base), and combine the Emoji emotion distribution information with the text semantic information for multi-label emotion recognition. The semantic information module uses the RoBERTa model (i.e., the preset language model) to extract the semantic information vector in the text; the Emoji information module uses the Emoji emotion distribution knowledge base to extract the Emoji emotion distribution vector (i.e., expression emotion distribution vector) of the sentence.

[0131] Here, the semantic information vector is extracted as follows:

[0132] First, a single sentence in the dataset is taken as input; then, each word is converted into a word vector using the pre-trained RoBERTa model; finally, these word vectors are converted into the hidden state representation of the sentence, i.e., the semantic information vector, through the RoBERTa encoder.

[0133] Input: Given sentence X i , using the pre-trained RoBERTa model, sentence X i Each word in is converted into a word vector to get the embedding E of the i-th sentence i ={e1,e2,…,e n}.

[0134] Output: E i Input the pre-trained RoBERTa encoder to get sentence X i The final hidden state of is represented by C, where C is the vector representation of the special tag [CLS], which contains the semantic information of the entire sentence. This paper takes C as the output of the semantic information module, that is, the semantic information vector h i :

[0135] h i =Encoder([cls]+X i +[sep])

[0136] Among them, [cls] and [sep] are special tags of the RoBERTa model.

[0137] In addition, the expression emotion distribution vector is extracted as follows:

[0138] First, we built an Emoji emotion distribution knowledge base. This knowledge base is based on the soft label method and learns the Emoji emotion distribution representation directly associated with the target emotion from the training data. The specific steps are as follows:

[0139] Define Emoji vector representation:

[0140] Given a specific Emoji e j , we define its vector representation as g j The purpose of this vector representation is to capture the characteristics and expressiveness of Emoji in different emotional dimensions.

[0141] Use the sample x containing the Emoji to calculate the sample's embedding representation V(x) and probability density p(x), where Ω is the integral domain of the sample, D(g j , V(x)) is g j and V(x). In order to obtain a better vector representation g j, it is necessary to minimize the expected value of the distance function D over the integral domain Ω.

[0142] So Emoji e j The vector representation calculation formula is

[0143]

[0144] In the specific implementation, since the data set is usually discrete, the above integral is simplified to the summation form:

[0145]

[0146] Among them, T r The dataset contains Emoji e j For all sample sets, the distance metric uses the square form of Euclidean distance, which simplifies the optimization problem and makes it easier to solve:

[0147]

[0148] By solving the above optimization problem, we get Emoji e j The vector representation of g j , which is actually the dataset T r All Emoji e j The average value of the sample vector V(x):

[0149]

[0150] Here |·| represents the size of the set, ensuring that the vector g j is the average of all relevant sample vectors.

[0151] Furthermore, the concept of soft labels is introduced, that is, the output (soft labels) of the emotion prediction model is used to enhance the vector representation of Emoji. j Each dimension not only reflects the characteristics of Emoji itself, but is also associated with a specific emotional label:

[0152]

[0153] Among them, V soft(x) is the soft label vector of sample x, which is obtained through the pre-trained BERT model and reflects the score or probability of sample x on each sentiment label.

[0154] Finally, the vector representation g of each Emoji calculated by the above method is jIn summary, an Emoji emotion distribution knowledge base is constructed. Each Emoji vector in this knowledge base is directly associated with an emotion, providing rich emotion information for subsequent emotion recognition tasks. Through this process, the Emoji information module links Emoji with the distribution of emotion labels, greatly enriching the model's emotion recognition ability when processing text containing Emoji.

[0155] After building the Emoji sentiment distribution knowledge base, use the knowledge base to extract the Emoji sentiment distribution vector from the sentence. The specific steps are as follows:

[0156] For each Emoji e in the sentence j , obtain its sentiment distribution vector g from the sentiment distribution knowledge base j . If there is only one Emoji in the sentence, use the Emoji sentiment distribution knowledge base to find the sentiment feature distribution vector represented by this Emoji, and use this feature vector as the output of the Emoji information module. If the sentence contains multiple Emojis, use the Emoji sentiment distribution knowledge base to extract the sentiment feature distribution vector of each Emoji, and then average these features to obtain a comprehensive sentiment distribution vector, which is used as the output of the Emoji information module. If there is no Emoji in the sentence, the output of the Emoji information module is 0, indicating that there is no Emoji-related sentiment information. Then, take the weighted average of the sentiment distribution vectors of all Emojis to obtain the Emoji sentiment distribution vector G of the sentence. i .

[0157] Emoji emotion distribution vector G i and the semantic information vector h i Combine to generate a high-dimensional vector representation: T = G i ⊕h i , where T is the semantic information vector h i and Emoji emotion distribution vector G i A high-dimensional vector representation of .

[0158] The above is the modal feature extraction method for text modal data. The modal feature extraction methods for image modal data, video modal data and audio modal data are as follows:

[0159] Use a pre-trained convolutional neural network (such as ResNet) to encode the image and generate a high-dimensional vector representation: I = ResNet (Image), where I is a high-dimensional vector representation of image features.

[0160] Use a pre-trained 3D convolutional neural network (such as C3D) to encode the video and generate a high-dimensional vector representation: V = C3D (Video), where V is a high-dimensional vector representation of video features.

[0161] Use a pre-trained audio model (such as VGGish) to encode the audio and generate a high-dimensional vector representation: A = VGGish (Audio), where A is a high-dimensional vector representation of audio features.

[0162] The above content is a detailed extraction method for text modal data, image modal data, video modal data and audio modal data. That is, in one embodiment, for text modal data, a preset language model is used to extract semantic information vectors in the text modal data, and a preset expression and emotion distribution knowledge base is used to extract expression and emotion distribution vectors in the text modal data; the semantic information vector and the expression and emotion distribution vector are fused to obtain the modal features of the text modal data; the expression and emotion distribution knowledge base contains the mapping relationship between expression and emotion; for image modal data, a preset convolutional neural network is used to extract features to obtain the modal features of the image modal data; for video modal data, a preset three-dimensional convolutional neural network is used to extract features to obtain the modal features of the video modal data; for audio modal data, a preset audio model is used to extract features to obtain the modal features of the audio modal data.

[0163] For the above-mentioned text modal data, image modal data, video modal data, and audio modal data, this embodiment uses a multimodal attention mechanism to fuse features from different modalities, and dynamically adjusts the weights of different modalities to enable the system to adaptively highlight the most relevant modal information in different situations. Specifically, this embodiment designs a multi-head attention mechanism to calculate the weights of each modal feature. First, the feature vector of each modality is linearly transformed to obtain the query vector Q, key vector K, and value vector V:

[0164]

[0165]

[0166]

[0167] Where W Q , W K and W V is the linear transformation matrix.

[0168] Then, the scaled dot product attention mechanism is used to calculate the attention score of each modality feature:

[0169]

[0170]

[0171]

[0172] Among them, d k is the dimension of the key vector.

[0173] By weighted summation, the attention scores of different modalities are fused to obtain the final multimodal feature representation Among them, α T , α I , α V and α A are the weights of different modes, satisfying α T +α I +α V +α A =1.

[0174] That is, in one embodiment, the modal features of all modal data are fused using a multimodal attention mechanism to obtain a multimodal fusion feature, including: performing a linear transformation on the modal features of each modal data to obtain a query vector, a key vector, and a value vector of each modal data; based on the query vector, the key vector, and the value vector of each modal data, using a scaled dot product attention mechanism to calculate an attention score of each modal data; performing a weighted summation on the attention score of each modal data to obtain a multimodal fusion feature;

[0175] The following calculation formula is used to perform weighted summation on the attention score of each modal data:

[0176]

[0177] Among them, F is the multimodal fusion feature, and are the attention scores of text modality data, image modality data, video modality data, and audio modality data, respectively, T , α I , α V and α A are the weights of text modality data, image modality data, video modality data, and audio modality data, respectively, T +α I +α V +α A =1, d k is the dimension of the key vector, W Q, W K and W V is a linear transformation matrix, T, I, V, and A are the modal features of text modality data, image modality data, video modality data, and audio modality data, respectively.

[0178] 3. Multi-layer graph convolutional network module

[0179] In the graph neural network module, we first need to build a propagation graph structure for rumor propagation. The propagation graph G = (V, E) consists of the following elements:

[0180] Node V: Each node represents an event statement or a response post related to it, and the obtained multimodal feature representation F is grouped into nodes.

[0181] Edge E: Edges represent the response relationship between nodes, that is, a post responds to or forwards another post. Each edge (u, v) represents the propagation path from node u to node v, and the weight of the edge can represent the response strength or propagation frequency.

[0182] Ensure that the node features have the same origin:

[0183] Node definition: Each node represents a specific post or event announcement. Make sure all modal features (text, image, audio, video, sentiment features) are from the same post. In other words, the features of a specific node should integrate all relevant data of that node.

[0184] Example: If node A is a specific post, then the text features of node A should come from the content of the post, the image features should come from the pictures in the post, the audio features (if any) should come from the audio in the post, and so on.

[0185] Definition of edge:

[0186] Response relationship: The edge (u, v) represents the communication relationship from node u to node v, that is, the response or forwarding of one post to another. The weight of the edge can represent the response strength or the communication frequency.

[0187] Graph convolutional networks are used to process propagation graphs and capture the relationships between nodes. The main steps include:

[0188] First, transform the edge E into a symmetric adjacency matrix A∈{0,1} |V|×|V| , where if x i With x j With a response relationship or i = j, then A i,j =1, otherwise A i,j = 0 Then we use the layer-by-layer propagation rule to update the node vector of the lth layer:

[0189]

[0190] in, is the symmetric normalized adjacency matrix, and D represents the degree matrix of A. is a layer-specific trainable transformation matrix. (l) is the node feature matrix of the lth layer, H (l+1) is the output feature matrix of the l+1th layer. Specifically, we adopt residual connections around the convolutional layers of each graph to fuse multi-scale information from the statement semantic scale and the event structure scale to obtain a refined representation:

[0191]

[0192] in, is the hidden representation declared at the (l-1)th layer, and ∥ is the concatenation operation with the broadcast mechanism.

[0193] For a GCN model with l layers, we obtain the final node representation And the feature representation reflected in the propagation thread is jointly captured through mean pooling:

[0194]

[0195] in, is the event-level structure representation of the entire propagation thread, d (l) is the output dimension of GCN.

[0196] Use event-level features to detect rumors. Use classifiers to determine whether an event is a rumor:

[0197]

[0198] in, is the prediction result of rumor detection.

[0199] 4. Domain Adaptive Contrastive Learning Module

[0200] Domain adaptive contrastive learning helps rumor detection models adapt to different application scenarios, such as the transfer from social media to news websites. An effective method is to use domain adaptation: given a source domain Ds and a target domain Dt, by designing a domain adaptive contrastive learning mechanism, the model aligns the representations of instances with the same authenticity (for example, both are rumors or non-rumors) in the source and target domains as much as possible, while distinguishing instances of different authenticity. This is achieved through a contrast loss between the source and target domains:

[0201]

[0202] Among them, N t Represents the number of samples in the target domain data. Represents the category label of the target sample i The number of samples with the same source field. and Represent the category labels of target sample i and source sample j respectively. If the category labels of the two are the same, that is, both are rumors or non-rumors, then the indicator function The value of is 1, otherwise it is 0. The function of this formula is to achieve feature alignment between fields by minimizing intra-class differences and maximizing inter-class differences, thereby enhancing the generalization ability of the model in low-resource fields. s Represents the number of samples in the source domain. It is used to calculate all samples in the source domain and helps the model achieve normalization in the feature representation of the source domain during contrastive learning. The event-level feature representation of the i-th sample in the target domain is obtained by encoding the semantics and propagation structure of the i-th sample in the target domain (such as a social media post or a group of related posts) through the model. It represents the feature representation of samples in the source domain that have the same category as sample iii in the target domain, by calculating The model measures the similarity between target sample i and source sample j. It represents the feature representation of samples in the source domain that are not necessarily the same as sample i in the target domain. In the normalization term, the model calculates the similarity between the target sample i and all samples in the source domain. This allows for a more comprehensive comparison.

[0203] 5. Target contrast learning module

[0204] Target contrastive learning is to solve the problem that when domain adaptive contrastive learning aligns the feature representations of the source domain and the target domain, the features in the target domain may be over-concentrated or degraded. This degradation may suppress the diversity of the features in the target domain, resulting in the limited generalization ability of the model in the target domain. This method ensures that the model can maintain strong generalization ability even when there are fewer target data samples through more fine-grained learning. Specifically, the module enhances the discriminative ability of the model by introducing a data enhancement strategy of multimodal hard negative samples. A common method is to construct pairs of samples of the same category and different categories: same category sample pairs: sample pairs from the same category, different category sample pairs: sample pairs from different categories. In target contrastive learning, especially adding data enhancement strategies to domain adaptive and contrastive learning modules, the performance of rumor detection models in low-resource environments can be significantly improved. Through data enhancement technology, the target dataset is expanded to ensure the uniformity of the target feature space, so that the target feature space retains as many features as possible, thereby improving the representation and generalization ability of the model. Specifically, data augmentation includes generating new samples, such as synonym replacement of text, rotation and scaling of images, and volume adjustment of audio, so as to make the feature distribution of the target dataset more uniform and reduce the distribution differences between samples in the target domain. This not only helps the domain adaptation model better adapt to the feature distribution of the target domain, but also enhances the discriminative ability in contrastive learning.

[0205] On the enhanced target event data, target contrast learning is applied to improve the uniformity and resolution of target domain features by pairing different enhanced views of the same event into positive sample pairs. The target contrast loss function is as follows:

[0206]

[0207] Among them, N t The number of samples in the target domain, similar to the same parameter in domain adaptive contrastive learning, is used to normalize the loss. Represents the enhanced representation of the target sample iii, that is, another view of the target sample generated by the data augmentation strategy. By comparing the original sample With enhanced samples The similarity between them can help the model learn more robust feature representations. This is a normalization term that contains the sum of the similarities between the target sample and all other samples (both original and enhanced). This term ensures that the model can distinguish between different target samples, preventing the representations of different events from being too close in the feature space, thereby enhancing the discriminability of features.

[0208] Using source domain D s and the target domain D tThe cross entropy of the training data and the contrast target jointly train the model:

[0209]

[0210] in, is the classification loss (Cross-Entropy Loss), which is used to calculate the difference between the model prediction and the true label to ensure the basic classification ability of the model. It is a domain adaptive contrast loss, which is mainly used for feature alignment between the source domain and the target domain. is the target contrast loss, which ensures that the event representation within the target domain can be evenly distributed and retains more domain-specific information. α is a trade-off parameter.

[0211] In this way, this embodiment can use features extracted from resource-rich data (such as features learned from news websites and social media) to provide high-quality initial feature representation for data in low-resource environments. These features help the model to perform more accurate rumor detection in the target domain.

[0212] Through domain adaptation and feature transfer, the knowledge of the pre-trained model is transferred to resource-scarce environments. Even with limited data, the model can still utilize the features and patterns learned during pre-training to maintain high detection performance.

[0213] In summary, from the above content, it can be known that the specific content of this embodiment is as follows:

[0214] 1. Data collection and preprocessing

[0215] The data collection module collects multimodal data from social media and online platforms, including text, images, videos, and audio. Each data type is processed through different collection techniques (API, crawlers, etc.) and processing steps (cleaning, format conversion, etc.) to ensure the quality and consistency of the data.

[0216] Text data collection: collected through API or web crawlers, filtered using keywords and data cleansing.

[0217] Image data collection: collected through image API or crawler technology, and format converted and compressed.

[0218] Video data collection: Use video API or crawler technology to collect videos and extract key frames and audio content.

[0219] Audio data collection: collected through audio API or crawler technology, and format conversion and noise reduction processing are performed.

[0220] 2. Feature extraction and fusion

[0221] The multimodal feature extraction module uses pre-trained models to extract features from text, images, videos, and audio:

[0222] Text features: Use pre-trained language models such as RoBERTa to generate high-dimensional vector representations.

[0223] Image features: Use pre-trained convolutional neural networks such as ResNet to generate high-dimensional vector representations.

[0224] Video features: Use pre-trained 3D convolutional networks such as C3D to generate high-dimensional vector representations.

[0225] Audio features: Use pre-trained audio models such as VGGish to generate high-dimensional vector representations.

[0226] Multimodal fusion integrates the features of each modality through a multi-head attention mechanism to dynamically adjust the weights of different modalities and generate a comprehensive feature representation F:

[0227] Among them, attention represents the attention score of each modality, α T , α I , α V and α A are the weights of different modes and satisfy α T +α I +

[0228] α V +α A =1.

[0229] 3. Graph Convolutional Network (GCN)

[0230] The graph neural network module uses the propagation graph structure to capture the structural characteristics of rumor propagation:

[0231] Node feature integration: Integrate multimodal features (text, image, audio, video, emotion) into node features h for each post v .

[0232] Propagation graph construction: nodes represent event declarations and related response posts, and edges represent response relationships between nodes.

[0233] Graph convolution operation: The propagation graph is processed through a multi-layer graph convolutional network (MGCN) to generate event-level feature representation o.

[0234] 4. Domain Adaptive Contrastive Learning Module

[0235] The domain-adaptive contrastive learning module uses supervised pre-training methods and cross-domain transfer learning methods to help the model adapt to low-resource scenarios:

[0236] Supervised pre-training: The encoder is pre-trained on labeled data to obtain feature representation.

[0237] Cross-domain transfer learning: Use domain adaptation technology to align the feature representations of samples with the same authenticity (such as rumors or non-rumors) in the source and target domains, making them closer in the feature space, and ensuring that samples of different categories (rumors and non-rumors) maintain a large distance in the feature space, thereby enhancing the model's ability to distinguish.

[0238] 5. Target contrast learning module

[0239] The target contrast learning module uses data enhancement technology to mine and enhance the discriminative ability of the model:

[0240] Sample pair construction: Through data enhancement strategies, a diverse sample view of the target domain is generated to ensure that the event-level feature representation in the target domain is more evenly distributed in the feature space. By comparing the enhanced views of different events, the model can better distinguish individual events in the target domain. This helps the model to classify and identify more accurately when dealing with complex or scarce data.

[0241] The beneficial effects of this embodiment are as follows:

[0242] 1. Improve the accuracy of rumor detection

[0243] By introducing the fusion and comprehensive analysis of multimodal data (text, images, videos, audio), rumors can be identified more comprehensively and accurately. The multimodal attention mechanism can dynamically adjust the information weights of different modalities, allowing the system to adaptively highlight the most relevant modal information in different situations and avoid misjudgments caused by a single modality.

[0244] 2. Capturing the structural characteristics of rumor propagation

[0245] By constructing a propagation graph structure and adopting a multi-layer graph convolutional network (MGCN), this embodiment can effectively capture the propagation path and propagation structure characteristics of rumors in social networks. By enhancing the propagation graph structure and designing a new graph neural network structure, it is possible to better understand the rumor propagation characteristics in multi-level propagation networks and improve the timeliness and accuracy of detection.

[0246] 3. Enhance the generalization ability of the model

[0247] By using the transfer contrastive learning module, self-supervised pre-training methods and cross-domain transfer learning methods, the model can learn feature representations from unlabeled data, which is suitable for rumor detection in low-resource scenarios. Through the domain adaptation method, the model can maintain high detection performance in different application scenarios, enhancing the generalization ability of the model.

[0248] 4. Improve the model’s discriminative ability

[0249] By introducing the transfer contrast learning module and adopting the data enhancement strategy, the model's discriminative ability is enhanced. During the training process, the same-category and different-category sample pairs are constructed to maximize the similarity of the same-category sample pairs and minimize the similarity of the different-category sample pairs, thereby optimizing the model and improving its robustness in practical applications.

[0250] 5. Continuously improve system performance

[0251] Through the user feedback module, we collect user reports and feedback information to continuously update and improve the rumor detection model. We use incremental learning or online learning strategies to timely adjust and optimize the model based on user feedback information to improve the robustness and accuracy of the system.

[0252] 6. Efficient data collection and preprocessing

[0253] Design an efficient data collection framework that can dynamically adapt to different data sources and content formats, and efficiently collect multimodal data from domestic social media and other online platforms. Preprocess and store data to ensure data quality and consistency, and provide a reliable data foundation for subsequent multimodal feature extraction and rumor detection.

[0254] 7. Adapt to various application scenarios

[0255] This embodiment is not only applicable to rumor detection on social media platforms, but can also be applied to various online platforms such as news media, forums, blogs, etc. Through the comprehensive analysis of multimodal data and the combination of multiple advanced algorithms, the system can adapt to a variety of application scenarios and improve the comprehensiveness and adaptability of rumor detection.

[0256] 8. Provide rich psychological and emotional analysis

[0257] The sentiment analysis module can not only detect rumors, but also analyze the sentiment and emotional fluctuations in the text and extract psychological characteristics. Including sentiment polarity analysis, emotional fluctuation detection and psychological feature extraction, the system can provide richer sentiment and psychological analysis while identifying rumors, providing data support for further research and intervention.

[0258] To sum up, this embodiment scheme brings significant beneficial effects through the combination of multiple innovative modules such as multimodal data fusion, multi-layer graph convolutional network, transfer contrast learning, etc., improves the accuracy and robustness of rumor detection, adapts to various application scenarios, and provides the society with an efficient and reliable rumor detection system.

[0259] The rumor detection method based on multimodal information in a low-resource environment provided by the embodiment of the present invention collects multimodal data of each post in the first field, and the multimodal data includes text modal data, image modal data, video modal data and audio modal data; for each modal data in the multimodal data, the corresponding preset model is used to extract features to obtain the modal features of each modal data; and the modal features of all modal data are fused using the multimodal attention mechanism to obtain multimodal fusion features; a propagation graph structure is constructed for all posts in the first field, and the propagation graph structure is convoluted using a graph convolutional network to obtain an event-level structural representation that characterizes the propagation relationship of each node in the propagation graph structure, and the rumor detection result of the post corresponding to each node is determined using the event-level structural representation of each node; each node in the propagation graph structure is each post in the first field; and the edge in the propagation graph structure is the propagation path between posts in the first field. The scheme provided by the present invention can effectively improve the accuracy of rumor detection in low-resource scenarios.

[0260] In order to implement the method of an embodiment of the present invention, an embodiment of the present invention also provides a rumor detection system based on multimodal information in a low-resource environment, comprising: a processor and a memory for storing a computer program that can be run on the processor; wherein, when the processor is used to run the computer program, the steps of any one of the above methods are executed.

[0261] The above-mentioned system provided in this embodiment belongs to the same concept as the above-mentioned method embodiment. The specific implementation process thereof is detailed in the method embodiment and will not be repeated here.

[0262] In order to implement the method of the embodiment of the present invention, the embodiment of the present invention also provides a computer program product, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps of the above method.

[0263] Based on the hardware implementation of the above program modules, and in order to implement the method of the embodiment of the present invention, the embodiment of the present invention also provides an electronic device (computer device). Specifically, in one embodiment, the computer device can be a terminal, and its internal structure diagram can be as follows: Figure 3As shown. The computer device includes a processor A01, a network interface A02, a display screen A04, an input device A05 and a memory (not shown in the figure) connected through a system bus. Among them, the processor A01 of the computer device is used to provide computing and control capabilities. The memory of the computer device includes an internal memory A03 and a non-volatile storage medium A06. The non-volatile storage medium A06 stores an operating system B01 and a computer program B02. The internal memory A03 provides an environment for the operation of the operating system B01 and the computer program B02 in the non-volatile storage medium A06. The network interface A02 of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor A01, the method of any one of the above embodiments is implemented. The display screen A04 of the computer device can be a liquid crystal display or an electronic ink display, and the input device A05 of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0264] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0265] The device provided by the embodiment of the present invention includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, the method of any one of the above embodiments is implemented.

[0266] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0267] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0268] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0269] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0270] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0271] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0272] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0273] It can be understood that the memory of the embodiment of the present invention can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM, SyncLink Dynamic Random Access Memory), and direct RAM bus random access memory (DRRAM, Direct Rambus Random Access Memory).The memories described in the embodiments of the present invention are intended to include, but are not limited to, these and any other suitable types of memories.

[0274] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0275] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A rumor detection method based on multimodal information in a low-resource environment, characterized in that: The method comprises: Collecting multimodal data of each post in the first domain, wherein the multimodal data includes text modal data, image modal data, video modal data, and audio modal data; For each modal data in the multimodal data, a corresponding preset model is used to perform feature extraction to obtain a modal feature of each modal data; and the modal features of all modal data are fused using a multimodal attention mechanism to obtain a multimodal fusion feature; A propagation graph structure is constructed for all posts in the first field, and a graph convolutional network is used to convolve the propagation graph structure to obtain an event-level structural representation that characterizes the propagation relationship of each node in the propagation graph structure, and the rumor detection result of the post corresponding to each node is determined using the event-level structural representation of each node; each node in the propagation graph structure is each post in the first field; and the edges in the propagation graph structure are propagation paths between posts in the first field.

2. The rumor detection method based on multimodal information in a low-resource environment according to claim 1 is characterized in that: Collect multimodal data for each post in the first domain, including: For each data source in the first domain, determine the data format of the data collected by each data source; the data format includes text, image, video and audio; According to the data format of the data collected by each data source, the data is collected using an interface corresponding to the data format.

3. The rumor detection method based on multimodal information in a low-resource environment according to claim 2 is characterized in that: After collecting data using an interface corresponding to the data format, the method further includes: Verify the collected data; the verification step includes at least one of the following: Calculating a checksum for each collected data, and determining whether the checksum changes during the data transmission process; if so, determining that the data checksum fails; At a preset checkpoint, checking whether the data is complete; if the data is incomplete, determining that the data verification fails; Verify whether the format of the data is correct; if the format of the data is incorrect, determine that the data verification fails; Verify whether the data source of the data is a secure data source, and if the data source of the data is not a secure data source, determine that the data verification fails; The data is detected for abnormality using a preset algorithm, and if the detection result is abnormal, the data verification is determined to have failed.

4. The rumor detection method based on multimodal information in a low-resource environment according to claim 3 is characterized in that: For each modal data in the multimodal data, feature extraction is performed using a corresponding preset model to obtain modal features of each modal data, including: For text modal data, a preset language model is used to extract semantic information vectors in the text modal data, and a preset expression emotion distribution knowledge base is used to extract expression emotion distribution vectors in the text modal data; the semantic information vectors and the expression emotion distribution vectors are fused to obtain modal features of the text modal data; the expression emotion distribution knowledge base contains a mapping relationship between expressions and emotions; For the image modal data, a preset convolutional neural network is used to perform feature extraction to obtain modal features of the image modal data; For the video modality data, a preset three-dimensional convolutional neural network is used to perform feature extraction to obtain modality features of the video modality data; For the audio modal data, feature extraction is performed using a preset audio model to obtain modal features of the audio modal data.

5. The rumor detection method based on multimodal information in a low-resource environment according to claim 4 is characterized in that: The semantic information vector and the expression emotion distribution vector are fused to obtain the modal features of the text modal data, including: The semantic information vector and the expression emotion distribution vector are fused using the following calculation formula: Among them, T is the modal feature of text modal data, G i is the expression emotion distribution vector, hi is the semantic information vector, G i is the expression emotion distribution vector g of all expressions in the text modal data sentence j The weighted average of T r is the data set, V soft(x) Is the expression X in the sentence i The soft label vector, h i =Encoder([cls]+X i +[sep]), [cls] and [sep] are special tags for the preset language model.

6. The rumor detection method based on multimodal information in a low-resource environment according to claim 5 is characterized in that: The multimodal attention mechanism is used to fuse the modal features of all modal data to obtain multimodal fusion features, including: Performing linear transformation on the modal features of each modal data to obtain a query vector, a key vector, and a value vector of each modal data; Based on the query vector, key vector and value vector of each modal data, calculate the attention score of each modal data using a scaled dot product attention mechanism; Performing weighted summation on the attention scores of each modal data to obtain a multimodal fusion feature; The following calculation formula is used to perform weighted summation on the attention score of each modal data: F=a T attention T +a I attention I +a V attention V +a A attention A expression(2) Among them, F is the multimodal fusion feature, attention T 、attention I 、attention V and attention A are the attention scores of text modality data, image modality data, video modality data, and audio modality data, respectively, T , α I , α V and α A are the weights of text modality data, image modality data, video modality data, and audio modality data, respectively, T +α I +α V +α A =1, d k is the dimension of the key vector, W Q , W K and W V is a linear transformation matrix, T, I, V, and A are the modal features of text modality data, image modality data, video modality data, and audio modality data, respectively.

7. The rumor detection method based on multimodal information in a low-resource environment according to claim 5 is characterized in that: Determining the rumor detection result of the post corresponding to each node by using the event-level structure representation of each node includes: The rumor detection result of the post corresponding to each node is determined by the following calculation formula using the event-level structure representation of each node: in, is the rumor detection result of each post, is the event-level structure representation of the entire propagation thread, d (l) is the output dimension of the graph neural network, is the hidden representation declared at the (l-1)th layer, ∥ is the concatenation operation with the broadcast mechanism, H (l) is the node feature matrix of the lth layer.

8. The rumor detection method based on multimodal information in a low-resource environment according to claim 6 is characterized in that: The method further comprises: Collect multimodal data of each post in the second domain; Using a domain adaptive contrastive learning mechanism, posts with the same authenticity in the first domain and the second domain are aligned, and posts with different authenticity are separated, so as to detect rumors in posts in the second domain; Wherein, during the alignment process, the contrast loss between the first domain and the second domain is: in, is the contrast loss between the first field and the second field, N t represents the number of samples in the second domain data, Represents the category label of sample i in the second field The number of samples of the same first field, and Represent the category labels of the second domain sample i and the first domain sample j, respectively, N s represents the number of samples in the first field data, represents the event-level feature representation of the i-th sample in the second domain, Represents the feature representation of samples in the first domain that have the same category as sample i in the second domain, represents the similarity between the second domain sample i and the first domain sample j, The feature representation of samples in the first domain that are not necessarily the same as sample i in the second domain, It represents the similarity between sample i in the second domain and all samples in the first domain.

9. A rumor detection system based on multimodal information in a low-resource environment, characterized in that: include: A processor and a memory for storing a computer program that can be run on the processor; wherein, when the processor is used to run the computer program, the steps of the method according to any one of claims 1 to 8 are executed.

10. A storage medium storing a computer program, wherein: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images