A rumor detection method and system based on multimodal information in a low-resource environment

Abstract

The present invention discloses a rumor detection method and system based on multimodal information in a low-resource environment. The method includes collecting multimodal data of each post in a first field; extracting features of each modal data in the multimodal data using a corresponding preset model to obtain modal features of each modal data; and fusing the modal features of all modal data using a multimodal attention mechanism to obtain multimodal fusion features; constructing a propagation graph structure for all posts in the first field, and convolving the propagation graph structure using a graph convolutional network to obtain an event-level structural representation that characterizes the propagation relationship of each node in the propagation graph structure, and determining the rumor detection result of the post corresponding to each node using the event-level structural representation of each node. The solution provided by the present invention can effectively improve the accuracy of rumor detection in low-resource scenarios.

Description

Technical Field

[0001] The present invention relates to the field of information processing technology, and in particular to a rumor detection method and system based on multimodal information in a low-resource environment. Background Art

[0002] Existing rumor detection systems are mainly based on single text processing and cannot effectively identify rumors that contain multimodal information (such as images, videos, and audio). With the popularity of social media and multimedia content, the application of multimodal information fusion in rumor detection is becoming increasingly important.

[0003] There are currently some studies on multimodal rumor detection systems that combine text and image features and use transfer learning and contrastive learning to improve detection results. Specifically, they can be divided into the following categories:

[0004] 1. Cross-modal contrastive learning: Research has shown that the cross-modal contrastive learning framework performs well in multimodal rumor detection. This approach improves the semantic consistency of image-text pairs by aligning image and text representations, and further optimizes the alignment accuracy through a cross-modal consistency learning task.

[0005] 2. Multimodal Fusion Network: Some models use self-supervised learning losses to align source post representations across different views, enabling better multimodal fusion. For example, multimodal feature-enhanced attention networks that use an attention mechanism to combine text, visual, and social graph features have achieved promising results in rumor detection.

[0006] 3. Weighted Cross-Modal Feature Aggregation: Another approach is the weighted cross-modal feature aggregation network, which combines highly correlated features from different modalities and obtains a unified representation in the same space. This method uses adversarial training to enhance the robustness of the model.

[0007] Although existing methods have achieved certain results in some aspects, they also have defects in some aspects, as follows:

[0008] Flaws of cross-modal contrastive learning: Single alignment mechanism: Existing cross-modal contrastive learning methods mainly rely on image-text alignment and lack processing of video and audio modalities. This limits the effectiveness of the system in dealing with rumors containing complex multimedia content. Insufficient hard negative sample mining: Although this method uses contrastive learning, it lacks an effective hard negative sample mining strategy, which may lead to insufficient discrimination ability of the model for difficult-to-distinguish samples. Insufficient real-time performance: This method mainly focuses on training and testing static datasets, fails to effectively process real-time data streams, and cannot detect and respond to emerging rumors in a timely manner.

[0009] The shortcomings of the multimodal feature enhanced attention network: Fixed modal weights: In this method, the weights of modal features are fixed during the training phase and cannot be dynamically adjusted to meet the needs of different situations. This may not perform well when processing dynamically changing information. Lack of sentiment and psychological analysis: The system mainly focuses on the enhancement of modal features and ignores the extraction of sentiment and psychological features, which may play an important role in the spread of rumors. Insufficient modeling of propagation structure: Although the attention mechanism is utilized, it is still insufficient in modeling the propagation paths and temporal dynamic features of social networks, and fails to fully utilize the information of the propagation network.

[0010] The drawbacks of the weighted cross-modal feature aggregation network include: Lack of consideration of the temporal dimension: This method does not take into account the temporal dimension of information propagation, but only performs weighted aggregation of static features, ignoring the temporal characteristics of rumor propagation. Insufficient domain adaptation: The system lacks domain adaptation technology, making it difficult to cope with the needs of rumor detection in different domains (such as different languages ​​and cultural backgrounds). Lack of a user feedback mechanism: Without a user feedback mechanism, the system cannot self-adjust and optimize based on user feedback in actual applications, resulting in poor robustness.

[0011] In summary, existing technologies still face many challenges in multimodal information fusion, cross-modal alignment, and real-time detection. Summary of the Invention

[0012] In order to solve the technical problem that the existing rumor detection system has low accuracy and robustness in low-resource scenarios, the embodiments of the present invention provide a rumor detection method and system based on multimodal information in a low-resource environment.

[0013] The technical solution of the embodiment of the present invention is achieved as follows:

[0014] An embodiment of the present invention provides a rumor detection method based on multimodal information in a low-resource environment, the method comprising: collecting multimodal data of each post in a first domain, the multimodal data comprising text modal data, image modal data, video modal data and audio modal data; performing feature extraction on each modal data in the multimodal data using a corresponding preset model to obtain modal features of each modal data; and fusing the modal features of all modal data using a multimodal attention mechanism to obtain multimodal fusion features; constructing a propagation graph structure for all posts in the first domain, and convolving the propagation graph structure using a graph convolutional network to obtain an event-level structural representation characterizing the propagation relationship of each node in the propagation graph structure, and determining the rumor detection result of the post corresponding to each node using the event-level structural representation of each node; each node in the propagation graph structure is each post in the first domain; and the edges in the propagation graph structure are propagation paths between posts in the first domain.

[0015] In one embodiment, collecting multimodal data of each post in the first field includes: for each data source in the first field, determining the data format of the data collected by each data source; the data format includes text, image, video and audio; according to the data format of the data collected by each data source, using an interface corresponding to the data format for collection.

[0016] In one embodiment, after collecting data using an interface corresponding to the data format, the method further includes: verifying the collected data; the verification step includes at least one of the following: calculating a checksum for each collected data, and determining whether the checksum changes during the data transmission process; if so, determining that the data verification fails; checking whether the data is complete at a preset checkpoint; if the data is incomplete, determining that the data verification fails; verifying whether the format of the data is correct, if the format of the data is incorrect, determining that the data verification fails; verifying whether the data source of the data is a secure data source, if the data source of the data is not a secure data source, determining that the data verification fails; performing anomaly detection on the data using a preset algorithm, if the detection result is abnormal, determining that the data verification fails.

[0017] In one embodiment, for each modal data in the multimodal data, feature extraction is performed using a corresponding preset model to obtain the modal features of each modal data, including: for text modal data, semantic information vectors in the text modal data are extracted using a preset language model, and expression and emotion distribution vectors in the text modal data are extracted using a preset expression and emotion distribution knowledge base; the semantic information vectors and the expression and emotion distribution vectors are fused to obtain the modal features of the text modal data; the expression and emotion distribution knowledge base contains a mapping relationship between expression and emotion; for image modal data, feature extraction is performed using a preset convolutional neural network to obtain the modal features of the image modal data; for video modal data, feature extraction is performed using a preset three-dimensional convolutional neural network to obtain the modal features of the video modal data; for audio modal data, feature extraction is performed using a preset audio model to obtain the modal features of the audio modal data.

[0018] In one embodiment, fusing the semantic information vector and the expression emotion distribution vector to obtain the modal features of the text modal data includes: fusing the semantic information vector and the expression emotion distribution vector using the following calculation formula:

[0019]

[0020] Among them, T is the modal feature of text modal data, G iis the expression emotion distribution vector, hi is the semantic information vector, G i is the expression emotion distribution vector g of all expressions in the text modal data sentence j The weighted average of T r is the dataset, V soft(x) Is the expression X in the sentence i The soft label vector, h i =Encoder([cls]+X i +[sep]), [cls] and [sep] are special tags for the preset language model.

[0021] In one embodiment, a multimodal attention mechanism is used to fuse the modal features of all modal data to obtain a multimodal fusion feature, including: performing a linear transformation on the modal features of each modal data to obtain a query vector, a key vector, and a value vector for each modal data; based on the query vector, the key vector, and the value vector of each modal data, using a scaled dot product attention mechanism to calculate an attention score for each modal data; performing a weighted summation on the attention score of each modal data to obtain a multimodal fusion feature; wherein the weighted summation of the attention score of each modal data is performed using the following calculation formula:

[0022]

[0023] Among them, F is the multimodal fusion feature, and are the attention scores of text modality data, image modality data, video modality data, and audio modality data, respectively, T , α I , α V and α A are the weights of text modality data, image modality data, video modality data, and audio modality data, respectively, T +α I +α V +α A =1, d k is the dimension of the key vector, W Q 、W K and W V is a linear transformation matrix, T, I, V, and A are the modal features of text modal data, image modal data, video modal data, and audio modal data, respectively.

[0024] In one embodiment, determining the rumor detection result of the post corresponding to each node using the event-level structural representation of each node includes: determining the rumor detection result of the post corresponding to each node using the event-level structural representation of each node by the following calculation formula:

[0025]

[0026] Among them, y is the rumor detection result of each post, It is the event-level structure representation of the entire propagation thread, d (l) is the output dimension of the graph neural network, is the hidden representation declared at the (l-1)th layer, ∥ is the concatenation operation with the broadcast mechanism, H (l) is the node feature matrix of the lth layer.

[0027] In one embodiment, the method further includes: collecting multimodal data of each post in the second domain; using a domain-adaptive contrastive learning mechanism to align posts with the same authenticity in the first domain and the second domain, and separate posts with different authenticity, so as to detect rumors in posts in the second domain; wherein, during the alignment process, the contrast loss between the first domain and the second domain is:

[0028]

[0029] in, is the contrast loss between the first field and the second field, N t represents the number of samples in the second domain data, Represents the category label of sample i in the second field The number of samples of the same first field, and Represent the category labels of target sample i and source sample j respectively, N s represents the number of samples in the first field data, represents the event-level feature representation of the i-th sample in the second domain, Represents the feature representation of the sample in the first domain that has the same category as the sample i in the second domain, represents the similarity between the second domain sample i and the first domain sample j, It represents the feature representation of samples in the first domain that are not necessarily the same as sample i in the second domain,

[0030] It represents the similarity between sample i in the second domain and all samples in the first domain.

[0031] An embodiment of the present invention also provides a rumor detection system based on multimodal information in a low-resource environment, comprising: a processor and a memory for storing a computer program that can be run on the processor; wherein, when the processor is used to run the computer program, it executes the steps of any of the above methods.

[0032] An embodiment of the present invention further provides a storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any of the above methods are implemented.

[0033] The embodiment of the present invention provides a rumor detection method and system based on multimodal information in a low-resource environment. The method collects multimodal data of each post in a first domain, wherein the multimodal data includes text modal data, image modal data, video modal data, and audio modal data. For each modal data in the multimodal data, a corresponding preset model is used to perform feature extraction to obtain the modal features of each modal data. The modal features of all modal data are fused using a multimodal attention mechanism to obtain multimodal fusion features. A propagation graph structure is constructed for all posts in the first domain, and a graph convolutional network is used to convolve the propagation graph structure to obtain an event-level structural representation that characterizes the propagation relationship of each node in the propagation graph structure. The rumor detection result of the post corresponding to each node is determined using the event-level structural representation of each node. Each node in the propagation graph structure is a post in the first domain. The edges in the propagation graph structure are propagation paths between posts in the first domain. The solution provided by the present invention can effectively improve the accuracy of rumor detection in low-resource scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 Schematic diagram of the process of a rumor detection method based on multimodal information in a low-resource environment according to an embodiment of the present invention;

[0035] Figure 2 This is a schematic diagram of the system flow of an embodiment of the present invention;

[0036] Figure 3 This is a diagram of the internal structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0037] The present invention will be described in further detail below with reference to the accompanying drawings and embodiments.

[0038] The embodiment of the present invention provides a rumor detection method based on multimodal information in a low-resource environment. Figure 1 As shown, the method includes:

[0039] Step 101: Collect multimodal data of each post in a first domain, wherein the multimodal data includes text modal data, image modal data, video modal data, and audio modal data;

[0040] Step 102: For each modal data in the multimodal data, perform feature extraction using a corresponding preset model to obtain a modal feature of each modal data; and fuse the modal features of all modal data using a multimodal attention mechanism to obtain a multimodal fusion feature;

[0041] Step 103: Construct a propagation graph structure for all posts in the first field, and use a graph convolutional network to convolve the propagation graph structure to obtain an event-level structural representation that characterizes the propagation relationship of each node in the propagation graph structure, and use the event-level structural representation of each node to determine the rumor detection result of the post corresponding to each node; each node in the propagation graph structure is each post in the first field; the edges in the propagation graph structure are propagation paths between posts in the first field.

[0042] This embodiment mainly focuses on the application of multimodal information fusion and graph neural networks in rumor detection.

[0043] Specifically, the solution of this embodiment can be applied to a rumor detection system, which may include a data acquisition module, a multimodal feature extraction module (including an expression and emotion analysis module), a multi-layer graph convolutional network module (including a domain-adaptive contrastive learning module and a target contrastive learning module) and a rumor detection module.

[0044] See also Figure 2 , the specific process based on the above modules can be:

[0045] 1. Data acquisition module

[0046] The data collection module is responsible for collecting multimodal data from domestic social media and other online platforms, including text modal data, image modal data, video modal data, and audio modal data. Data source identification and management are the first step in data collection, ensuring that the system can identify and manage different data sources. The specific steps are as follows: Data source identification: Determine the main data sources, such as Weibo, WeChat, Douyin, Zhihu, Toutiao and other domestic social media platforms. Data source interface: Design a corresponding API interface for each data source to ensure that the system can communicate with the data source through the API. Data source metadata management: Establish a data source metadata management system to record information such as access frequency, data format, content type, etc. of each data source.

[0047] Data collection: Use system-provided APIs for data collection, such as the Weibo API and WeChat Public Platform API. For data sources that don't provide APIs, use web crawler technology to collect data and filter relevant content based on predefined keywords to improve collection efficiency.

[0048] That is, in one embodiment, for each data source in the first field, the data format of the data collected by each data source is determined; the data format includes text, image, video and audio; according to the data format of the data collected by each data source, the interface corresponding to the data format is used for collection.

[0049] Design corresponding data collection strategies based on different data sources and content types to ensure the comprehensiveness and efficiency of data collection.

[0050] Specifically, text modality data can be collected in the following ways:

[0051] The Python code for the API call is as follows:

[0052] import requests

[0053] api_key="YOUR_API_KEY"

[0054] url=f"https: / / api.weibo.com / 2 / statuses / public_timeline.json?access_token={api_key}"

[0055] response = requests.get(url)

[0056] statuses=response.json()['statuses']

[0057] #Print the collected Weibo content

[0058] for status in statuses:

[0059] print(status['text'])

[0060] To filter keywords, the python code is as follows:

[0061]

[0062] Image modality data can be collected in the following ways:

[0063] Use the image API provided by the platform to collect data, use web crawler technology to collect image content on web pages, and convert and compress the collected images to facilitate subsequent processing.

[0064] The python code of the image crawler is as follows:

[0065]

[0066] from bs4 import BeautifulSoup

[0067] import os

[0068] def download_images(url,folder_path):

[0069] page = requests.get(url)

[0070] soup=BeautifulSoup(page.content,'html.parser')

[0071] img_tags=soup.find_all('img')

[0072] if not os.path.exists(folder_path):

[0073] os.makedirs(folder_path)

[0074] for img in img_tags:

[0075] img_url = img['src']

[0076] img_data=requests.get(img_url).content

[0077] img_name=os.path.join(folder_path,img_url.split(' / ')[-1])

[0078] with open(img_name,'wb')as handler:

[0079] handler.write(img_data)

[0080] download_images('https: / / example.cn','downloaded_images')

[0081] To process the image, the python code is as follows:

[0082] from PIL import Image

[0083] import os

[0084] def process_images(folder_path):

[0085] for filename in os.listdir(folder_path):

[0086] if filename.endswith(".jpg")or filename.endswith(".png"):

[0087] img=Image.open(os.path.join(folder_path,filename))

[0088] img = img.convert("RGB")

[0089] img = img.resize((256,256))

[0090] img.save(os.path.join(folder_path,filename))

[0091] process_images('downloaded_images')

[0092] Video modality data can be collected in the following ways:

[0093] Use the video APIs provided by platforms such as Douyin API and Kuaishou API to collect data, use web crawler technology to collect video links and metadata on web pages, transcode and edit videos, and extract key frames and audio content.

[0094] The Python code for calling the video API is as follows:

[0095] import requests

[0096] api_key="YOUR_API_KEY"

[0097] url=f"https: / / open.douyin.com / video / search / ?access_token={api_key}&keyword=example"

[0098] response = requests.get(url)

[0099]

[0100] The video is then processed, and the python code is as follows:

[0101]

[0102] Audio modal data can be collected in the following ways:

[0103] Use the audio API provided by platforms such as Himalaya API to collect data, use web crawler technology to collect audio links and metadata from web pages, convert audio formats and perform noise reduction processing to improve audio quality.

[0104] The Python code for calling the audio API is as follows:

[0105]

[0106] Then process the audio, the python code is as follows:

[0107]

[0108] process_audio('example_audio.mp3','processed_audio.wav')

[0109] Through the algorithms and formulas detailed above, this embodiment can efficiently collect multimodal data from domestic social media and other online platforms, preprocess and store the data, and provide a reliable data foundation for subsequent multimodal feature extraction and rumor detection. During the data collection process, the data is preprocessed to ensure data quality and consistency. The data is cleaned to remove duplicate, invalid, and noisy data to ensure data accuracy. The data format is converted to a unified standard format to facilitate subsequent processing.

[0110] Data quality control is a crucial step in data collection, ensuring the high quality and reliability of the collected data. Check the integrity of the collected data, including the integrity of text, images, video, and audio content.

[0111] Use data verification technology to check whether the collected data is accurate. The data verification technology is described in detail. The specific steps are as follows:

[0112] Data integrity check:

[0113] Checksum: A checksum (e.g., using SHA-256) is calculated for each data file or block and compared with the checksum of the original data. This can detect whether the data has been altered or corrupted during collection or storage.

[0114] Data integrity check: Set integrity checkpoints for data acquisition and storage systems to regularly verify data integrity. For example, when storing image data, check that each image can be loaded and displayed correctly.

[0115] Data Accuracy Verification:

[0116] Data format validation: Ensures that data conforms to predefined formatting standards. For text data, this includes checking for grammatical compliance; for images, this includes verifying the expected resolution and file type.

[0117] Data source verification: Verify that the data comes from a reliable source. For example, during the acquisition of image and video data, ensure that the acquisition equipment is working properly and that the metadata of the image or video (such as shooting time and location) is consistent with expectations.

[0118] Data consistency check (for specific scenarios):

[0119] Cross-modal consistency checking: Although direct consistency checking of data from different modalities may not always make sense, it can be verified in specific scenarios. For example, whether key information in a text description can be found in corresponding features in related images or videos.

[0120] Automated matching tools: Use automated tools (such as image recognition and natural language processing models) to perform preliminary matching on the data, checking the relevance of text descriptions to image content. This can help identify potential errors or inconsistencies, but should not be used as the sole means of verification.

[0121] Data quality monitoring:

[0122] Anomaly detection: Use machine learning algorithms to detect anomalies in data and identify possible errors or irregularities. For example, it can detect unusual characters or formatting errors in text data or identify unusual pixels in images.

[0123] Data auditing: Schedule regular manual reviews, especially when the data volume is large or critical. Manual reviews can ensure that the data meets quality standards and make any necessary corrections.

[0124] Data verification report:

[0125] Generate verification report: After data collection and processing, generate a detailed verification report, including verification and results, data format verification results, source verification records, etc. The report should clearly identify any issues found and their resolution status.

[0126] That is, in one embodiment, the collected data is verified; the verification step includes at least one of the following: calculating a checksum for each collected data, and determining whether the checksum changes during the data transmission process; if it changes, determining that the data verification fails; checking whether the data is complete at a preset checkpoint; if the data is incomplete, determining that the data verification fails; verifying whether the format of the data is correct, if the format of the data is incorrect, determining that the data verification fails; verifying whether the data source of the data is a secure data source, if the data source of the data is not a secure data source, determining that the data verification fails; using a preset algorithm to detect anomalies in the data, if the detection result is abnormal, determining that the data verification fails.

[0127] 2. Multimodal feature extraction module

[0128] The multimodal feature extraction module uses pre-trained models to extract features from each modality and fuses them through a multimodal attention mechanism. The core of this module is to comprehensively utilize multimodal information such as text, images, video, and audio, dynamically adjusting the weights of different modalities so that the system adaptively highlights the most relevant modal information in different contexts. Feature extraction for each modality relies on pre-trained models to obtain high-quality feature representations.

[0129] The text is encoded using a pre-trained language model (such as RoBERTa). This embodiment also adds an Emoji sentiment analysis module (i.e., an expression sentiment analysis module) to analyze the expressions appearing in the text to obtain a fusion vector with more features, thereby improving the detection capability of the model.

[0130] Here, this embodiment adopts a soft label method to learn the direct relationship between emoji and target emotions from training data, construct an emoji emotion distribution knowledge base (i.e., expression emotion distribution knowledge base), and combine the emoji emotion distribution information with text semantic information for multi-label emotion recognition. The semantic information module uses the RoBERTa model (i.e., the preset language model) to extract the semantic information vector in the text; the emoji information module uses the emoji emotion distribution knowledge base to extract the emoji emotion distribution vector (i.e., expression emotion distribution vector) of the sentence.

[0131] Here, the semantic information vector is extracted as follows:

[0132] First, a single sentence in the dataset is taken as input; then, the pre-trained RoBERTa model is used to convert each word into a word vector; finally, these word vectors are converted into the hidden state representation of the sentence, that is, the semantic information vector, through the RoBERTa encoder.

[0133] Input: Given sentence X i , using the pre-trained RoBERTa model, sentence X i Each word in is converted into a word vector to get the embedding E of the i-th sentence i ={e1,e2,…,e n}.

[0134] Output: E i Input the pre-trained RoBERTa encoder to get sentence X i The final hidden state representation C, where C is the vector representation of the special tag [CLS], contains the semantic information of the entire sentence. This paper takes C as the output of the semantic information module, that is, the semantic information vector h i :

[0135] h i =Encoder([cls]+X i +[sep])

[0136] Among them, [cls] and [sep] are special tags of the RoBERTa model.

[0137] In addition, the expression emotion distribution vector is extracted as follows:

[0138] First, we built an Emoji emotion distribution knowledge base. This knowledge base uses a soft labeling approach to learn Emoji emotion distribution representations directly associated with the target emotion from training data. The specific steps are as follows:

[0139] Define Emoji vector representation:

[0140] Given a specific Emoji e j , we define its vector representation as g j The purpose of this vector representation is to capture the characteristics and expressiveness of Emoji in different emotional dimensions.

[0141] Use the sample x containing the Emoji to calculate the sample's embedding representation V(x) and probability density p(x), where Ω is the integration domain of the sample, D(g j , V(x)) is g j and V(x). In order to obtain a better vector representation g j, it is necessary to minimize the expected value of the distance function D over the integral domain Ω.

[0142] So Emoji e j The vector representation calculation formula is

[0143]

[0144] In practical implementation, since the data set is usually discrete, the above integral is simplified to a summation form:

[0145]

[0146] Among them, T r The dataset contains Emoji e j For all sample sets, the distance metric uses the square form of Euclidean distance, which simplifies the optimization problem and makes it easier to solve:

[0147]

[0148] By solving the above optimization problem, we get Emoji e j The vector representation g j , which is actually the dataset T r All Emoji e j The average value of the sample vector V(x):

[0149]

[0150] Here |·| represents the size of the set, ensuring that the vector g j is the average of all relevant sample vectors.

[0151] Furthermore, the concept of soft labels is introduced, that is, the output of the emotion prediction model (soft labels) is used to enhance the vector representation of Emoji. j Each dimension not only reflects the characteristics of the Emoji itself, but is also associated with a specific emotional label:

[0152]

[0153] Among them, V soft(x) is the soft label vector of sample x, which is obtained through the pre-trained BERT model and reflects the score or probability of sample x on each emotion label.

[0154] Finally, the vector representation g of each Emoji calculated by the above method is jThis information is aggregated to form an Emoji emotion distribution knowledge base. Each Emoji vector in this knowledge base is directly associated with an emotion, providing rich emotional information for subsequent emotion recognition tasks. Through this process, the Emoji Information Module connects Emojis with the distribution of emotion labels, significantly enriching the model's emotion recognition capabilities when processing text containing Emojis.

[0155] After building the Emoji emotion distribution knowledge base, we use it to extract Emoji emotion distribution vectors from sentences. The specific steps are as follows:

[0156] For each Emoji e in the sentence j , obtain its sentiment distribution vector g from the sentiment distribution knowledge base j . If there is only one Emoji in the sentence, use the Emoji sentiment distribution knowledge base to find the sentiment feature distribution vector represented by this Emoji, and use this feature vector as the output result of the Emoji information module. If the sentence contains multiple Emojis, also use the Emoji sentiment distribution knowledge base to extract the sentiment feature distribution vector of each Emoji, and then average these features to obtain a comprehensive sentiment distribution vector, which is used as the output of the Emoji information module. If there is no Emoji in the sentence, the output of the Emoji information module is 0, indicating that there is no Emoji-related sentiment information. Then, take a weighted average of the sentiment distribution vectors of all Emojis to obtain the Emoji sentiment distribution vector G of the sentence. i .

[0157] Emoji emotion distribution vector G i and semantic information vector h i Combine to generate a high-dimensional vector representation: T = G i ⊕h i , where T is the fusion of semantic information vector h i and Emoji emotion distribution vector G i high-dimensional vector representation of .

[0158] The above is the modal feature extraction method for text modal data. The modal feature extraction methods for image modal data, video modal data, and audio modal data are as follows:

[0159] Use a pre-trained convolutional neural network (such as ResNet) to encode the image and generate a high-dimensional vector representation: I = ResNet(Image), where I is a high-dimensional vector representation of image features.

[0160] Use a pre-trained 3D convolutional neural network (such as C3D) to encode the video and generate a high-dimensional vector representation: V = C3D (Video), where V is a high-dimensional vector representation of video features.

[0161] Use a pre-trained audio model (such as VGGish) to encode the audio and generate a high-dimensional vector representation: A = VGGish (Audio), where A is a high-dimensional vector representation of the audio feature.

[0162] The above content is a detailed extraction method for text modal data, image modal data, video modal data and audio modal data. That is, in one embodiment, for text modal data, a preset language model is used to extract the semantic information vector in the text modal data, and a preset expression and emotion distribution knowledge base is used to extract the expression and emotion distribution vector in the text modal data; the semantic information vector and the expression and emotion distribution vector are fused to obtain the modal features of the text modal data; the expression and emotion distribution knowledge base contains the mapping relationship between expression and emotion; for image modal data, a preset convolutional neural network is used to perform feature extraction to obtain the modal features of the image modal data; for video modal data, a preset three-dimensional convolutional neural network is used to perform feature extraction to obtain the modal features of the video modal data; for audio modal data, a preset audio model is used to perform feature extraction to obtain the modal features of the audio modal data.

[0163] For the above-mentioned text modal data, image modal data, video modal data, and audio modal data, this embodiment uses a multimodal attention mechanism to fuse features from different modalities. By dynamically adjusting the weights of different modalities, the system adaptively highlights the most relevant modal information in different situations. Specifically, this embodiment designs a multi-head attention mechanism to calculate the weights of each modal feature. First, the feature vector of each modality is linearly transformed to obtain the query vector Q, key vector K, and value vector V:

[0164]

[0165]

[0166]

[0167] Where W Q 、W K and W V is the linear transformation matrix.

[0168] Then, we use the scaled dot product attention mechanism to calculate the attention score for each modality feature:

[0169]

[0170]

[0171]

[0172] Among them, d k is the dimension of the key vector.

[0173] By weighted summation, the attention scores of different modalities are fused to obtain the final multimodal feature representation Among them, α T , α I , α V and α A are the weights of different modes, satisfying α T +α I +α V +α A =1.

[0174] That is, in one embodiment, a multimodal attention mechanism is used to fuse the modal features of all modal data to obtain a multimodal fusion feature, including: performing a linear transformation on the modal features of each modal data to obtain a query vector, a key vector, and a value vector for each modal data; based on the query vector, key vector, and value vector of each modal data, using a scaled dot product attention mechanism to calculate an attention score for each modal data; performing a weighted summation on the attention score of each modal data to obtain a multimodal fusion feature;

[0175] The following formula is used to perform weighted summation on the attention score of each modal data:

[0176]

[0177] Among them, F is the multimodal fusion feature, and are the attention scores of text modality data, image modality data, video modality data, and audio modality data, respectively, T , α I , α V and α A are the weights of text modality data, image modality data, video modality data, and audio modality data, respectively, T +α I +α V +α A =1, d k is the dimension of the key vector, W Q、W K and W V is a linear transformation matrix, T, I, V, and A are the modal features of text modal data, image modal data, video modal data, and audio modal data, respectively.

[0178] 3. Multi-layer graph convolutional network module

[0179] In the graph neural network module, we first need to build a rumor propagation graph structure. The propagation graph G = (V, E) consists of the following elements:

[0180] Node V: Each node represents an event statement or a response post related to it, and the obtained multimodal feature representation F is grouped into nodes.

[0181] Edges E: Edges represent the response relationship between nodes, i.e., one post responds to or reposts another. Each edge (u, v) represents the propagation path from node u to node v. The edge weight can represent the response strength or propagation frequency.

[0182] Ensure that the node features have the same origin:

[0183] Node Definition: Each node represents a specific post or event announcement. Ensure that all modal features (text, image, audio, video, sentiment) come from the same post. In other words, the features of a specific node should integrate all relevant data for that node.

[0184] For example: if node A is a specific post, then the text features of node A should come from the content of the post, the image features should come from the pictures in the post, the audio features (if any) should come from the audio in the post, and so on.

[0185] Definition of edge:

[0186] Response relationship: An edge (u, v) represents the communication relationship from node u to node v, i.e., a post responding to or forwarding another post. The weight of the edge can represent the response strength or the frequency of communication.

[0187] Graph convolutional networks are used to process propagation graphs and capture the relationships between nodes. The main steps include:

[0188] First, transform the edge E into a symmetric adjacency matrix A∈{0,1} |V|×|V| , where if x i with x j With a response relationship or i=j, then A i,j =1, otherwise A i,j = 0 Then we use the layer-by-layer propagation rule to update the node vector of the lth layer:

[0189]

[0190] in, is the symmetric normalized adjacency matrix, and D represents the degree matrix of A. H is a layer-specific trainable transformation matrix. (l) is the node feature matrix of the lth layer, H (l+1) is the output feature matrix of the l+1th layer. Specifically, we employ residual connections around the convolutional layers of each graph to fuse multi-scale information from both the statement semantic scale and the event structure scale to obtain a refined representation:

[0191]

[0192] in, is the hidden representation declared at the (l-1)th layer, and ∥ is the concatenation operation with the broadcast mechanism.

[0193] For a GCN model with l layers, we obtain the final node representation And the feature representation reflected in the propagation thread is captured jointly through mean pooling:

[0194]

[0195] in, It is the event-level structure representation of the entire propagation thread, d (l) is the output dimension of GCN.

[0196] Use event-level features to detect rumors. Use a classifier to determine whether an event is a rumor:

[0197]

[0198] in, is the prediction result of rumor detection.

[0199] 4. Domain-adaptive contrastive learning module

[0200] Domain-adaptive contrastive learning helps rumor detection models adapt to different application scenarios, such as the transition from social media to news websites. An effective approach is to use domain adaptation: given a source domain Ds and a target domain Dt, by designing a domain-adaptive contrastive learning mechanism, the model aligns the representations of instances with the same authenticity (e.g., both are rumors or not rumors) in the source and target domains as closely as possible, while distinguishing instances of different authenticity. This is achieved through a contrastive loss between the source and target domains:

[0201]

[0202] Among them, N t Indicates the number of samples in the target domain data. Represents the category label of the target sample i The number of samples with the same source field. and Represent the category labels of target sample i and source sample j respectively. If the category labels of the two are the same, that is, both are rumors or non-rumors, then the indicator function The value of is 1, otherwise it is 0. The function of this formula is to achieve feature alignment between domains by minimizing intra-class differences and maximizing inter-class differences, thereby enhancing the generalization ability of the model in low-resource domains. s Indicates the number of samples in the source domain. It is used to calculate all samples in the source domain and helps the model achieve normalization in the feature representation of the source domain during contrastive learning. Represents the event-level feature representation of the i-th sample in the target domain. It is a vector representation obtained by encoding the semantics and propagation structure of the i-th sample in the target domain (such as a social media post or a group of related posts) through the model. It represents the feature representation of the sample in the source domain that has the same category as the target domain sample iii, by calculating The model measures the similarity between target sample i and source sample j, Represents the feature representation of samples in the source domain that are not necessarily the same as the target domain sample i. In the normalization term, the model calculates the similarity between the target sample i and all samples in the source domain. This allows for a more comprehensive comparison.

[0203] 5. Target contrast learning module

[0204] Target contrastive learning addresses the problem that domain adaptive contrastive learning, when aligning feature representations in the source and target domains, can lead to over-concentration or degradation of target domain features. This degradation can suppress the diversity of target domain features and limit the model's generalization ability in the target domain. This approach uses finer-grained learning to ensure that the model maintains strong generalization even when the target data sample size is small. Specifically, the module enhances the model's discriminative ability by introducing a data augmentation strategy using multimodal hard negative samples. A common approach is to construct same-category and different-category pairs of samples: same-category pairs are pairs of samples from the same category, and different-category pairs are pairs of samples from different categories. Incorporating data augmentation strategies into target contrastive learning, particularly in the domain adaptation and contrastive learning modules, can significantly improve the performance of rumor detection models in low-resource environments. By augmenting the target dataset with data augmentation techniques, the target feature space is expanded to ensure uniformity and retain as many features as possible, thereby improving the model's representation and generalization capabilities. Specifically, data augmentation involves generating new samples, such as replacing synonyms in text, rotating and scaling images, and adjusting the volume of audio. This makes the feature distribution of the target dataset more uniform and reduces the distribution differences between samples within the target domain. This not only helps the domain adaptation model better adapt to the feature distribution of the target domain but also enhances the discriminative ability of contrastive learning.

[0205] On the enhanced target event data, target contrast learning is applied to improve the uniformity and resolution of target domain features by pairing different enhanced views of the same event into positive sample pairs. The target contrast loss function is as follows:

[0206]

[0207] Among them, N t The number of samples in the target domain is similar to the same parameter in domain adaptive contrastive learning and is used to normalize the loss. Represents the enhanced representation of the target sample iii, that is, another view of the target sample generated by the data enhancement strategy. By comparing the original sample With enhanced samples The similarity between them can help the model learn more robust feature representations. This is a normalization term that contains the sum of the similarities between the target sample and all other samples (including original and enhanced samples). This term ensures that the model can distinguish different target samples and prevents the representations of different events from being too close in the feature space, thereby enhancing the discriminability of features.

[0208] Using the source domain D s and target domain D tThe cross entropy of the training data and the contrast target are used to train the model:

[0209]

[0210] in, is the classification loss (Cross-Entropy Loss), which is used to calculate the difference between the model prediction and the true label to ensure the basic classification ability of the model. It is a domain adaptive contrast loss, which is mainly used for feature alignment between the source domain and the target domain. It is the target contrast loss, which ensures that the event representation within the target domain can be evenly distributed and retains more domain-specific information. α is a trade-off parameter.

[0211] Through the above approach, this embodiment can leverage features extracted from rich data sources (e.g., features learned from news websites and social media) to provide high-quality initial feature representations for data in low-resource environments. These features help the model perform more accurate rumor detection in the target domain.

[0212] Through domain adaptation and feature transfer, the knowledge of pre-trained models is transferred to resource-scarce environments. Even with limited data, the model can still leverage the features and patterns learned during pre-training to maintain high detection performance.

[0213] In summary, from the above content, it can be seen that the specific content of this embodiment is as follows:

[0214] 1. Data Collection and Preprocessing

[0215] The data collection module collects multimodal data from social media and online platforms, including text, images, videos, and audio. Each data type is processed through different collection technologies (API, crawlers, etc.) and processing steps (cleaning, format conversion, etc.) to ensure data quality and consistency.

[0216] Text data collection: collected through API or web crawlers, filtered using keywords and data cleansing.

[0217] Image data collection: collected through image API or crawler technology, and format converted and compressed.

[0218] Video data collection: Use video APIs or crawler technology to collect videos and extract key frames and audio content.

[0219] Audio data collection: collected through audio API or crawler technology, and then format converted and noise reduced.

[0220] 2. Feature extraction and fusion

[0221] The multimodal feature extraction module uses pre-trained models to extract features from text, images, videos, and audio:

[0222] Text features: Use pre-trained language models such as RoBERTa to generate high-dimensional vector representations.

[0223] Image features: Use pre-trained convolutional neural networks such as ResNet to generate high-dimensional vector representations.

[0224] Video features: Use pre-trained 3D convolutional networks such as C3D to generate high-dimensional vector representations.

[0225] Audio features: Use pre-trained audio models such as VGGish to generate high-dimensional vector representations.

[0226] Multimodal fusion integrates the features of each modality through a multi-head attention mechanism to dynamically adjust the weights of different modalities and generate a comprehensive feature representation F:

[0227] Among them, atten ion represents the attention score of each modality, α T , α I , α V and α A are the weights of different modes, and satisfy α T +α I +

[0228] α V +α A =1.

[0229] 3. Graph Convolutional Network (GCN)

[0230] The graph neural network module uses the propagation graph structure to capture the structural characteristics of rumor propagation:

[0231] Node feature integration: Integrate multimodal features (text, image, audio, video, emotion) into node features h for each post v .

[0232] Communication graph construction: nodes represent event declarations and related response posts, and edges represent response relationships between nodes.

[0233] Graph convolution operation: The propagation graph is processed through a multi-layer graph convolutional network (MGCN) to generate event-level feature representation o.

[0234] 4. Domain Adaptive Contrastive Learning Module

[0235] The domain-adaptive contrastive learning module uses supervised pre-training and cross-domain transfer learning methods to help the model adapt to low-resource scenarios:

[0236] Supervised pre-training: The encoder is pre-trained on labeled data to obtain feature representations.

[0237] Cross-domain transfer learning: Using domain adaptation technology, the feature representations of samples with the same authenticity (for example, both are rumors or not rumors) in the source and target domains are aligned, making them closer in the feature space. This ensures that samples of different categories (rumors and non-rumors) maintain a large distance in the feature space, thereby enhancing the model's discriminative ability.

[0238] 5. Target contrast learning module

[0239] The target contrast learning module uses data augmentation technology to enhance the discriminative ability of the model:

[0240] Sample Pair Construction: Through data augmentation strategies, we generate diverse sample views of the target domain, ensuring that event-level feature representations within the target domain are more evenly distributed across the feature space. By comparing enhanced views of different events, the model can better distinguish individual events within the target domain. This helps the model achieve more accurate classification and recognition when dealing with complex or sparse data.

[0241] The beneficial effects of this embodiment are as follows:

[0242] 1. Improve the accuracy of rumor detection

[0243] By integrating and comprehensively analyzing multimodal data (text, images, video, and audio), rumors can be identified more comprehensively and accurately. The multimodal attention mechanism dynamically adjusts the weight of information from different modalities, enabling the system to adaptively highlight the most relevant modal information in different situations and avoid misjudgments caused by a single modality.

[0244] 2. Capturing the structural characteristics of rumor propagation

[0245] By constructing a propagation graph structure and employing a multi-layer graph convolutional network (MGCN), this embodiment effectively captures the propagation paths and structural characteristics of rumors in social networks. By enhancing the propagation graph structure and designing a new graph neural network architecture, we can better understand the characteristics of rumor propagation in multi-layered networks and improve the timeliness and accuracy of detection.

[0246] 3. Enhance the generalization ability of the model

[0247] Leveraging the transfer contrastive learning module, self-supervised pre-training, and cross-domain transfer learning, the model is able to learn feature representations from unlabeled data, making it suitable for rumor detection in low-resource scenarios. Through domain adaptation, the model maintains high detection performance across diverse application scenarios, enhancing its generalization capabilities.

[0248] 4. Improve the model’s discriminative ability

[0249] By introducing a transfer contrastive learning module and employing a data augmentation strategy, the model's discriminative capabilities are enhanced. During training, pairs of samples from the same and different categories are constructed, maximizing the similarity of pairs of samples from the same category while minimizing the similarity of pairs of samples from different categories. This optimizes the model and improves its robustness in practical applications.

[0250] 5. Continuously improve system performance

[0251] The user feedback module collects user reports and feedback to continuously update and improve the rumor detection model. Using incremental learning or online learning strategies, the model is adjusted and optimized based on user feedback to improve the robustness and accuracy of the system.

[0252] 6. Efficient data collection and preprocessing

[0253] Design an efficient data collection framework that can dynamically adapt to different data sources and content formats, efficiently collecting multimodal data from domestic social media and other online platforms. Preprocess and store the data to ensure data quality and consistency, providing a reliable data foundation for subsequent multimodal feature extraction and rumor detection.

[0254] 7. Adapt to various application scenarios

[0255] This embodiment is not only applicable to rumor detection on social media platforms, but can also be applied to various online platforms such as news media, forums, and blogs. By comprehensively analyzing multimodal data and combining multiple advanced algorithms, the system can adapt to various application scenarios and improve the comprehensiveness and adaptability of rumor detection.

[0256] 8. Provide rich psychological and emotional analysis

[0257] The sentiment analysis module not only detects rumors but also analyzes the sentiment and emotional fluctuations within text, extracting psychological characteristics. This includes sentiment polarity analysis, emotional fluctuation detection, and psychological feature extraction. This allows the system to not only identify rumors but also provide richer sentiment and psychological analysis, providing data support for further research and intervention.

[0258] In summary, this embodiment scheme brings significant beneficial effects through the combination of multiple innovative modules such as multimodal data fusion, multi-layer graph convolutional networks, and transfer contrastive learning. It improves the accuracy and robustness of rumor detection, adapts to various application scenarios, and provides society with an efficient and reliable rumor detection system.

[0259] The embodiment of the present invention provides a rumor detection method based on multimodal information in a low-resource environment. The method collects multimodal data of each post in a first domain, wherein the multimodal data includes text modal data, image modal data, video modal data, and audio modal data. For each modal data in the multimodal data, a corresponding preset model is used to perform feature extraction to obtain the modal features of each modal data. The modal features of all modal data are fused using a multimodal attention mechanism to obtain multimodal fusion features. A propagation graph structure is constructed for all posts in the first domain, and a graph convolutional network is used to convolve the propagation graph structure to obtain an event-level structural representation that characterizes the propagation relationship of each node in the propagation graph structure. The rumor detection result of the post corresponding to each node is determined using the event-level structural representation of each node. Each node in the propagation graph structure is a post in the first domain. The edges in the propagation graph structure are propagation paths between posts in the first domain. The solution provided by the present invention can effectively improve the accuracy of rumor detection in low-resource scenarios.

[0260] In order to implement the method of an embodiment of the present invention, an embodiment of the present invention also provides a rumor detection system based on multimodal information in a low-resource environment, including: a processor and a memory for storing a computer program that can be run on the processor; wherein, when the processor is used to run the computer program, it executes the steps of any of the above methods.

[0261] The above-mentioned system provided in this embodiment and the above-mentioned method embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0262] To implement the method of an embodiment of the present invention, an embodiment of the present invention further provides a computer program product. The computer program product includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of the above method.

[0263] Based on the hardware implementation of the above program modules, and in order to implement the method of the embodiment of the present invention, the embodiment of the present invention further provides an electronic device (computer device). Specifically, in one embodiment, the computer device can be a terminal, and its internal structure diagram can be as follows: Figure 3As shown. The computer device includes a processor A01, a network interface A02, a display screen A04, an input device A05 and a memory (not shown in the figure) connected via a system bus. Among them, the processor A01 of the computer device is used to provide computing and control capabilities. The memory of the computer device includes an internal memory A03 and a non-volatile storage medium A06. The non-volatile storage medium A06 stores an operating system B01 and a computer program B02. The internal memory A03 provides an environment for the operation of the operating system B01 and the computer program B02 in the non-volatile storage medium A06. The network interface A02 of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor A01, the method of any one of the above embodiments is implemented. The display screen A04 of the computer device can be a liquid crystal display or an electronic ink display screen, and the input device A05 of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad provided on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0264] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0265] The device provided by an embodiment of the present invention includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, the method of any one of the above embodiments is implemented.

[0266] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0267] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0268] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0269] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0270] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0271] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0272] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0273] It is understood that the memory of the embodiments of the present invention can be volatile memory or non-volatile memory, or can include both volatile and non-volatile memory. Among them, the non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disk, or compact disc read-only memory (CD-ROM); magnetic surface memory can be magnetic disk memory or tape memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus random access memory (DRRAM).The memories described in the embodiments of the present invention are intended to include, but are not limited to, these and any other suitable types of memories.

[0274] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0275] The above are merely embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A rumor detection method based on multimodal information in a low-resource environment, characterized by: The method comprises: Collecting multimodal data of each post in the first domain, wherein the multimodal data includes text modal data, image modal data, video modal data, and audio modal data; For each modal data in the multimodal data, feature extraction is performed using a corresponding preset model to obtain modal features of each modal data; and the modal features of all modal data are fused using a multimodal attention mechanism to obtain a multimodal fusion feature; A propagation graph structure is constructed for all posts in the first domain, and a graph convolutional network is used to convolve the propagation graph structure to obtain an event-level structural representation that characterizes the propagation relationship of each node in the propagation graph structure. The event-level structural representation of each node is used to determine the rumor detection result of the post corresponding to each node; each node in the propagation graph structure is each post in the first domain, and the data of each node is the multimodal fusion feature of each post in the first domain; the edges in the propagation graph structure are the propagation paths between posts in the first domain, that is, the response or forwarding of one post to another post; The method of extracting features of each modal data in the multimodal data using a corresponding preset model to obtain modal features of each modal data includes: For text modal data, a preset language model is used to perform feature extraction on the text in the text modal data to obtain a semantic information vector, and a preset expression and emotion distribution knowledge base is used to extract the expression in the text modal data to obtain an expression and emotion distribution vector; the semantic information vector and the expression and emotion distribution vector are fused to obtain the modal features of the text modal data; the expression and emotion distribution knowledge base contains a mapping relationship between expression and emotion; For the image modal data, a preset convolutional neural network is used to perform feature extraction to obtain modal features of the image modal data; For the video modality data, a preset three-dimensional convolutional neural network is used to perform feature extraction to obtain modal features of the video modality data; For the audio modal data, extract features using a preset audio model to obtain modal features of the audio modal data; The expression in the text modality data is extracted using a preset expression emotion distribution knowledge base to obtain an expression emotion distribution vector, including: For each expression in the sentence, obtain its expression emotion distribution vector from the expression emotion distribution knowledge base. If there is only one expression in the sentence, use the expression emotion distribution knowledge base to find the expression emotion distribution vector represented by this expression and use this expression emotion distribution vector as the output result. If the sentence contains multiple expressions, the expression emotion distribution knowledge base is also used to extract the expression emotion distribution vector of each expression, and these expression emotion distribution vectors are averaged to obtain a comprehensive expression emotion distribution vector, which is used as the output; If there is no expression in the sentence, the output is 0; The graph convolution network is used to convolve the propagation graph structure to obtain an event-level structural representation representing the propagation relationship of each node in the propagation graph structure, including: Transform the edge E in the graph convolutional network into a symmetric adjacency matrix A∈{0,1} |V|×|V| , where if x i with x j With a response relationship or i=j, then A i,j =1, otherwise A i,j =0; Use the layer-by-layer propagation rule to update the node vector of the lth layer: in, is the symmetric normalized adjacency matrix, D represents the degree matrix of A, is the trainable transformation matrix for a specific layer, H (l) is the node feature matrix of the lth layer, H (l+1) is the output feature matrix of the l+1th layer, for Specifically, residual connections are used around the convolutional layers of each graph to fuse multi-scale information from the statement semantic scale and event structure scale to obtain a refined representation: in, is the hidden representation of the (l-1)th layer declaration; For a GCN model with l layers, the final node representation is obtained And the feature representation reflected in the propagation thread is jointly captured through mean pooling: in, It is the event-level structure representation of the entire propagation thread, d (l) is the output dimension of GCN.

2. The rumor detection method based on multimodal information in a low-resource environment according to claim 1 is characterized in that: Collect multimodal data for each post in the first domain, including: For each data source in the first domain, determining a data format of data collected by each data source; the data format includes text, image, video, and audio; According to the data format of the data collected by each data source, the data is collected using an interface corresponding to the data format.

3. The rumor detection method based on multimodal information in a low-resource environment according to claim 2 is characterized in that: After collecting data using an interface corresponding to the data format, the method further includes: Verify the collected data; the verification step includes at least one of the following: Calculating a checksum for each piece of collected data, and determining whether the checksum changes during the data transmission process; if so, determining that the data checksum fails; At a preset checkpoint, checking whether the data is complete; if the data is incomplete, determining that the data verification fails; Verify whether the format of the data is correct; if the format of the data is incorrect, determine that the data verification fails; Verifying whether the data source of the data is a secure data source; if the data source of the data is not a secure data source, determining that the data verification fails; The data is checked for abnormalities using a preset algorithm. If the detection result is abnormal, the data verification is determined to have failed.

4. The rumor detection method based on multimodal information in a low-resource environment according to claim 1 is characterized in that: The semantic information vector and the expression emotion distribution vector are fused to obtain the modal features of the text modal data, including: The semantic information vector and the expression emotion distribution vector are fused using the following formula: T=G i ⊕h i Formula (1) Among them, T is the modal feature of text modal data, G i is the expression emotion distribution vector, hi is the semantic information vector, G i is the expression emotion distribution vector g of all expressions in the text modal data sentence j The weighted average of T r is the dataset, V soft(x) Is the expression X in the sentence i The soft label vector, h i =Encoder([cls]+X i +[sep]), [cls] and [sep] are special tags for the preset language model.

5. The rumor detection method based on multimodal information in a low-resource environment according to claim 4 is characterized in that: The multimodal attention mechanism is used to fuse the modal features of all modal data to obtain multimodal fusion features, including: Performing linear transformation on the modal features of each modal data to obtain the query vector, key vector and value vector of each modal data; Calculate an attention score for each modal data using a scaled dot product attention mechanism based on the query vector, key vector, and value vector of each modal data; Performing weighted summation on the attention scores of each modal data to obtain a multimodal fusion feature; The following formula is used to perform weighted summation on the attention score of each modal data: F=a T attention T +a I attention I +a V attention V +a A attention A expression(2) Among them, F is the multimodal fusion feature, attention T 、attention I 、attention V and attention A are the attention scores of text modality data, image modality data, video modality data, and audio modality data, respectively, T , α I , α V and α A are the weights of text modality data, image modality data, video modality data, and audio modality data, respectively, T +α I +α V +α A =1, d k is the dimension of the key vector, W Q 、W K and W V is a linear transformation matrix, T, I, V, and A are the modal features of text modal data, image modal data, video modal data, and audio modal data, respectively.

6. The rumor detection method based on multimodal information in a low-resource environment according to claim 5 is characterized in that: Determining the rumor detection result of the post corresponding to each node using the event-level structure representation of each node includes: The rumor detection result of the post corresponding to each node is determined by the following calculation formula using the event-level structure representation of each node: in, is the rumor detection result of each post, It is the event-level structure representation of the entire propagation thread, d (l) is the output dimension of the graph neural network, is the hidden representation declared at the (l-1)th layer, || is the concatenation operation with the broadcast mechanism, H (l) is the node feature matrix of the lth layer.

7. The rumor detection method based on multimodal information in a low-resource environment according to claim 6 is characterized in that: The method further comprises: Collect multimodal data for each post in the second domain; Using a domain-adaptive contrastive learning mechanism, posts with the same authenticity in the first and second domains are aligned, and posts with different authenticity are separated to detect rumors in posts in the second domain. Wherein, during the alignment process, the contrast loss between the first domain and the second domain is: in, is the contrast loss between the first field and the second field, N t represents the number of samples in the second domain data, Represents the category label of sample i in the second field The number of samples of the same first field, and Represent the category labels of the second domain sample i and the first domain sample j, N s represents the number of samples in the first field data, represents the event-level feature representation of the i-th sample in the second domain, Represents the feature representation of the sample in the first domain that has the same category as the sample i in the second domain, represents the similarity between the second domain sample i and the first domain sample j, represents the feature representation of samples in the first domain that are different from the sample i in the second domain, It represents the similarity between sample i in the second domain and all samples in the first domain.

8. A rumor detection system based on multimodal information in a low-resource environment, characterized by: include: A processor and a memory for storing a computer program that can be run on the processor; wherein, when the processor is used to run the computer program, the steps of the method according to any one of claims 1 to 7 are performed.

9. A storage medium storing a computer program, wherein: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images