Multi-modal rumor detection system based on multi-scale enhancement

Abstract

The invention discloses a multi-modal rumor detection system based on multi-scale enhancement, relates to the technical field of rumor detection, and is used for solving the technical problem of low detection precision of a multi-modal rumor in the prior art. The multi-modal rumor detection system comprises a multi-scale feature extraction module, a modal perception filtering module, a modal alignment module, a category alignment module and a classifier. Wherein the multi-scale feature extraction module is used for acquiring different modal features; the modal sensing filtering module is used for extracting modal sharing information and modal specific information; the modal alignment module is used for aligning the text features after image enhancement with the self-enhanced social graph features; the category alignment module is used for enabling rumors and rumors to be close in the same vector space and rumors and non-rumors to be far away in the same vector space when the multi-mode rumor detection system is trained; and the classifier is used for acquiring probability distribution of rumor detection according to the modal sharing information and the modal specific information obtained by the modal perception filtering module.

Description

Technical Field

[0001] The present invention relates to the technical field of rumor detection, and more specifically, to a multi-modal rumor detection system based on multi-scale enhancement. Background Art

[0002] A large number of vivid rumors emerging on social media have brought serious negative impacts to people's lives.

[0003] Most of the existing rumor detection methods rely on pre-trained models to extract high-level semantic features of single-modal data, but ignore the complementarity between multi-scale features; on the one hand, the existing rumor detection methods only extract features at a single scale, such as only using high-level semantic features, while ignoring the underlying details, and mostly rely on pre-trained models to extract high-level semantic features of single-modal data, but ignore the complementarity between multi-scale features; on the other hand, the existing rumor detection methods do not consider the feature interaction at different scales during multi-modal fusion, and mostly adopt the global attention mechanism in cross-modal interaction, failing to achieve multi-level feature alignment, resulting in the inability of shallow features and deep semantics to effectively complement each other. Generally speaking, due to the above problems in the existing rumor detection methods, the detection accuracy of multi-modal rumors is low and cannot meet the actual detection requirements. Summary of the Invention

[0004] The purpose of the present invention is to provide a multi-modal rumor detection system based on multi-scale enhancement, which is used to solve the technical problem of low detection accuracy of multi-modal rumors in the prior art. In view of this, the present invention is realized through the following solutions.

[0005] The present invention provides a multi-modal rumor detection system based on multi-scale enhancement, including a multi-scale feature extraction module, a modality-aware filtering module, a modality alignment module, a class alignment module, and a classifier; Among them: the multi-scale feature extraction module is used to obtain different modality features; The modality-aware filtering module is used to fuse different modality features and then extract modality-shared information and modality-specific information; The modality alignment module is used to obtain the self-enhanced features of each modality and the text features after image enhancement, and align the text features after image enhancement with the self-enhanced social graph features; The class alignment module is used to make the multi-modal rumor detection system close rumors and rumors in the same vector space during training, and far away rumors and non-rumors in the same vector space; The classifier is used to obtain the probability distribution of rumor detection according to the modality-shared information and modality-specific information obtained by the modality-aware filtering module; and obtain the objective function for training the multi-modal rumor detection system according to the loss functions of the modality alignment module and the class alignment module.

[0006] Compared with the prior art, the multimodal rumor detection system based on multiscale enhancement of the present invention includes a multiscale feature extraction module, a modal perception filtering module, a modal alignment module, a category alignment module and a classifier, and defines the role of each module and classifier; according to the joint role of each module and the probability distribution of rumor detection by the classifier, rumor detection is completed. Furthermore, the multiscale feature extraction module processes text, image and social graph data respectively to obtain fine-grained features of different scales. The text encoder of this module can use Bert and LSTM+CNN to extract global and local fine-grained features of the text respectively; the image encoder uses the image extractor of the CLIP model and combines the wavelet-improved WTConv to enhance the image resolution and the ability to capture low-frequency information; the social graph encoder extracts the propagation characteristics of the social graph through hidden link inference and signed graph attention network; the multiscale feature extraction method effectively makes up for the fact that the prior art only relies on In order to overcome the defects of high-level semantic features, the complementarity of features at different scales is fully explored. For example, in text feature extraction, not only the high-level semantics of the last layer of Bert can be used, but also the semantic information of the middle and low layers can be obtained through LSTM+CNN, such as word-level, phrase-level and sentence-level features, to provide more comprehensive information support for subsequent rumor detection. In terms of image feature extraction, WTConv expands the receptive field and improves the response ability to low-frequency features, which are crucial in judging whether a post is a rumor. Furthermore, in the multimodal rumor detection system based on multi-scale enhancement of the present invention, The modal-aware filtering module innovatively integrates multiple modal data and performs denoising operations in the frequency domain. The module can first map different modal features to the same vector space, extract modal sharing and specific information through the attention mechanism, and then use an adaptive filtering algorithm based on the frequency domain to replace the traditional standard filtering algorithm with a learnable variable. It automatically filters noise based on training data, reduces the interference of noise in multimodal data on the rumor detection system, and improves the accuracy and stability of the rumor detection system. Furthermore, the modal alignment module and the category alignment module optimize the performance of the rumor detection system from different angles. The modal alignment module can An unsupervised approach is used to enhance the correlation between data of different modalities of rumor posts, and the text after image enhancement and the self-enhanced social graph features are aligned through mean square error loss, so that the rumor detection system can better learn the intrinsic connection between the various modalities; further, the category alignment module can introduce supervised comparative learning, which prompts the system to bring rumors and rumors closer together in the vector space and rumors and non-rumors away during training, thereby learning the essential characteristics of rumors and non-rumors, and improving the rumor detection system's ability to distinguish rumors from non-rumors; through the above-mentioned technical solution of the present invention, the technical problem of low detection accuracy of multimodal rumors in the prior art is solved.

[0007] Further, in the multi-modal rumor detection system based on multi-scale enhancement of the present invention, the multi-scale feature extraction module includes a text encoder, an image encoder, and a social graph encoder; the different modalities include a text modality, an image modality, and a social graph modality; The text encoder is used to obtain text features; The image encoder is used to obtain image features; The social graph encoder is used to obtain social graph features; the social graph is a directed graph.

[0008] Further, in the multi-modal rumor detection system based on multi-scale enhancement of the present invention, the text encoder obtaining text features includes: For the text modality of a post, extract the global fine-grained features and local fine-grained features of the text; Obtain the text features according to the global fine-grained features and local fine-grained features of the text; And / or, the image encoder obtaining image features includes: After performing wavelet graph convolution transformation on the image modality of a post, decompose the input image through wavelet decomposition and decompose the image into different frequency components; Perform small convolution kernel operations at each frequency and use inverse wavelet reconstruction to form an enhanced output to obtain image features; And / or, the social graph encoder obtaining social graph features includes: Perform hidden link inference on the social graph matrix to enrich social graph information; Use a signed graph attention network to learn heterogeneous graph features, and then extract social graph features.

[0009] Further, in the multi-modal rumor detection system based on multi-scale enhancement of the present invention, the modality-aware filtering module extracting modality-shared information and modality-specific information includes: Map the data corresponding to the text features, image features, and social graph features to the same vector space to obtain first multi-modal features; Transform the first multi-modal features through matrix transformation to obtain variants of the first multi-modal features from different perspectives; Obtain a preliminarily fused multi-modal attention score matrix according to the variants of the first multi-modal features; Add an identity matrix to the basis of the preliminarily fused multi-modal attention score matrix and perform cross-modal shared aggregation to obtain second multi-modal features; Obtain modality-shared information and modality-specific information according to the second multi-modal features.

[0010] Further, in the multi-modal rumor detection system based on multi-scale enhancement of the present invention, the modality-shared information is represented by the following formula: ; Among them, represents the th deeply fused multi-modal shared feature, and the subscript represents shared, represents a trainable matrix transformation, represents starting from the first attention head, performing a concatenation operation on the calculation results of H multi-head attention heads, represents the concatenation operation, represents the activation function, represents the trainable matrix transformation of the hth attention head, represents row-by-row concatenation, represents a parameter used to control the proportion of retained information, represents the identity matrix, represents the transpose of the ith preliminarily fused multi-modal feature, represents in the multi-head self-attention mechanism, for the ith data instance, the multi-modal attention score matrix calculated by the th attention head, , represents the activation function, represents the preliminarily fused multi-modal feature, represents the transpose of the preliminarily fused multi-modal feature, represents K the dimension of , that is, the dimension of the multi-modal feature.

[0011] Furthermore, in the multi-modal rumor detection system based on multi-scale enhancement of the present invention, the modal-specific information is represented by the following formula: ; Among them, represents the ith final modal-specific feature, and the subscript represents specific, represents a trainable matrix transformation for obtaining modal-specific information, represents row-by-row concatenation, represents starting from the 0th attention head, performing a concatenation operation on the calculation results of H multi-head attention heads, represents the concatenation operation, represents taking the mean of each attention score matrix, represents the multi-modal attention score matrix calculated by the ith with h heads.

[0012] Furthermore, in the multi-modal rumor detection system based on multi-scale enhancement of the present invention, after obtaining the modal shared information and modal-specific information, it further includes: The final modal features are obtained by using the modality-shared information and modality-specific information, and the final modal features are expressed as: ; where represents the final multi-modal feature, represents the concatenation operation, represents the i-th multi-modal shared feature after deep fusion, s represents shared, represents the i-th final modality-specific feature, represents specific; The classifier is used to obtain the probability distribution of rumor detection based on the final modal features.

[0013] Furthermore, in the multi-modal rumor detection system based on multi-scale enhancement of the present invention, the loss function of the modality alignment module is set as , and the loss function of the category alignment module is , then the objective function for training the multi-modal rumor detection system is expressed as: ; where represents the objective function, represents the cross-entropy loss function, represents the loss function of the modality alignment module, represents the loss function of the category alignment module.

[0014] Furthermore, in the multi-modal rumor detection system based on multi-scale enhancement of the present invention, the loss function of the modality alignment module is expressed as: ; where represents the loss function of the modality alignment module, n represents the number of modality data, represents the shared text feature after image enhancement, s represents shared, represents the self-enhanced image feature; And / or, the loss function of the category alignment module is expressed as: ; where represents the loss function of the category alignment module, represents the sample set, represents the i-th element of the positive sample set of , represents the positive sample set with the same label as sample i, represents the exponential function, It represents the feature vector of the depth-filtered fusion multi-modal data after the i-th sample passes through the modal perception filtering learning module and is normalized. It represents the set of positive samples Medium samples The feature vector of the depth-filtered fusion multi-modal data after passing through the modal perception filtering learning module and being normalized. It represents the temperature coefficient, which is used to adjust the degree of attention to hard negative samples. For the i-th sample, it represents the set of all samples except the i-th sample itself. It represents being a set Medium samples The feature vector of the depth-filtered fusion multi-modal data after passing through the modal perception filtering learning module and being normalized.

[0015] Furthermore, in the multi-modal rumor detection system based on multi-scale enhancement of the present invention, after obtaining the final modal features through the modal shared information and modal specific information, it further includes: Performing fast Fourier transform, filtering algorithm, and inverse fast Fourier transform processing on the final modal feature data in sequence to remove the noise data existing in the final modal feature data. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation to the present invention. In the drawings: Figure 1 It is a schematic structural diagram of the multi-modal rumor detection system of the present invention; Figure 2 It is a schematic diagram showing the influence of different kernel sizes on the performance of the Weibo dataset in Embodiment 4 of the present invention; Figure 3 It is a schematic diagram showing the influence of different kernel sizes on the performance of the Pheme dataset in Embodiment 4 of the present invention; Figure 4 It is a schematic diagram showing the influence of different dropout rates on the performance of the Weibo dataset in Embodiment 4 of the present invention; Figure 5 It is a schematic diagram showing the influence of different dropout rates on the performance of the Pheme dataset in Embodiment 4 of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0017] In order to make the technical problems, technical solutions, and beneficial effects to be solved by the present invention clearer and more understandable, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0018] It should be noted that when an element is referred to as "fixed to" or "disposed on" another element, it can be directly on the other element or indirectly on the other element. When an element is referred to as "connected to" another element, it can be directly connected to the other element or indirectly connected to the other element.

[0019] In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of the present invention, "a plurality of" means two or more, unless otherwise specifically defined. "Several" means one or more, unless otherwise specifically defined.

[0020] Most of the existing rumor detection methods rely on pre-trained models to extract high-level semantic features of single-modal data, but ignore the complementarity between multi-scale features. On the one hand, the existing rumor detection methods only extract features at a single scale, such as only using high-level semantic features, ignoring the underlying details, and mostly relying on pre-trained models to extract high-level semantic features of single-modal data, but ignoring the complementarity between multi-scale features. On the other hand, the existing rumor detection methods do not consider the feature interaction at different scales during multi-modal fusion, and mostly adopt the global attention mechanism in cross-modal interaction, failing to achieve multi-level feature alignment, resulting in the inability to effectively complement shallow features and deep semantics. Generally speaking, due to the above problems, the existing rumor detection methods have low detection accuracy for multi-modal rumors and cannot meet the actual detection requirements.

[0021] To solve the above technical problems, the present invention provides a multi-modal rumor detection system based on multi-scale enhancement, including a multi-scale feature extraction module, a modality-aware filtering module, a modality alignment module, a category alignment module, and a classifier; Among them: the multi-scale feature extraction module is used to obtain different modality features; The modality-aware filtering module is used to fuse different modality features and then extract modality-shared information and modality-specific information; The modality alignment module is used to obtain the self-enhanced features of each modality and the text features after image enhancement, and align the text features after image enhancement with the self-enhanced social graph features; The category alignment module is used to make the multi-modal rumor detection system close rumors and rumors in the same vector space during training, and far away rumors and non-rumors in the same vector space; The classifier is used to obtain the probability distribution of rumor detection based on the modality-shared information and modality-specific information obtained by the modality perception filtering module; and obtain the objective function for training the multi-modal rumor detection system according to the loss functions of the modality alignment module and the category alignment module.

[0022] When the above technical solution is adopted, the multimodal rumor detection system based on multiscale enhancement of the present invention includes a multiscale feature extraction module, a modal perception filtering module, a modal alignment module, a category alignment module and a classifier, and defines the role of each module and the classifier; according to the joint action of each module and the probability distribution of rumor detection by the classifier, rumor detection is completed; further, the multiscale feature extraction module processes text, image and social graph data respectively to obtain fine-grained features of different scales. The text encoder of this module can use Bert and LSTM+CNN to extract global and local fine-grained features of the text respectively; the image encoder uses the image extractor of the CLIP model and combines the wavelet transform improved WTConv to enhance the image resolution and the ability to capture low-frequency information; the social graph encoder extracts the propagation characteristics of the social graph through hidden link inference and signed graph attention network. The multi-scale feature extraction method effectively makes up for the defect of the existing technology that only relies on high-level semantic features, and fully explores the complementarity of features of different scales; for example, in text feature extraction, not only can the high-level semantics of the last layer of Bert be used, but also the semantic information of the middle and low layers can be obtained through LSTM+CNN, such as word-level, phrase-level and sentence-level features, to provide more comprehensive information support for subsequent rumor detection. In terms of image feature extraction, WTConv expands the receptive field and improves the response ability to low-frequency features, which are crucial in judging whether a post is a rumor; further, in the multi-modal rumor detection system based on multi-scale enhancement of the present invention, the modal perception filtering module innovatively integrates multiple modal data and performs denoising operations in the frequency domain. The module can first map different modal features to the same vector space, extract modal sharing and specific information through the attention mechanism, and then adopt an adaptive filtering algorithm based on the frequency domain to replace the traditional standard filtering algorithm with a learnable variable, automatically filter noise according to the training data, reduce the interference of noise in multi-modal data on the rumor detection system, and improve the accuracy and stability of the rumor detection system; In one step, the modal alignment module and the category alignment module respectively optimize the performance of the rumor detection system from different angles. The modal alignment module can use an unsupervised method to enhance the correlation between the modal data of the rumor post, and align the image-enhanced text and the self-enhanced social graph features through the mean square error loss, so that the rumor detection system can better learn the intrinsic connection between the modalities; further, the category alignment module can introduce supervised comparative learning, prompting the system to bring rumors and rumors closer in the vector space and rumors and non-rumors away during training, thereby learning the essential characteristics of rumors and non-rumors, and improving the rumor detection system's ability to distinguish rumors from non-rumors; through the above-mentioned technical solution of the present invention, the technical problem of low detection accuracy of multimodal rumors in the prior art is solved.

[0023] To better understand the present invention, the content of the present invention will be further clarified below in conjunction with specific embodiments. However, the content of the present invention is not limited to the following embodiments.

[0024] Example 1 This embodiment provides a multi-modal rumor detection system based on multi-scale enhancement, including: A multi-scale feature extraction module for obtaining different modal features; A modal perception filtering module for fusing different modal features and then extracting modal shared information and modal specific information; A modal alignment module for obtaining the self-enhanced features of each modality and the text features after image enhancement, and aligning the text features after image enhancement with the self-enhanced social graph features; A category alignment module for making the multi-modal rumor detection system close rumors and rumors in the same vector space during training, and far away rumors and non-rumors in the same vector space; A classifier for obtaining the probability distribution of rumor detection according to the modal shared information and modal specific information obtained by the modal perception filtering module; and obtaining the objective function for training the multi-modal rumor detection system according to the loss functions of the modal alignment module and the category alignment module.

[0025] Example 2 This embodiment provides a multi-modal rumor detection system based on multi-scale enhancement, including: A multi-scale feature extraction module for obtaining different modal features; Further, the multi-scale feature extraction module includes a text encoder, an image encoder, and a social graph encoder; the different modalities include text modality, image modality, and social graph modality; where: The text encoder is used to obtain text features; The image encoder is used to obtain image features; The social graph encoder is used to obtain social graph features; the social graph is a directed graph; Further, the text encoder obtains text features, including: For the text modality of the post, extract the global fine-grained features and local fine-grained features of the text; Obtain the text features according to the global fine-grained features and local fine-grained features of the text; The image encoder obtains image features, including: After performing wavelet graph convolution transformation on the image modality of the post, decompose the input image through wavelet decomposition and decompose the image into different frequency components; Perform small convolution kernel operations at each frequency and use inverse wavelet reconstruction to form an enhanced output to obtain image features; The social graph encoder obtains social graph features, including: Perform hidden link inference on the social graph matrix to enrich social graph information; Use a signed graph attention network to learn heterogeneous graph features and then extract social graph features; A modality-aware filtering module for fusing different modality features and then extracting modality-shared information and modality-specific information; Further, the modality-aware filtering module extracts modality-shared information and modality-specific information, including: S101, Map the data corresponding to the text features, image features, and social graph features to the same vector space to obtain the first multi-modal feature; S102, Obtain variants of the first multi-modal feature from different perspectives through matrix transformation of the first multi-modal feature; S103, Obtain a preliminarily fused multi-modal attention score matrix based on the variants of the first multi-modal feature; S104, Add an identity matrix to the preliminarily fused multi-modal attention score matrix and perform cross-modal shared aggregation to obtain the second multi-modal feature; S105, Obtain modality-shared information and modality-specific information based on the second multi-modal feature; S106, Obtain the final modality feature through the modality-shared information and modality-specific information, and the final modality feature is expressed as: ; where represents the final multi-modal feature, represents the concatenation operation, represents the i-th deeply fused multi-modal shared feature, s represents shared, represents the i-th final modality-specific feature, represents specific; S107, Perform fast Fourier transform, filtering algorithm, and inverse fast Fourier transform processing on the final modality feature data in sequence to remove the noise data existing in the final modality feature data, and the classifier is used to obtain the probability distribution of rumor detection based on the final modality feature after removing the noise data; Further, the modality-shared information is expressed by the following formula: ; where represents the -th deeply fused multi-modal shared feature, and the subscript represents shared, Represents a trainable matrix transformation, Represents starting from the first attention head and performing a concatenation operation on the calculation results of H multi-head attention heads, Represents a concatenation operation, Represents the Sigmoid activation function, Represents the trainable matrix transformation of the h-th attention head, Represents row-wise concatenation, Represents a parameter used to control the proportion of retained information, Represents the identity matrix, Represents the transpose of the i-th preliminarily fused multi-modal feature, Represents in the multi-head self-attention mechanism, for the i-th data instance, the multi-modal attention score matrix calculated by the th attention head, , Represents Activation function, Represents the preliminarily fused multi-modal feature, Represents the transpose of the preliminarily fused multi-modal feature, Represents K The dimension of, that is, the dimension of the multi-modal feature; Furthermore, the modality-specific information is represented by the following formula: ; where, Represents the i-th final modality-specific feature, the subscript Represents specific, Represents a trainable matrix transformation used to obtain modality-specific information, Represents row-wise concatenation, Represents starting from the 0th attention head and performing a concatenation operation on the calculation results of H multi-head attention heads, Represents a concatenation operation, Represents taking the mean of each attention score matrix, Represents the multi-modal attention score matrix calculated by the h-th head for the i-th; Modality alignment module, used to obtain the self-enhanced features of each modality and the text features after image enhancement, and align the text features after image enhancement with the self-enhanced social graph features; Furthermore, the loss function of the modality alignment module is represented as: ; Where, Represents the loss function of the modality alignment module, n Represents the number of modality data, Represents the shared text features after image enhancement,s Indicates sharing Indicates the image self-enhancement feature; A category alignment module for making the multi-modal rumor detection system close rumors and rumors in the same vector space and far away rumors and non-rumors in the same vector space during training; Furthermore, the loss function of the category alignment module is expressed as: ; Wherein, Represents the loss function of the category alignment module, Represents the sample set, Represents the positive sample set The th element, Represents the positive sample set with the same label as sample i, Represents the exponential function, Represents the feature vector of the depth-filtered fusion multi-modal data after the i-th sample passes through the modality-aware filtering learning module and is normalized, Represents the positive sample set The sample in After passing through the modality-aware filtering learning module and being normalized, the feature vector of the depth-filtered fusion multi-modal data, Represents the temperature coefficient, which is used to adjust the attention degree to hard negative samples, Represents, for the i-th sample, the set of all samples except the i-th sample itself, Represents being the set The sample in After passing through the modality-aware filtering learning module and being normalized, the feature vector of the depth-filtered fusion multi-modal data; A classifier for obtaining the probability distribution of rumor detection according to the modality sharing information and modality-specific information obtained by the modality-aware filtering module; and obtaining the objective function for training the multi-modal rumor detection system according to the loss functions of the modality alignment module and the category alignment module; Furthermore, the objective function for training the multi-modal rumor detection system is expressed as: ; Wherein, Represents the objective function, Represents the cross-entropy loss function, Represents the loss function of the modality alignment module, Represents the loss function of the category alignment module.

[0026] Example 3 This embodiment provides a multi-modal rumor detection system based on multi-scale enhancement, including: A multi-scale feature extraction module for obtaining different modal features; Further, the multi-scale feature extraction module includes a text encoder, an image encoder, and a social graph encoder; the different modalities include text modality, image modality, and social graph modality; where: The text encoder is used to obtain text features; the image encoder is used to obtain image features; The social graph encoder is used to obtain social graph features; the social graph is a directed graph; Further, the text encoder obtains text features, including: For the text modality of the post, this embodiment uses two methods to extract features of different granularities; specifically, first use the Bert (Bidirectional Encoder Representations from Transformers) model to extract the global fine-grained features of the text modality, denoted as: ; where represents the global fine-grained features of the text modality, represents the global fine-grained features of the text extracted using the Bert model, represents the text modality of the post; further, for the Chinese dataset, this embodiment uses bert-base-chinese (a BERT model pre-trained specifically for the Chinese language) to obtain the global fine-grained features of the text modality , and for the English dataset, this embodiment uses bert-base-uncased (the English lowercase letter version of the BERT base model) to obtain the global fine-grained features of the text modality ; Further, use the LSTM+CNN (Long Short-Term Memory Network + Convolutional Neural Network) method to extract the local fine-grained features of the text; specifically, this embodiment first processes the text part of the post into a token sequence of length L and denotes it as ; where represents the processed text token sequence, and , d represents the dimension of the word embedding, represents the embedding of the L-th word of the text; further, use LSTM (Long Short-Term Memory Network) to obtain the complete context features of the text, and use k filters with different receptive fields to extract features of different scales, denoted as ; where Denotes the set of features extracted using LSTM and filters with different receptive fields, Denotes using Filters with different receptive fields to extract the last feature in a set of features S obtained by processing the text token sequence of length through LSTM; further, for the features of k filters, respectively, by using the mean pooling method, the multi-scale local fine-grained features of the text modality are obtained after merging, denoted as: ; where Denotes the multi-scale local fine-grained features of the text modality, Denotes the merging operation, Denotes the set of features The th feature in, Denotes taking the mean; further, the fine-grained text features can be denoted as: ; where Denotes the fine-grained text features, Denotes the trainable matrix for fusing text features, Denotes the concatenation operation, Denotes the global fine-grained features of the text modality, Denotes the multi-scale local fine-grained features of the text modality; Further, the image encoder obtains image features, including: For the image modality of the post, in this embodiment, the image extractor of the CLIP (Contrastive Language-Image Pre-Training) model pre-trained on a dataset of 400 million image-text pairs is used to extract the features of the pictures contained in the post, and ResNet50 is selected as the basic architecture of the CLIP model (the CLIP model has made improvements to the original ResNet50); further, considering that the image encoder contains a large number of CNN modules, in order to use CLIP to capture the fine-grained features of the image and expand the perception ability of the modality-aware filter for the image modality, in this embodiment, the wavelet transform (WT) method in the field of signal processing is used. The wavelet transform can reconstruct the image to obtain an image with higher resolution; further, WTConv combines wavelet transform and convolution to capture the low-frequency information in the image and improve the shape deviation of the CNN. WTConv first filters and downsamples the input low-frequency and high-frequency content using wavelet transform, and performs a small kernel depth convolution on different frequency maps before constructing the output using the inverse wavelet transform (IWT); specifically, the processing process of the above WTConv can be expressed as: ; where Y represents the final feature result obtained by performing wavelet transform on the input image, then performing convolution operation with the convolution kernel W, and finally passing through the inverse wavelet transform, represents the inverse wavelet transform operation, represents convolution, represents the trainable convolution kernel parameter with a dimension of , and its number of channels is X four times that of represents wavelet transform, X represents the input of an image; Further, in this embodiment, WTConv is first used to perform wavelet graph convolution transformation on the image. The WTConv layer decomposes the input image through wavelet decomposition, decomposes it into different frequency components, performs small convolution kernel operations on each frequency, and then uses inverse wavelet reconstruction to form an enhanced output. This design allows the WTConv layer to replace the depth convolution operation in the CNN, which can not only expand the receptive field but also significantly enhance the model's response ability to low-frequency information, thereby improving the classification performance of rumors while maintaining the efficiency of parameter growth; further, given an image v , in this embodiment, the image features are obtained through the following formula: ; where represents the image features, represents the trainable matrix for processing the image features, Indicates the use of a pre-trained model to extract image features, Indicates that the input low-frequency and high-frequency content is first filtered and downscaled using wavelet transform, and then a small kernel depth convolution is performed on different frequency maps before using the inverse wavelet transform to construct the output. v Indicates the input image; Furthermore, the social graph encoder obtains social graph features, including: In this embodiment, hidden link inference is first performed on the social graph matrix to enrich the social graph information. Since the social graph is a directed graph, a signed graph attention network is used in this embodiment to learn heterogeneous graph features and then extract propagation features. Specifically, this embodiment creates a heterogeneous social graph G with three types of nodes, namely post nodes, comment nodes, and user nodes. This embodiment uses the vector representation of the post as the initial feature of the post node. Similarly, this embodiment initializes the initial feature of the comment to the vector representation of the comment. In particular, this embodiment uses the mean of the vector representations of all posts published by the user as the initial feature of the user node. Further, this embodiment calculates the similarity between every two nodes. If the similarity between two nodes is greater than the threshold k , and there is no edge between these two nodes, this embodiment connects an edge between these two nodes, that is: ; ; where represents the similarity between nodes and , represents the vector of node , represents the vector of node j, represents the norm (i.e., the modulus) of the vector , represents the norm (i.e., the modulus) of the vector , represents the flag indicating whether there is an edge between nodes i and j after processing, represents logical or, represents whether there is an edge between nodes i and j, and otherwise represents other cases except the above conditions; Furthermore, an enhanced adjacency matrix can be obtained through the above process. Subsequently, in this embodiment, a graph attention network is used to extract graph structure features to simultaneously focus on the importance between both sides of a pair of nodes. Specifically, in this embodiment, the weights between two nodes are first calculated, and then the weights of all nodes are normalized. However, the weights of some edges may be negative, and the weights will become very small after normalization. These small weights are very likely to be large weights in the reverse direction. To avoid this situation, in this embodiment, after calculating the weights of the domain nodes, an inverted weight is added to it, and then the nodes are updated, that is: ; ; ; Among them, represents the set of weights related to node , which contains multiple weight values, represents the weight set in the th weight value, represents the inverted weight set, represents the updated feature representation of node after processing and calculation, represents the activation function, represents the trainable matrix for processing image features, represents node 's feature vector, represents the concatenation operation; Furthermore, after each node in graph G is updated using the above process, the feature matrix after domain aggregation is obtained. Subsequently, the embodiment uses a multi-head attention mechanism to obtain the final social graph features, which is expressed as: ; Among them, represents the social graph features, represents the trainable matrix for obtaining the final information of the social graph, represents starting from the first attention head, performing a concatenation operation on the calculation results of the th multi-head attention head. H represents the number of heads of the multi-head attention, represents the activation function, represents the neighbor nodes of node i, represents the th head, and the attention weight between node and , represents the A trainable weight matrix of the head represents the feature of neighbor node j; A modality-aware filtering module for fusing different modality features and then extracting modality-shared information and modality-specific information; Furthermore, the modality-aware filtering module extracts modality-shared information and modality-specific information, including: S101, mapping the data corresponding to the text feature, image feature, and social graph feature to the same vector space to obtain a first multi-modal feature, and the first multi-modal feature is expressed as: ; where represents the first multi-modal feature, represents a trainable matrix for mapping the text feature to the same vector space, represents the text feature of the i-th sample, represents a trainable matrix for mapping the image feature to the same vector space, represents the image feature of the i-th sample, represents a trainable matrix for mapping the social graph feature to the same vector space, represents the social graph feature of the i-th sample; S102, in order to extract deep modality-shared information and modality-specific information, in this embodiment, the first multi-modal feature is transformed by a matrix to obtain variants of the first multi-modal feature from different perspectives; specifically: ; ; ; where represents the first multi-modal feature after being transformed by the matrix to obtain a query vector (Query vector), represents a trainable weight matrix, represents the query vector 's third component, which is the value of the vector on a specific dimension, represents the first multi-modal feature after being transformed by the matrix to obtain a key vector (Key vector), represents a trainable weight matrix, represents the key vector 's third component, which is the value of the vector on a specific dimension, represents the first multi-modal feature After passing through the matrix The value vector (Value vector) obtained after transformation, Represents a trainable weight matrix, Represents the first multi-modal feature, which is the result of mapping the corresponding data of text, image, and social graph features to the same vector space, Represents the value vector The third component of, is The value taken by the vector on a specific dimension; S103. Obtain a preliminarily fused multi-modal attention score matrix according to the variant of the first multi-modal feature; the preliminarily fused multi-modal attention score matrix is expressed as: ; where, Represents the preliminarily fused multi-modal attention score matrix, Represents Activation function, Represents the query vector obtained by matrix transformation of the first multi-modal feature, Represents the key vector The transpose matrix of, Represents the dimension of K; S104. Add the identity matrix to the preliminarily fused multi-modal attention score matrix and perform cross-modal shared aggregation to obtain the second multi-modal feature; the second multi-modal feature is expressed as: ; where, Represents the second multi-modal feature, Represents a trainable matrix for cross-modal shared aggregation, Represents row-by-row concatenation, Represents a parameter for controlling the proportion of retained information, Represents the identity matrix, Represents the preliminarily fused multi-modal attention score matrix, Represents the value vector The transpose matrix of; S105. Obtain modal shared information and modal-specific information according to the second multi-modal feature; Furthermore, the modal shared information is expressed by the following formula: ; Where, Represents the th deeply fused multi-modal shared feature, the subscript Indicates shared, Represents a trainable matrix transformation, Denotes starting from the 1st attention head and performing a concatenation operation on the calculation results of H multi-head attention heads. Denotes the concatenation operation. Denotes the Sigmoid activation function. Denotes the trainable matrix transformation of the h-th attention head. Denotes row-wise concatenation. Denotes a parameter used to control the proportion of retained information. Denotes the identity matrix. Denotes the transpose of the i-th preliminarily fused multi-modal feature. Denotes in the multi-head self-attention mechanism, for the i-th data instance, the multi-modal attention score matrix calculated by the th attention head. , Denotes the activation function. Denotes the preliminarily fused multi-modal feature. Denotes the transpose of the preliminarily fused multi-modal feature. Denotes K the dimension of, i.e., the dimension of the multi-modal feature; Furthermore, the modality-specific information is represented by the following formula: ; where Denotes the i-th final modality-specific feature, and the subscript Denotes specific. Denotes the trainable matrix transformation used to obtain the modality-specific information. Denotes row-wise concatenation. Denotes starting from the 0th attention head and performing a concatenation operation on the calculation results of H multi-head attention heads. Denotes the concatenation operation. Denotes taking the mean of each attention score matrix. Denotes the i-th multi-modal attention score matrix calculated by h heads; S106, obtaining the final modality feature through the modality-shared information and the modality-specific information, and the final modality feature is represented as: ; where Denotes the final multi-modal feature. Denotes the concatenation operation. Denotes the i-th deeply fused multi-modal shared feature. s Denotes shared. Denotes the i-th final modality-specific feature. Denotes specific; S107. Perform fast Fourier transform, filtering algorithm, and inverse fast Fourier transform on the final modal feature data in sequence to remove the noise data existing in the final modal feature data. The classifier is used to obtain the probability distribution of rumor detection based on the final modal feature after removing the noise data. Further, performing fast Fourier transform, filtering algorithm, and inverse fast Fourier transform on the final modal feature data in sequence includes: S1071. Perform a fast Fourier transform operation on the final modal feature to obtain the multi-modal spectrum data after deep fusion, which can convert the tensor of real numbers into a tensor of complex numbers. The fast Fourier transform operation can be expressed as: ; where represents the spectrum data after the fast Fourier transform of the multi-modal output, represents the one-dimensional fast Fourier transform operation, represents the final modal feature; S1072. Different filtering algorithms may need to be set for posts in different fields in the existing manner. In this embodiment, a learnable variable is used, which can adaptively filter the noise existing in the data according to the training data, and the multi-modal spectrum data after frequency domain filtering can be obtained. This process can be expressed as: ; where represents the multi-modal spectrum data obtained after frequency domain filtering, represents the learnable filtering variable, represents element-wise multiplication, represents the spectrum data after the fast Fourier transform of the multi-modal output; in this way, the multi-modal rumor detection system of this embodiment can automatically learn a filter that conforms to its own data characteristics; S1073. Perform an inverse fast Fourier transform on the filtered spectrum data to obtain the filtered multi-modal data, that is, obtain the filtered final modal feature data. This process can be expressed as: ; represents the final modal feature, specifically representing the filtered final modal feature data here, represents the inverse fast Fourier transform operation, which can convert the tensor of complex numbers into a tensor of real numbers, that is, convert the filtered spectrum data into a filtered multi-modal vector, represents the multi-modal spectrum data obtained after frequency domain filtering; through the fast Fourier transform and the inverse fast Fourier transform, combined with the adaptively learned filtering algorithm, the noise in the data can be effectively reduced; Further, to avoid the problem of vanishing gradients during training, this embodiment can also incorporate operations such as layer normalization, self-connection, and Dropout. The resulting pure multi-modal vector can be expressed as: ; where represents the finally obtained pure multi-modal vector, represents the layer normalization operation, represents the multi-modal features obtained after operations such as frequency domain filtering, represents regularization, which randomly sets the output of neurons to 0 with a certain probability during training to avoid overfitting of the system, represents the data obtained after inverse fast Fourier transform of the frequency domain filtered data; The modality alignment module is used to obtain the self-enhanced features of each modality and the text features after image enhancement, and align the text features after image enhancement with the self-enhanced social graph features; Further, the loss function of the modality alignment module is expressed as: ; where represents the loss function of the modality alignment module, n represents the number of modality data, represents the shared text features after image enhancement, s represents sharing, represents the image self-enhanced features; The category alignment module is used to make the multi-modal rumor detection system close rumors and rumors in the same vector space during training, and far away rumors and non-rumors in the same vector space; Further, the loss function of the category alignment module is expressed as: ; where represents the loss function of the category alignment module, represents the sample set, represents the positive sample set of the th element, represents the positive sample set with the same label as sample i, represents the exponential function, represents the feature vector of the deep filtered fusion multi-modal data after the i-th sample passes through the modality-aware filtering learning module and is normalized, represents the positive sample set in the sample after passing through the modality-aware filtering learning module and being normalized, the feature vector of the deep filtered fusion multi-modal data, Represents the temperature coefficient, which is used to adjust the degree of attention to hard negative samples. Represents the set of all samples except the i-th sample itself for the i-th sample. Indicates it is a set The samples in The feature vector of the depth-filtered fused multi-modal data that has passed through the modality-aware filtering learning module and been normalized. It should be noted that in order to enable the rumor detection system to learn the essential features of rumors and non-rumors, this embodiment uses supervised contrastive learning to prompt the rumor detection system to make rumors as close as possible in the same vector space during training, while making rumors and non-rumors as far apart as possible. Specifically, this embodiment passes the data through the modality-aware filtering learning module twice continuously to obtain two perspectives of the depth-filtered fused multi-modal data, then normalizes them respectively, and calculates the contrastive loss of the data under these two perspectives, that is, obtains the loss function of the above-mentioned class alignment module. A classifier, which is used to obtain the probability distribution of rumor detection according to the modality-shared information and modality-specific information obtained by the modality-aware filtering module; and obtain the objective function for training the multi-modal rumor detection system according to the loss functions of the modality alignment module and the class alignment module. Furthermore, the objective function for training the multi-modal rumor detection system is expressed as: ; Wherein, Represents the objective function, Represents the cross-entropy loss function, Represents the loss function of the modality alignment module, Represents the loss function of the class alignment module.

[0027] The class alignment module is used to make rumors close in the same vector space and rumors and non-rumors far apart in the same vector space when the multi-modal rumor detection system is being trained. The classifier is used to obtain the probability distribution of rumor detection according to the modality-shared information and modality-specific information obtained by the modality-aware filtering module; and obtain the objective function for training the multi-modal rumor detection system according to the loss functions of the modality alignment module and the class alignment module.

[0028] Example 4 This embodiment adopts the same multi-modal rumor detection system based on multi-scale enhancement as in Embodiment 3. The purpose of this embodiment is to verify the effectiveness of the multi-modal rumor detection system based on multi-scale enhancement of the present invention through setting up experiments.

[0029] In this embodiment, the used dataset will be introduced first, followed by the selection of the baseline model for comparison with the present invention. In addition, the evaluation metrics of this embodiment are also described. Subsequently, insights into different implementation details are provided, a comprehensive analysis of the experimental results is carried out, the necessity of the existence of each module is proven through ablation studies, the effectiveness of the rumor detection system in detecting rumors is also confirmed through case analysis, a detailed experimental process for the selection of hyperparameters is given, and finally, the analysis of the time complexity of the algorithm is concluded. The goal of this embodiment is to answer the following questions: Question 1: How does the rumor detection system of the present invention perform compared with existing methods? Question 2: Are the features extracted in a multi-scale manner effective for the final rumor detection? What is the effect of modality-aware filtering? Question 3: What is the performance of the adaptive filtering algorithm compared with traditional filtering algorithms? Question 4: How do different hyperparameter settings affect the model performance? Furthermore, this embodiment uses two publicly available datasets that are widely used at home and abroad, Weibo and PHEME (or Pheme); among them, the Weibo dataset comes from Sina Weibo, the largest online social network platform in China, and the PHEME dataset is collected from X (formerly Twitter). Both of these datasets contain text, images, and comment information. The specific dataset statistics are shown in Table 1.

[0030] Table 1 Dataset Statistics

[0031] Furthermore, to demonstrate the performance of the rumor detection system of the present invention, this embodiment compares 9 strong baseline models in recent years; the rumor detection system of this embodiment can be implemented based on Pytorch, the operating system used is CentOS 7, and the graphics card is GeForce RTX 3090. Like most deep learning tasks, this embodiment selects the Adam algorithm to optimize the training of the model, sets the learning rate to 0.002, the data batch size to 64, the dimensions of the hidden layer and unimodal data to 300, and sets the maximum number of training times on both datasets to 60. The experimental results of this embodiment come from the average value of five experiments. Like most classification tasks, this embodiment selects accuracy (Acc.), precision (Pre.), recall (Rec.), and F1-score (F1.) as the evaluation metrics of this system.

[0032] Furthermore, to answer Question 1, this embodiment compares this system with the baseline method. The results on the two datasets of Weibo and PHEME are shown in Tables 2 and 3.

[0033] Table 2 Performance of different models or systems on the PHEME dataset

[0034] Table 3 Performance of different models or systems on Weibo dataset

[0035] It should be noted that SAFE and previous methods in Tables 2 and 3 above focus on data in two modalities: text and image. SAFE is superior to the previous three methods, reflecting the importance of interaction between data in various modalities. EBGCN and GLAN are superior to previous models, reflecting the importance of social graph information. MFAN first uses social graph as a data perspective of a modality, achieving better results than previous methods. CLFFRD focuses on the complexity of sample difficulty based on MFAN (another module that mainly uses modality-aware representation learning to obtain modality sharing and modality-specific information), while MARN focuses more on modality sharing and modality-specific features of multimodal data. The model of this embodiment (i.e., the rumor detection system of this embodiment) is superior to MARN, and the reasons can be summarized as follows: (1) Multi-scale feature extraction: For text encoders, the present invention uses two different encoding methods to obtain text features of different scales, which can capture the generalized features and chivalrous features of text sequences. For image encoders, the WTConv adopted by the present invention can enhance the resolution of images, expand the receptive field of images under convolutional neural networks, and enhance the model's perception of modal shared information and modal specific information; (2) Denoising method based on frequency domain: The present invention denoises the acquired multimodal vector based on a frequency domain multi-channel method. Through continuous training, it can retain as much important basis for the final classification as possible, and combine with modal perception to complement each other, playing a huge role in capturing the essential attributes of rumors and non-rumors.

[0036] Furthermore, to verify the effectiveness of each module and, at the same time, to answer Question 2, the following variants are proposed in this embodiment: wo-BE only uses LSTM+CNN as the text encoder; wo-LC only uses BERT as the text encoder; wo-WT does not perform wavelet graph convolution transformation on the image; wo-FI does not perform frequency-domain denoising on the features after modality fusion. Please refer to Table 4 for the ablation experiment results. First, generally speaking, the complete model or system achieves the highest accuracy (93.56% / 90.91%) and F1 score (93.24% / 88.77%) on both datasets (Weibo / PHEME), indicating the effectiveness of each component in the rumor detection system proposed by the present invention; after removing BERT, the impact on the Weibo dataset is the greatest, with the accuracy dropping by 4.07% and the F1 score also dropping by 4.5%. On the PHEME dataset, it also causes a 3.9% drop in accuracy and a 4.87% drop in the F1 score, which shows that the semantic understanding ability of BERT is crucial for the rumor detection task in this embodiment. After removing the LSTM+CNN structure, the impact on the PHEME dataset is greater than that on the Weibo dataset, but both drop, indicating the importance of capturing temporal features (LSTM) and local patterns (CNN). Generally speaking, through different levels of feature extraction methods, the performance of downstream tasks can be significantly improved. After removing WTConv, both datasets are affected to varying degrees, which shows that expanding the receptive field of the image and obtaining high-pixel images can effectively improve the performance of the model. After removing the frequency-domain filtering module, the performance of both modules also drops, even more than when removing WTConv, which shows the importance of noise filtering and the essential features of events for rumor detection.

[0037] Table 4 Ablation Experiment Results

[0038] Furthermore, to demonstrate the effectiveness of the adaptive filtering method in this embodiment or the present invention, and to answer Question 3, this embodiment also presents a comparison between the adaptive filtering method in the present invention and traditional filtering algorithms. This embodiment mainly compares with the following several classical filtering methods: The High-Pass Filter (HPF) passes signals with higher frequencies. In this embodiment, after the Fast Fourier Transform, the low-frequency part of the signal is set to 0; The Low-Pass Filter (LPF) passes signals with lower frequencies and attenuates signals with higher frequencies. In this embodiment, after the Fast Fourier Transform, the high-frequency part of the signal is set to 0; The Band-Stop Filter (BSF) attenuates intermediate-frequency signals and passes other signals. In this embodiment, after the Fast Fourier Transform, half of the values of the intermediate-frequency part of the signal are set to 0. Specifically, Table 5 records the comparison of the three algorithms with the adaptive method proposed in this paper on two datasets. It can be observed that on the Weibo and PHEME datasets, the accuracy and F1 score of the rumor detection system used in this embodiment are significantly higher than those of all variants using traditional filters. Among them, on the Weibo dataset, compared with the best-performing low-pass filter, the accuracy is increased by 1.36%, and the F1 score is increased by 1.9% in the rumor category. On the PHEME dataset, compared with the best-performing high-pass filter, the accuracy is increased by 1.56%, and the F1 score is increased by 2.94% in the rumor category. These data indicate that the Weibo dataset may be more suitable for low-frequency features, indicating that global semantic information (i.e., overall consistency) is more important for rumor detection on Weibo. PHEME is more dependent on high-frequency features, which may be related to abnormal propagation patterns in social communication (such as sudden forwarding surges, etc.). Generally speaking, the rumor detection system proposed in this embodiment significantly reduces the performance gap between rumor and non-rumor categories. The F1 score of the rumor category on the Weibo dataset is increased by 2.1% compared with the low-pass filter, and the recall rate of the rumor category on the PHEME dataset is increased by 3.5% compared with the high-pass filter, indicating that the rumor detection system can alleviate category bias to a certain extent by using the adaptive filtering method.

[0039] Table 5 Comparison results of classical filtering algorithms and this embodiment on two datasets

[0040] Furthermore, to answer Question 4, this embodiment also analyzes some hyperparameters, mainly involving the convolution kernel size of the wavelet convolutional transform and the dropout rate of the modal perception filtering module; specifically, this embodiment sets the convolution kernel size to 1-5 respectively, and the results are as Figure 2 and Figure 3 shown. In Figure 2 and Figure 3Among them, Kernel size represents the size of the convolutional kernel; in the Weibo dataset, when the convolutional kernel is less than or equal to 3, the accuracy shows an upward trend, while when the convolutional kernel size is 3 - 5, there is a certain fluctuation. The F1 score also follows this pattern for both rumors and non-rumors. When the convolutional kernel size is equal to 5, the accuracy is the same as when the convolutional kernel size is equal to 3, but the F1 score for the rumor category is slightly larger than that (when the convolutional kernel size is equal to 3), and the F1 score for the non-rumor category is slightly smaller than it. In the PHEME dataset, when the convolutional kernel is less than 3, both the accuracy and the F1 scores for the two categories show an upward trend, but after exceeding 3, they will decline and level off, indicating that the peak has been reached. In summary, it can be found that too small a convolutional kernel (such as 1) will lead to insufficient receptive fields, and too large a convolutional kernel (>=4) may lead to overfitting or computational redundancy, resulting in a decline in metrics. Therefore, in this embodiment, the convolutional kernel size is finally selected to be 3, which also shows that a larger receptive field can effectively improve the model's extraction effect on complex rumor features.

[0041] Furthermore, this embodiment also tested the impact of setting different dropout rates on the modality perception filtering module. Specifically, this embodiment tested the effects of dropout rates from 0.1 to 0.8 on two datasets, and the results are as Figure 4 and Figure 5 shown. In Figure 4 and Figure 5 , Dropout represents the dropout rate; it can be seen that in the Weibo dataset, the accuracy reaches its peak (93.98%) when the dropout rate is 0.5, and the performance is similar under other dropout rates. The F1 score is also the highest (92.31% and 94.94%) when the dropout rate is 0.5, and the performance is slightly lower under other dropout rates. This indicates that a moderate dropout rate can effectively balance overfitting and feature retention. In the PHEME dataset, the accuracy is the highest (accuracy is 90.91%) when the dropout rate is 0.3, and the second highest (accuracy is 90.65%) when it is 0.6. In terms of the F1 score metric, when the dropout rate is 0.6, the accuracy for the rumor category is the highest (84.35%), but it is not as high as when the dropout rate is 0.3 (compared with the accuracy of 90.91% and the accuracy of 90.65%). At this time, the F1 score for the non-rumor category is also relatively high (compared with the accuracy of 93.67% and the accuracy of 93.33%). When the dropout rate is greater than 0.6, the performance on both datasets drops significantly, indicating that too high a dropout rate will lead to a significant reduction in the system's performance, causing underfitting. Considering the performance of the two datasets, this embodiment finally selects to set the dropout rate of the modality perception filtering module to 0.3.

[0042] Please refer to Figure 1 , Figure 1 which is the structural schematic diagram of the multi-modal rumor detection system based on multi-scale enhancement of the present invention. InFigure 1 It can be seen that the present system consists of multiple key modules; specifically: The multi-scale feature extraction module is the front-end part of the entire multi-modal rumor detection system, responsible for preliminarily processing different modal data and extracting rich features; in terms of text data processing, this module adopts two different strategies; on the one hand, it uses the Bert model to obtain the global fine-grained features of the text. The Bert model can deeply analyze the text content of the post from a macroscopic perspective and grasp the overall semantic context of the text; on the other hand, through LSTM combined with CNN filters with different receptive fields, it mines the multi-scale local fine-grained features of the text from a microscopic level and captures the key information in the local area of the text; then, the text features obtained from these two parts are fused to form comprehensive fine-grained text features; for image data, the system uses the image extractor of the CLIP model for preliminary feature extraction, and then uses the improved WTConv technology to further optimize the feature extraction effect, so as to obtain high-quality image features, which helps the system better understand the information contained in the image; in the processing of social graph data, the system constructs a heterogeneous social graph through hidden link inference, and then uses the signed graph attention network to extract the features in the social graph, so as to obtain the social graph propagation features and provide information support at the social relationship level for subsequent analysis.

[0043] The modal perception filtering module plays a connecting role in the multi-modal rumor detection system; this module receives the text, image, and social graph features output by the multi-scale feature extraction module. The primary task of this module is to map these features from different modalities to the same vector space, so that the features of different modalities can be processed in a unified framework for subsequent processing; further, by using the attention mechanism, this module can extract the modality-shared information and modality-specific information, so as to conduct a more in-depth mining and integration of the multi-modal features; in addition, in order to improve the quality of the features, the modal perception filtering module also performs denoising operations on the features in the frequency domain, adaptively adjusts the learnable filtering variables, and automatically removes noise according to the characteristics of the training data, and finally outputs more pure and effective multi-modal features, providing a better data basis for the processing of subsequent modules.

[0044] The modal alignment module mainly focuses on enhancing the correlation between the data of each modality in the rumor post; in actual multi-modal data, there are often internal connections between different modalities, but these connections may not be obvious; the modal alignment module adopts an unsupervised method to calculate the mean square error loss between the text features after image enhancement and the self-enhanced social graph features, so as to guide the model to learn the potential relationships between modalities, making the representations of the features of different modalities in the vector space more coordinated and consistent, thereby improving the effect of the entire multi-modal data processing.

[0045] The category alignment module introduces a supervised contrastive learning method, aiming to enable the model to better distinguish the essential features of rumors and non-rumors; this module processes the multi-modal data feature vectors after deep filtering, fusion, and normalization. Based on the sample data from two perspectives, by promoting the mutual proximity of rumor samples in the vector space while making rumor samples and non-rumor samples move away from each other, the multi-modal rumor detection system learns the key feature differences that can effectively distinguish rumors and non-rumors, thereby improving the accuracy and reliability of the multi-modal rumor detection system in the rumor detection task.

[0046] The classifier module is the last link of the entire system and is responsible for making the final classification judgment on the pure multi-modal vectors processed by multiple previous modules; the classifier module inputs these vectors into the fully connected layer, performs a linear transformation through the trainable weight matrix and bias in the fully connected layer, and then uses the activation function to calculate the predicted probability distribution, thereby determining whether the input data is a rumor; furthermore, to optimize the performance of the system, the classifier adopts an improved multi-task loss combination method as the objective function. By adaptively adjusting the weight parameters of each task loss and comprehensively considering the requirements of classification tasks and modality alignment and other related tasks, the overall performance of the system in the rumor detection task is further improved.

[0047] In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in a suitable manner in any one or more embodiments or examples.

[0048] The above is only the specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of changes or substitutions, which should all be covered by the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claimed rights.

Claims

1. A multi-modal rumor detection system based on multi-scale enhancement, characterized in that, It includes a multi-scale feature extraction module, a modality perception filtering module, a modality alignment module, a category alignment module, and a classifier; Among them: the multi-scale feature extraction module is used to obtain different modality features; The modality perception filtering module is used to fuse different modality features and then extract modality-shared information and modality-specific information; The modality alignment module is used to obtain the self-enhanced features of each modality and the text features after image enhancement, and align the text features after image enhancement with the self-enhanced social graph features; The category alignment module is used to make the multi-modal rumor detection system close rumors and rumors in the same vector space during training, and far away rumors and non-rumors in the same vector space; The classifier is used to obtain the probability distribution of rumor detection according to the modality-shared information and modality-specific information obtained by the modality perception filtering module; and obtain the objective function for training the multi-modal rumor detection system according to the loss functions of the modality alignment module and the category alignment module.

2. The multi-modal rumor detection system based on multi-scale enhancement according to claim 1, wherein The multi-scale feature extraction module includes a text encoder, an image encoder, and a social graph encoder; the different modalities include text modality, image modality, and social graph modality; The text encoder is used to obtain text features; The image encoder is used to obtain image features; The social graph encoder is used to obtain social graph features; The social graph is a directed graph.

3. The multi-modal rumor detection system based on multi-scale enhancement according to claim 2, wherein The text encoder obtains text features, including: For the text modality of the post, extract the global fine-grained features and local fine-grained features of the text; Obtain the text features according to the global fine-grained features and local fine-grained features of the text; And / or, the image encoder obtains image features, including: After performing wavelet graph convolution transformation on the image modality of the post, decompose the input image through wavelet decomposition and decompose the image into different frequency components; Perform small convolution kernel operations at each frequency and use inverse wavelet reconstruction to form an enhanced output to obtain image features; And / or, the social graph encoder obtains social graph features, including: Perform hidden link inference on the social graph matrix to enrich social graph information; Use a signed graph attention network to learn heterogeneous graph features, and then extract social graph features.

4. The multi-modal rumor detection system based on multi-scale enhancement according to claim 3, characterized in that, The modality perception filtering module extracts modality-shared information and modality-specific information, including: Map the data corresponding to the text features, image features, and social graph features to the same vector space to obtain the first multi-modal feature; Transform the first multi-modal feature through matrix transformation to obtain variants of the first multi-modal feature from different perspectives; Obtain a preliminarily fused multi-modal attention score matrix according to the variants of the first multi-modal feature; Add an identity matrix to the basis of the preliminarily fused multi-modal attention score matrix and perform cross-modal shared aggregation to obtain the second multi-modal feature; Obtain modality-shared information and modality-specific information according to the second multi-modal feature.

5. The multi-modal rumor detection system based on multi-scale enhancement according to claim 4, wherein The modality-shared information is represented by the following formula: ； Among them, represents the th deeply fused multi-modal shared feature, and the subscript represents shared, represents a trainable matrix transformation, represents starting from the 1st attention head, concatenating the calculation results of H multi-head attention heads, represents the concatenation operation, represents the activation function, represents the trainable matrix transformation of the hth attention head, represents row-wise concatenation, represents the parameter used to control the proportion of retained information, represents the identity matrix, represents the transpose of the ith preliminarily fused multi-modal feature, represents in the multi-head self-attention mechanism, for the ith data instance, the multi-modal attention score matrix calculated by the th attention head, , represents the activation function, represents the preliminarily fused multi-modal feature, represents the transpose of the preliminarily fused multi-modal feature, represents K the dimension of, that is, the dimension of the multi-modal feature.

6. The multi-modal rumor detection system based on multi-scale enhancement according to claim 5, characterized in that, The modality-specific information is represented by the following formula: ; where represents the i-th final modality-specific feature, and the subscript represents specific represents a trainable matrix transformation for obtaining modality-specific information represents concatenation row by row represents starting from the 0-th attention head, performing a concatenation operation on the calculation results of H multi-head attention heads represents a concatenation operation represents taking the mean of each attention score matrix represents the i-th multi-modal attention score matrix calculated by h heads.

7. The multi-modal rumor detection system based on multi-scale enhancement according to claim 6, characterized in that, After obtaining the modality-shared information and modality-specific information, it further includes: Obtain the final modal feature by means of the said modal-shared information and modal-specific information, and the final modal feature is expressed as: ; where represents the final multi-modal feature, represents the concatenation operation, represents the i-th deeply fused multi-modal shared feature, s represents shared, represents the i-th final modal-specific feature, represents specific; The classifier is used to obtain the probability distribution of rumor detection according to the final modality features.

8. The multi-modal rumor detection system based on multi-scale enhancement according to claim 7, characterized in that Set the loss function of the modality alignment module as , and the loss function of the category alignment module as . Then the objective function for training the multi-modal rumor detection system is expressed as: ； Among them, represents the objective function, represents the cross-entropy loss function, represents the loss function of the modality alignment module, represents the loss function of the class alignment module.

9. The multi-modal rumor detection system based on multi-scale enhancement according to claim 8, wherein The loss function of the modal alignment module is expressed as: ； Among them, represents the loss function of the modality alignment module, n represents the number of modality data, represents the shared text feature after image enhancement, s represents sharing, represents the image self-enhancement feature; And / or, the loss function of the category alignment module is expressed as: ； Among them, represents the loss function of the category alignment module, represents the sample set, represents the positive sample set the th element, represents the positive sample set with the same label as sample i, represents the exponential function, represents the feature vector of the depth-filtered fusion multi-modal data after the i-th sample passes through the modality-aware filtering learning module and is normalized, represents the positive sample set the sample in passes through the modality-aware filtering learning module and is normalized, and the feature vector of the depth-filtered fusion multi-modal data, represents the temperature coefficient, which is used to adjust the attention degree to hard negative samples, represents, for the i-th sample, the set of all samples except sample i itself, represents being the set the sample in passes through the modality-aware filtering learning module and is normalized, and the feature vector of the depth-filtered fusion multi-modal data.

10. The multi-modal rumor detection system based on multi-scale enhancement according to claim 9, characterized in that, After obtaining the final modal features through the modal shared information and modal-specific information, it further includes: Performing fast Fourier transform, filtering algorithm, and inverse fast Fourier transform processing on the final modal feature data in sequence to remove the noise data existing in the final modal feature data.

Images