A rumor detection method and system based on cross-modal consistency enhancement

Abstract

The present application relates to the field of rumor detection, and provides a rumor detection method and system based on cross-modal consistency enhancement, which comprises classifying rumor events and constructing data to obtain rumor data; obtaining multi-modal feature information of the rumor data, obtaining multi-modal graph representation according to the multi-modal feature information, and obtaining multi-modal representation through the multi-modal graph representation; calculating confidence weights of the multi-modal representation, establishing a multi-modal gating network, and performing modal reconstruction through the multi-modal gating network and the confidence weights to obtain multi-modal enhanced representation; establishing a feature projection layer, inputting the multi-modal enhanced representation into the feature projection layer and performing parameter updating to obtain a self-distillation loss, establishing a cross-modal similarity representation based on the multi-modal representation, performing representation enhancement according to the cross-modal similarity representation to obtain structure perception representation; and training a target detection model through cross-entropy loss and self-distillation loss, and using the target detection model to complete rumor detection.

Description

Technical Field

[0001] This invention relates to the field of rumor detection technology, and in particular to a rumor detection method and system based on cross-modal consistency enhancement. Background Technology

[0002] With the widespread adoption of social media platforms, online information dissemination is faster and more extensive than ever before. The popularity of social media has greatly enhanced public communication and information access. However, it has also brought some negative impacts. Among these, the rapid spread of rumors is a key issue, threatening public trust and undermining social stability. Therefore, there is a growing need for effective methods to detect and reduce the spread of rumors. As the methods of rumor dissemination become increasingly diverse, multimodal rumor forms, including textual, visual, and combined forms, are receiving increasing attention. Researchers have explored modalities such as textual, visual, and dissemination structures to enhance representation capabilities. However, these methods either treat each modality independently, leading to significant distributional differences and hindering effective integration between modalities; or they fail to model cross-modal semantic collaboration and complementarity, thus limiting the utilization of cross-modal interactions and hindering the effective comprehensive use of information from various modalities to accurately and promptly identify rumors on social media. Summary of the Invention

[0003] This invention aims to at least solve one of the technical problems existing in related technologies. To this end, this invention provides a rumor detection method and system based on cross-modal consistency enhancement, enabling timely and rapid detection of online rumors.

[0004] This invention provides a rumor detection method based on cross-modal consistency enhancement, comprising: S1: Obtain rumor events, classify and construct data from the rumor events to obtain rumor data; S2: Obtain multimodal feature information of rumor data, obtain multimodal graph representation based on multimodal feature information, and obtain multimodal representation through multimodal graph representation; S3: Calculate the confidence weights of the multimodal representation, establish a multimodal gating network, and perform modal reconstruction through the multimodal gating network and confidence weights to obtain the multimodal enhanced representation; S4: Establish a feature projection layer, input the multimodal enhanced representation into the feature projection layer and update the parameters to obtain the self-distillation loss. Based on the multimodal representation, establish a cross-modal similarity representation, and perform representation enhancement based on the cross-modal similarity representation to obtain the structure-aware representation. S5: Obtain cross-entropy loss through structure-aware representation, train the target detection model using cross-entropy loss and self-distillation loss, and use the target detection model to complete rumor detection.

[0005] According to the rumor detection method based on cross-modal consistency enhancement provided by the present invention, in step S1, the rumor event is processed by text acquisition, propagation structure establishment and image acquisition, and the rumor event is classified into two categories according to its authenticity to obtain rumor data.

[0006] According to the rumor detection method based on cross-modal consistency enhancement provided by the present invention, in step S2, the multimodal feature information includes rumor text representation, rumor image representation and rumor propagation features, to obtain the text vector of the rumor text representation, to determine the word adjacency relationship between words in the rumor text representation, and to obtain the text graph representation through the word adjacency relationship and the text vector; The rumor image representation is visually encoded to obtain visual node features. The visual node features are reconstructed and connected to neighboring nodes to obtain an adjacency matrix. The adjacency matrix and visual node features are used to obtain a visual graph representation. The propagation node representation and node adjacency matrix are extracted from the propagation features of the rumor. The propagation graph representation is obtained through the propagation node representation and node adjacency matrix. The multimodal graph representation includes the text graph representation, the visual graph representation, and the propagation graph representation.

[0007] According to the rumor detection method based on cross-modal consistency enhancement provided by the present invention, in step S2, the multimodal graph representation is input into a graph attention network and mean pooling is performed to obtain the multimodal representation.

[0008] According to the rumor detection method based on cross-modal consistency enhancement provided by the present invention, in step S3, a standard deviation calculation formula is established, the multimodal representation is input into the standard deviation calculation formula and normalized to obtain the confidence weight.

[0009] According to the rumor detection method based on cross-modal consistency enhancement provided by the present invention, in step S3, the gated network parameters are obtained through the confidence weights, the gated network parameters are input into the multimodal gated network to obtain the remodeled mode, and the remodeled mode is spliced ​​and projected to obtain the multimodal enhanced representation.

[0010] According to a rumor detection method based on cross-modal consistency enhancement provided by the present invention, in step S4, the feature projection layer includes a student projection layer and a teacher projection layer. The multimodal enhancement representation is input into the teacher projection layer to obtain the teacher projection layer output, and the multimodal representation is input into the student projection layer to obtain the student projection layer output. The parameters of the feature projection layer are updated, and the divergence loss of the teacher projection layer output and the student projection layer output is calculated to obtain the self-distillation loss.

[0011] According to the rumor detection method based on cross-modal consistency enhancement provided by the present invention, in step S4, after obtaining the cross-modal similarity representation, the enhanced modal representation is calculated through the student projection layer output and the cross-modal similarity representation, and the enhanced modal representation is spliced ​​together to obtain the structure-aware representation.

[0012] According to the rumor detection method based on cross-modal consistency enhancement provided by the present invention, in step S5, a rumor detection model is constructed. The rumor detection model makes predictions based on the structure-aware representation to obtain a predicted probability distribution. The cross-entropy loss is calculated based on the predicted probability distribution. The model detection loss is calculated using the cross-entropy loss and the self-distillation loss. The rumor detection model is trained using the model detection loss to obtain a target detection model.

[0013] This invention also provides a rumor detection system based on cross-modal consistency enhancement, comprising: Rumor Data Module: Used to acquire rumor events, classify rumor events, and construct data to obtain rumor data; Multimodal representation module: used to obtain multimodal feature information of rumor data, obtain multimodal graph representation based on multimodal feature information, and obtain multimodal representation through multimodal graph representation; Multimodal Enhanced Representation Module: Used to calculate the confidence weights of multimodal representations, establish a multimodal gating network, and perform modal reconstruction through the multimodal gating network and confidence weights to obtain multimodal enhanced representations; Structure-aware representation module: used to build a feature projection layer, input multimodal enhanced representation into the feature projection layer and update parameters to obtain self-distillation loss, build cross-modal similarity representation based on multimodal representation, perform representation enhancement based on cross-modal similarity representation, and obtain structure-aware representation; Model training module: used to obtain cross-entropy loss through structure-aware representation, train the object detection model using cross-entropy loss and self-distillation loss, and use the object detection model to perform rumor detection.

[0014] The above-described one or more technical solutions in the embodiments of the present invention have at least one of the following technical effects: This invention provides a rumor detection method and system based on cross-modal consistency enhancement. It obtains multimodal enhanced representations of rumor information through rumor text representation, rumor image representation, and rumor propagation features. This allows the establishment of cross-modal semantic and relational pathways through multimodal features in the rumor propagation process. Furthermore, it improves the consistency of cross-modal representations through cross-modal similarity representation, enhances the model's ability to understand cross-modal semantic consistency and conflicts, and improves the model's performance in identifying rumors through multimodal information.

[0015] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating a rumor detection method based on cross-modal consistency enhancement provided by the present invention.

[0018] Figure 2 This is a schematic diagram of the structure of a rumor detection system based on cross-modal consistency enhancement provided by the present invention.

[0019] Figure label: 100. Rumor Data Module; 200. Multimodal Representation Module; 300. Multimodal Enhanced Representation Module; 400. Structure-Aware Representation Module; 500. Model Training Module. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention. The following embodiments are used to illustrate this invention but cannot be used to limit the scope of this invention.

[0021] In the description of the embodiments of the present invention, it should be noted that the terms "first", "second" and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0022] In the description of the embodiments of the present invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "connected" and "linked" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms in the embodiments of the present invention based on the specific circumstances.

[0023] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0024] The following is combined with Figures 1 to 2 Specific embodiments of the present invention are described below. Figure 1 A flowchart illustrating a rumor detection method based on cross-modal consistency enhancement provided by this invention includes: S1: Obtain rumor events, classify and construct data from the rumor events to obtain rumor data; Furthermore, the objective of this stage is to classify and construct data on rumor events to obtain rumor data. Specifically, in step S1, text acquisition, propagation structure establishment, and image acquisition are performed on rumor events, and the events are classified into two categories based on their veracity to obtain rumor data.

[0025] The specific implementation method for the above steps in this embodiment is as follows: First, it's necessary to collect text related to the rumor incident, the rumor's propagation structure, and images containing the rumor content from social media, thus completing text acquisition, propagation structure establishment, and image acquisition. Furthermore, it's necessary to perform binary classification based on whether the rumor incident is true or not to determine its veracity, thereby obtaining a veracity label for the rumor. Ultimately, all of the above content is used as rumor data.

[0026] S2: Obtain multimodal feature information of rumor data, obtain multimodal graph representation based on multimodal feature information, and obtain multimodal representation through multimodal graph representation; Furthermore, the objective of this stage is to obtain a multimodal graph representation based on multimodal feature information, thereby ultimately obtaining a multimodal representation. Specifically, in step S2, the multimodal feature information includes rumor text representation, rumor image representation, and rumor propagation features. The text vector of the rumor text representation is obtained, the word adjacency relationships between words in the rumor text representation are determined, and the text graph representation is obtained through the word adjacency relationships and the text vector. The rumor image representation is visually encoded to obtain visual node features. The visual node features are reconstructed and connected to neighboring nodes to obtain an adjacency matrix. The adjacency matrix and visual node features are used to obtain a visual graph representation. The propagation node representation and node adjacency matrix are extracted from the propagation features of the rumor. The propagation graph representation is obtained through the propagation node representation and node adjacency matrix. The multimodal graph representation includes the text graph representation, the visual graph representation, and the propagation graph representation.

[0027] In step S2, the multimodal graph representation is input into a graph attention network and mean pooling is performed to obtain the multimodal representation.

[0028] The specific implementation method for the above steps in this embodiment is as follows: Multimodal feature information includes the rumor text representation t, the rumor image representation v, and the rumor propagation representation p. For each word in the rumor text representation, in order to encode the word and its contextual semantic information, a bidirectional transformer model can be used as a text feature extractor to convert it into a text vector. Subsequently, in order to represent the text information in a structured form, the text is converted into a text graph, and the word set in it is defined as a text node set. Then, based on the text graph, the word adjacency relationships of each word in the rumor text representation are obtained, thus yielding the text adjacency matrix of the rumor text representation. This allows us to obtain a text graph representation that includes the text adjacency matrix, text vectors, and text node sets. ,and .

[0029] For the representation of rumor images, a pre-trained ResNet50 is used as the visual encoder, and its final global average pooling layer and fully connected layer are removed to preserve the spatial structure of intermediate features. The final output convolutional features yield visual node features I. Among them, ResNet50() is the ResNet50 visual encoder.

[0030] The visual node features are reconstructed based on the spatial dimensions of the visual encoder's feature output to obtain region-level feature vectors. The dimension of each region-level feature vector is the same as the number of channels in the visual encoder. The node feature matrix is ​​obtained from these region-level feature vectors. To represent visual information in a structured form, a region-level visual map is constructed for each text-related image. The region-level feature vectors of each image are then aggregated to obtain a region-level feature set, which is defined as a node set. Using the K-nearest neighbor algorithm, for region-level feature vectors, each node is connected to its k nearest neighbors based on their cosine similarity in the feature space, thus obtaining the proximity relationships between nodes. An adjacency matrix is ​​then constructed based on these proximity relationships. This allows us to obtain a visual representation. ,and .

[0031] The characteristics of rumor propagation include the propagation chain of the rumor from the source post to other posts, resulting in a set of post nodes. The adjacency matrix between posts and nodes In addition, it is necessary to obtain the features of each post, treat these posts as propagation nodes, and use their features as propagation node representations. Thus, the set of propagation node representations becomes the post feature set. This allows us to obtain a propagation diagram representation. ,and Multimodal graph representations include text graph representations, visual graph representations, and propagation graph representations.

[0032] Finally, the multimodal representations are input into the graph attention network to obtain their respective structure-aware representations, which are then subjected to mean pooling to obtain the multimodal representations. Here, we take text graph representation as an example, inputting the text graph representation into the graph attention network to obtain the text graph structure-aware representation. : Here, GAT() represents a graph attention network, which then performs average pooling on the text graph structure-aware representation to obtain the text graph representation. : Here, MEAN() represents the average pooling operator, W represents the learnable first average pooling parameter, and b represents the learnable second average pooling parameter. By replacing the input with visual graph representation and propagation graph representation, we can obtain visual graph representation and propagation graph representation respectively. Multimodal representation includes text graph representation, visual graph representation, and propagation graph representation.

[0033] S3: Calculate the confidence weights of the multimodal representation, establish a multimodal gating network, and perform modal reconstruction through the multimodal gating network and confidence weights to obtain the multimodal enhanced representation; Furthermore, the objective of this stage is to establish a multimodal gating network to perform modality reconstruction and obtain a multimodal enhanced representation. Specifically, in step S3, a standard deviation calculation formula is established, the multimodal representation is input into the standard deviation calculation formula and normalized to obtain the confidence weight.

[0034] In step S3, the gated network parameters are obtained through the confidence weights, and the gated network parameters are input into the multimodal gated network to obtain the remodeled mode. The remodeled mode is then spliced ​​and projected to obtain the multimodal enhanced representation.

[0035] The specific implementation method for the above steps in this embodiment is as follows: First, a standard deviation calculation formula needs to be established. The text graph representation, visual graph representation, and propagation graph representation from the multimodal representation are then input into the standard deviation calculation formula to calculate the standard deviation, thus obtaining the modal standard deviation. : in, For multimodal representation, exp() denotes the exponential function. The first standard deviation parameter that can be learned. In this embodiment, the second standard deviation parameter is a learnable parameter. It can be used to represent one of the three modalities: text, visual, and communication. When using it, simply substitute each modality in turn.

[0036] Next, the modal standard deviation is normalized to obtain the confidence weight of each mode. : in, The median confidence weights for each modality.

[0037] Subsequently, a multimodal gating network is established, and its gating network parameters are calculated based on the confidence weights of each modality. Here, the gating networks of each modality included in the multimodal gating network have the same structure, and the gating network parameters are the two parameters input to the gating network. For each modality's gating network... In other words, it includes two parameters: the source modal parameter and the source modal parameter. and target mode Furthermore, the following calculation holds true in a gated network: in, These are intermediate parameters for the gating network. For activation function, To remodel the state, These are the first learnable gating network parameters. These are the learnable parameters of the second gating network.

[0038] This allows us to calculate the gate network parameters for each mode and input them into a multimodal gate network for cross-modal reconstruction, resulting in the reconstructed mode. in, For visual representation, For propagation graph representation, Text-based graph representation, Assign text confidence weights, For visual confidence weights, To propagate confidence weights, For text-gated networks, For visual gating networks, To propagate the gated network, Remodel the text morphology. Remodeling the state for vision, To propagate the model, we remodel the modes. Here, the gating network structures for each mode are the same, as described above, the difference being the input gating network parameters.

[0039] Finally, the reconstructed modes are spliced ​​together to obtain the spliced ​​mode M: Here, concat() represents concatenation, which projects the concatenated modes to obtain the multimodal augmentation representation. : in, For the first projection parameter, This is the second projection parameter.

[0040] S4: Establish a feature projection layer, input the multimodal enhanced representation into the feature projection layer and update the parameters to obtain the self-distillation loss. Based on the multimodal representation, establish a cross-modal similarity representation, and perform representation enhancement based on the cross-modal similarity representation to obtain the structure-aware representation. Furthermore, the objective of this stage is to obtain the self-distillation loss and the structure-aware representation. Specifically, in step S4, the feature projection layer includes a student projection layer and a teacher projection layer. The multimodal augmentation representation is input into the teacher projection layer to obtain the teacher projection layer output, and the multimodal representation is input into the student projection layer to obtain the student projection layer output. The parameters of the feature projection layer are updated, and the divergence loss of the teacher projection layer output and the student projection layer output is calculated to obtain the self-distillation loss.

[0041] In step S4, after obtaining the cross-modal similarity representation, the enhanced modal representation is calculated using the student projection layer output and the cross-modal similarity representation. The enhanced modal representation is then concatenated to obtain the structure-aware representation.

[0042] Regarding the above steps, the specific steps in this embodiment are as follows: First, a feature projection layer is established, which includes a student projection layer. and teacher projection layer The multimodal augmentation representation is input into the teacher projection layer, and the output of the teacher projection layer for each modality is obtained. The multimodal representations are input into the student projection layer, and the student projection layer outputs for each modality are obtained. : During this process, the student projection layer is updated using backpropagation. The parameters of the student projection layer are used. The momentum coefficient μ is obtained by exponential moving average, thus affecting the parameters of the teacher projection layer. Update to stabilize the distillation target. Specifically: in, This represents the updated parameters of the teacher projection layer. Then, the divergence loss of the teacher projection layer output and the student projection layer output is calculated to obtain the self-distillation loss. : in, This indicates the calculation of the divergence loss between the two.

[0043] Next, cross-modal similarity representation is established based on multimodal representation: Where T represents transpose. Expressing the request Norm, For cross-modal similarity representation of text, For visual cross-modal similarity representation, To propagate cross-modal similarity representations.

[0044] The enhanced modal representation can be computed using the teacher projection layer output, the student projection layer output, and the cross-modal similarity representation: in, Output the text to the student projection layer. Output to the visual student projection layer. To transmit the student projection layer output, For learnable text-enhanced modal representation parameters, For learnable visual enhancement modal representation parameters, To enhance the modal representation parameters for learnable propagation, Enhanced modal representation for text, For visual enhancement modal representation, To propagate enhanced modal representations.

[0045] Finally, the enhanced modal representations are concatenated to obtain the structure-aware representation. : S5: Obtain cross-entropy loss through structure-aware representation, train the target detection model using cross-entropy loss and self-distillation loss, and use the target detection model to complete rumor detection.

[0046] Furthermore, the objective of this stage is to obtain the cross-entropy loss, thereby training a rumor detection model, and using the rumor detection model to perform rumor detection. Specifically, in step S5, a rumor detection model is constructed. The rumor detection model makes predictions based on the structure-aware representation to obtain a predicted probability distribution. The cross-entropy loss is calculated based on the predicted probability distribution. The model detection loss is calculated using the cross-entropy loss and the self-distillation loss. The rumor detection model is then trained using the model detection loss to obtain a target detection model.

[0047] The specific implementation method for the above steps in this embodiment is as follows: Here, a rumor detection model needs to be pre-built. The structure-aware representation containing multimodal information is input into the rumor detection model to obtain the probability distribution of whether a statement is a rumor, i.e., the predicted probability distribution. The cross-entropy loss between the predicted probability distribution and the rumor's authenticity label is calculated. The model detection loss is then calculated by summing the cross-entropy loss with the pre-weighted self-distillation loss. The rumor detection model is trained multiple times with the goal of minimizing the model detection loss, resulting in a trained object detection model. This object detection model can then be used to verify rumors on social media.

[0048] Here, the effectiveness of a rumor detection method based on cross-modal consistency enhancement is also verified, where CME is the method proposed in this invention. In existing technologies, MVAE lacks the ability to model deep semantic relationships between text and visual features, making it unable to effectively capture complex cross-modal semantic interactions. SpotFake heavily relies on pre-trained models to extract text and visual features and lacks effective interaction and alignment between the two modalities, thus limiting the model's representational ability. SpotFake+ does not explicitly model or align deep cross-modal information, resulting in insufficient multimodal fusion and ultimately affecting model performance. HMCAN's hierarchical coding network provides hierarchical semantics for text, but insufficient exploration of image features leads to insufficient modal information interaction. CAFE relies on cross-modal fuzzy learning, which may not accurately capture complex modal conflicts. Moreover, the above models only utilize text and image modalities and lack the ability to model propagation paths and structural information, which limits their detection capabilities. MFAN and FSRU model text, image, and social graph modalities, but insufficient modeling of social propagation paths negatively impacts model performance. MFCL relies on a pre-trained propagation network and image-text matching enhancement, making its performance dependent on the pre-training results. Datasets 1 and 2 are publicly available datasets. F indicates that the rumor event is indeed a rumor, and T indicates that it is not a rumor. Table 1 compares the rumor detection results of the method provided in this invention with other methods.

[0049] As can be seen from the table above, the method provided by this invention achieves excellent performance across all parameters. To illustrate the necessity of each part of this invention, an ablation experiment was also conducted, which involved removing some modules and evaluating the detection effect. It can be seen that removing any part reduces the effectiveness of rumor detection, as shown in Table 2.

[0050] Table 2. Schematic diagram of ablation test results

[0051] The following describes a rumor detection device based on cross-modal consistency enhancement provided by the present invention. The rumor detection device based on cross-modal consistency enhancement described below and the rumor detection method based on cross-modal consistency enhancement described above can be referred to and correspond to each other.

[0052] Figure 2 An example is a schematic diagram of the structure of a rumor detection system based on cross-modal consistency enhancement, such as... Figure 2As shown, a rumor detection method based on cross-modal consistency enhancement, as described above, includes: Rumor Data Module 100: Used to acquire rumor events, classify rumor events and construct data to obtain rumor data; Multimodal representation module 200: used to obtain multimodal feature information of rumor data, obtain multimodal graph representation based on multimodal feature information, and obtain multimodal representation through multimodal graph representation; Multimodal Enhanced Representation Module 300: Used to calculate the confidence weights of multimodal representations, establish a multimodal gating network, and perform modal reconstruction through the multimodal gating network and confidence weights to obtain multimodal enhanced representations; Structure-aware representation module 400: used to establish a feature projection layer, input multimodal enhanced representation into the feature projection layer and update parameters to obtain self-distillation loss, establish cross-modal similarity representation based on multimodal representation, perform representation enhancement based on cross-modal similarity representation, and obtain structure-aware representation; Model training module 500: used to obtain cross-entropy loss through structure-aware representation, train the object detection model through cross-entropy loss and self-distillation loss, and use the object detection model to complete rumor detection.

[0053] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A rumor detection method based on cross-modal consistency enhancement, characterized in that, include: S1: Obtain rumor events, classify and construct data from the rumor events to obtain rumor data; S2: Obtain multimodal feature information of rumor data, obtain multimodal graph representation based on multimodal feature information, and obtain multimodal representation through multimodal graph representation; S3: Calculate the confidence weights of the multimodal representation, establish a multimodal gating network, and perform modal reconstruction through the multimodal gating network and confidence weights to obtain the multimodal enhanced representation; S4: Establish a feature projection layer, input the multimodal enhanced representation into the feature projection layer and update the parameters to obtain the self-distillation loss. Based on the multimodal representation, establish a cross-modal similarity representation, and perform representation enhancement based on the cross-modal similarity representation to obtain the structure-aware representation. S5: Obtain cross-entropy loss through structure-aware representation, train the target detection model using cross-entropy loss and self-distillation loss, and use the target detection model to complete rumor detection.

2. The rumor detection method based on cross-modal consistency enhancement according to claim 1, characterized in that, In step S1, the rumor event is processed by acquiring text, establishing a propagation structure, and acquiring images. The rumor event is then classified into two categories based on its authenticity to obtain rumor data.

3. The rumor detection method based on cross-modal consistency enhancement according to claim 1, characterized in that, In step S2, the multimodal feature information includes rumor text representation, rumor image representation, and rumor propagation features. The text vector of the rumor text representation is obtained, the word adjacency relationship between words in the rumor text representation is determined, and the text graph representation is obtained through the word adjacency relationship and the text vector. The rumor image representation is visually encoded to obtain visual node features. The visual node features are reconstructed and connected to neighboring nodes to obtain an adjacency matrix. The adjacency matrix and visual node features are used to obtain a visual graph representation. The propagation node representation and node adjacency matrix are extracted from the propagation features of the rumor. The propagation graph representation is obtained through the propagation node representation and node adjacency matrix. The multimodal graph representation includes the text graph representation, the visual graph representation, and the propagation graph representation.

4. The rumor detection method based on cross-modal consistency enhancement according to claim 1, characterized in that, In step S2, the multimodal graph representation is input into a graph attention network and mean pooling is performed to obtain the multimodal representation.

5. The rumor detection method based on cross-modal consistency enhancement according to claim 1, characterized in that, In step S3, a standard deviation calculation formula is established, the multimodal representation is input into the standard deviation calculation formula and normalized to obtain the confidence weight.

6. The rumor detection method based on cross-modal consistency enhancement according to claim 1, characterized in that, In step S3, the gated network parameters are obtained through the confidence weights, and the gated network parameters are input into the multimodal gated network to obtain the remodeled mode. The remodeled mode is then spliced ​​and projected to obtain the multimodal enhanced representation.

7. The rumor detection method based on cross-modal consistency enhancement according to claim 1, characterized in that, In step S4, the feature projection layer includes a student projection layer and a teacher projection layer. The multimodal augmentation representation is input into the teacher projection layer to obtain the teacher projection layer output, and the multimodal representation is input into the student projection layer to obtain the student projection layer output. The parameters of the feature projection layer are updated, and the divergence loss of the teacher projection layer output and the student projection layer output is calculated to obtain the self-distillation loss.

8. The rumor detection method based on cross-modal consistency enhancement according to claim 7, characterized in that, In step S4, after obtaining the cross-modal similarity representation, the enhanced modal representation is calculated using the student projection layer output and the cross-modal similarity representation. The enhanced modal representation is then concatenated to obtain the structure-aware representation.

9. A rumor detection method based on cross-modal consistency enhancement according to claim 7, characterized in that, In step S5, a rumor detection model is constructed. The rumor detection model makes predictions based on the structure-aware representation to obtain a predicted probability distribution. The cross-entropy loss is calculated based on the predicted probability distribution. The model detection loss is calculated using the cross-entropy loss and the self-distillation loss. The rumor detection model is trained using the model detection loss to obtain a target detection model.

10. A rumor detection system based on cross-modal consistency enhancement, used to execute a rumor detection method based on cross-modal consistency enhancement as described in any one of claims 1 to 9, characterized in that, include: Rumor Data Module: Used to acquire rumor events, classify rumor events, and construct data to obtain rumor data; Multimodal representation module: used to obtain multimodal feature information of rumor data, obtain multimodal graph representation based on multimodal feature information, and obtain multimodal representation through multimodal graph representation; Multimodal Enhanced Representation Module: Used to calculate the confidence weights of multimodal representations, establish a multimodal gating network, and perform modal reconstruction through the multimodal gating network and confidence weights to obtain multimodal enhanced representations; Structure-aware representation module: used to build a feature projection layer, input multimodal enhanced representation into the feature projection layer and update parameters to obtain self-distillation loss, build cross-modal similarity representation based on multimodal representation, perform representation enhancement based on cross-modal similarity representation, and obtain structure-aware representation; Model training module: used to obtain cross-entropy loss through structure-aware representation, train the object detection model using cross-entropy loss and self-distillation loss, and use the object detection model to perform rumor detection.

Images