A multi-modal early rumor detection method based on meta-adversarial and reinforcement learning

Abstract

The application provides a multi-modal early rumor detection method based on meta-adversarial and reinforcement learning, and relates to the technical field of social media content analysis. First, obtain the to-be-detected data of a social media platform. A multi-modal early rumor detection network is constructed. The multi-modal early rumor detection network is trained based on a meta-adversarial algorithm to obtain shared initialization parameters. A strategy network is constructed and trained based on a reinforcement learning algorithm to obtain a trained strategy network. For the to-be-detected data of the social media platform, the multi-modal early rumor detection network is quickly adapted using the shared initialization parameters, fusion features are extracted and input into the trained strategy network, and a rumor timing decision action is output. If the output is to wait, the to-be-detected data is continuously received for timing decision. Otherwise, a multi-modal early rumor detection result is output. The application detects based on the constructed and trained multi-modal early rumor detection network, and improves the accuracy and timeliness of early rumor detection.

Description

Technical Field

[0001] This invention belongs to the technical field of social media content analysis, and more specifically, relates to a multimodal early rumor detection method based on meta-adversarial and reinforcement learning. Background Technology

[0002] With the rapid popularization of mobile internet and smart terminals, social media has become a major channel for people to obtain information and participate in public discussions. Social platforms, with their rapid dissemination, wide user reach, and diverse content formats, enable information to spread among a large user base in a very short time. However, the openness and low barriers to entry of social media also provide conditions for the rapid spread of false information, rumors, and misleading content. Especially in highly sensitive scenarios such as public health emergencies, natural disasters, and social emergencies, rumors often spread rapidly at extremely low cost, easily triggering social panic, interfering with public judgment, and even inducing mass risks.

[0003] Existing rumor detection methods mainly include text analysis-based methods, social graph methods based on propagation features, and multimodal methods. However, in practical applications, social media content often coexists in multiple modalities, including text, images, videos, comment interactions, and user relationship chains, with strong coupling and complementarity between different modalities. Single-modal rumor detection methods often fail to guarantee detection reliability due to insufficient information. While multimodal deep fusion methods have made some progress, they still face challenges such as inconsistencies in features between modalities, strong noise interference, and insufficient cross-platform generalization ability.

[0004] Furthermore, rumors in their early stages typically exhibit characteristics such as limited sample size, an unformed propagation structure, and sparse user interaction, making it difficult for traditional deep learning methods that rely on large-scale training samples to achieve ideal results at this stage. Therefore, early detection of social media rumors faces challenges such as data sparsity, insufficient information on rumor structure, and the presence of multimodal noise, resulting in lower accuracy and timeliness in early rumor detection. Summary of the Invention

[0005] To address the issues of low accuracy and timeliness in existing multimodal early rumor detection methods, this invention proposes a multimodal early rumor detection method based on meta-adversarial and reinforcement learning, thereby improving the accuracy and timeliness of early rumor detection.

[0006] To achieve the above-mentioned technical effects, the technical solution of the present invention is as follows:

[0007] S1: Obtain the data to be detected from social media platforms; S2: Construct a multimodal early rumor detection network; the multimodal early rumor detection network includes: a multimodal feature extraction module, a multimodal enhancement module, and a cross-modal interaction module; S3: Train a multimodal early rumor detection network based on the meta-adversarial algorithm and obtain the shared initialization parameters of the multimodal early rumor detection network; S4: Based on the reinforcement learning algorithm, construct and train a policy network to obtain a trained policy network; the policy network is used to output rumor time-series decision actions based on the fusion features extracted by the multimodal early rumor detection network; S5: For the data to be detected on social media platforms, the shared initialization parameters are used to quickly adapt the multimodal early rumor detection network; S6: Input the fusion features extracted by the adapted multimodal early rumor detection network into the trained policy network, and output the rumor time-series decision action; S7: If the output rumor timing decision action is to wait, continue to receive the data to be detected from the social media platform and return to step S6; otherwise, output the multimodal early rumor detection result based on the adapted multimodal early rumor detection network.

[0008] Furthermore, the multimodal feature extraction module includes: a first text feature extraction unit, a first image feature extraction unit, and a first graph structure feature extraction unit; The multimodal enhancement module includes: a first multi-head self-attention mechanism unit and a first channel attention (SE) unit connected in sequence; The cross-modal interaction module includes: a first interaction attention mechanism unit and a first fusion output unit connected in sequence; The outputs of the first text feature extraction unit, the first image feature extraction unit, and the first graph structure feature extraction unit are all connected to the input of the first multi-head self-attention mechanism unit; the output of the first channel attention (SE) unit is connected to the input of the first interactive attention mechanism unit.

[0009] Furthermore, in the first text feature extraction unit: The data to be detected on social media platforms includes: text data. ,in, This represents the Nth word in the text, where N represents the number of words, and T represents the text data; text-related image data. ,in, This indicates that the text is associated with the image data. Represents the Mth image; interactive data ,in, Indicates interactive data, This represents the Hth interaction action; In the first text feature extraction unit: the text data is encoded using a preset BERT encoder to obtain the text representation of the text data, expressed as:

[0010] In the formula, Text representation, Indicates the BERT encoder; Based on the text representation of the text data, local feature extraction is performed using a pre-defined convolutional neural network to obtain the text dimension features, expressed as:

[0011]

[0012]

[0013] In the formula, This represents the activation function. This represents the output feature of the i-th convolution operation. Represents the weight matrix. Represents the paranoia vector. Represents the input features of the convolution operation. This indicates the operation of taking the maximum value in the time dimension. This represents the output features of the convolution operation. This represents the output characteristics after the pooling operation. Represents the residual embedding vector. This represents the output feature after the Nth pooling operation. Represents text-dimensional features; In the first image feature extraction unit: image feature extraction is performed using a preset VGG16 model to obtain image dimensional features, expressed as:

[0014] In the formula, Represents image dimensional features, Represents the VGG16 model; In the first graph structure feature extraction unit: a graph structure is constructed based on the interaction data; the graph structure is then processed using a pre-defined graph attention network to extract interaction features, resulting in graph structure dimensional features, expressed as:

[0015]

[0016]

[0017]

[0018] In the formula, This represents the original features of the i-th node in the graph structure. This represents the learnable weight matrix. Let represent the feature vector of the i-th node after the linear transformation. This represents the activation function. This indicates a splicing operation. This represents the feature vector of the j-th node after the linear transformation. This represents the original attention scores for nodes i and j. This represents the original attention score between node i and its neighbor node k. Represents an exponential function. Let i represent the set of neighboring nodes. This represents the normalized attention score. Represents a non-linear activation function. Represents a node i The graph structure dimension features.

[0019] Furthermore, in the first multi-head self-attention mechanism unit: multi-head self-attention operations are performed on the text dimension features, image dimension features, and graph structure dimension features respectively to generate initial enhanced features; the initial enhanced features include: text dimension enhanced features, initial image dimension enhanced features, and initial graph structure dimension enhanced features; The multi-head self-attention operation involves generating a query vector Q, a key vector K, and a value vector V from the input features through three linear transformations, expressed as:

[0020]

[0021]

[0022] In the formula, Q represents the query vector, K represents the key matrix, and V represents the value matrix. Represents the attention weight matrix. Represents input features; The query vector matrix Q, key matrix K, and value matrix V are partitioned along the feature dimension into multi-head attention segments. The dot product is calculated and scaled to obtain the attention score for each head, expressed as:

[0023] In the formula, Indicates the scaling factor. This represents the head attention score. Represents the normalized exponential function, This represents the query vector for the i-th attention head. This represents the key matrix of the i-th attention head. This represents the value matrix of the i-th attention head; The outputs of all attention heads are concatenated and then linearly projected to obtain the multi-head attention output. The output and input features of the multi-head attention operation are then subjected to a residual connection operation, followed by layer normalization to obtain the initial enhanced features, expressed as:

[0024] In the formula, Indicates the initial enhancement features, MultiHead This indicates a bullish self-attention strategy. Presentation layer normalization operation, This indicates a random path discard operation; In the first channel attention SE unit: The initial enhancement features are segmented along the channel dimension to obtain the first initial enhancement sub-feature and the second initial enhancement sub-feature, expressed as follows:

[0025] In the formula, Indicates the first initial enhancer feature. This represents the second initial enhancer feature. This indicates a channel segmentation operation; Activate the first initial enhancer feature to obtain the activated first initial enhancer feature; perform element-wise multiplication of the activated first initial enhancer feature and the second initial enhancer feature to generate the gated feature; the expression is as follows;

[0026] In the formula, Indicates the first initial enhancer feature. Indicates the activation function; Performing a linear transformation on the gated features yields the transformed gated features, expressed as:

[0027] In the formula, Indicates gating features, Represents the weight matrix. Represents the bias vector; The transformed gated features and the first multi-head self-attention mechanism unit are input into a preset feedforward network, and residual connection operations are performed to obtain the enhanced features, expressed as:

[0028] In the formula, Indicates enhanced features, This represents a feedforward network.

[0029] Furthermore, in the first interactive attention mechanism unit: By utilizing any two of the text-dimensional enhancement features, image-dimensional enhancement features, and graph structure-dimensional enhancement features, a multi-head self-attention operation is performed to generate image-text fusion features, text-image fusion features, text-graph structure fusion features, graph structure-text fusion features, graph structure-image fusion features, and image-graph structure fusion features. The image-text fusion feature, text-image fusion feature, text-graph structure fusion feature, graph structure-text fusion feature, graph structure-image fusion feature, and image-graph structure fusion feature are fused together to obtain the final fused feature, which is expressed as follows:

[0030] In the formula, Indicates the final fusion characteristics, Indicates the weight value. Indicates the fusion parameters; In the first fusion output unit: based on the final fusion features, a pre-defined multilayer perceptron classifier is used to obtain the rumor detection result, expressed as:

[0031]

[0032]

[0033] In the formula, This represents the features after the first convolution operation. This represents the activation function. This represents a regularization operation. This represents the features after the second convolution operation. , , Represents the weight matrix. This represents the final two-dimensional output vector.

[0034] Furthermore, the steps for training a multimodal early rumor detection network based on the meta-adversarial algorithm and obtaining the shared initialization parameters of the multimodal early rumor detection network are as follows: Based on data from social media platforms, each social media event is set as a task, and N tasks are constructed; each task is divided into a support set and a validation set; the meta-adversarial algorithm is trained in multiple tasks through inner and outer loops; In the inner loop training, for each task in the support set, the parameters of the multimodal early rumor detection network are updated using gradient descent with a joint loss function. The update expression is:

[0035]

[0036]

[0037]

[0038]

[0039] In the formula, This represents learnable initialization parameters. This represents the learning rate obtained through rapid updates. Indicates the learning rate. This represents the parameter updated after the (k+1)th iteration; In the outer loop training, for each task on the validation set, the parameters of the multimodal early rumor detection network are updated using the inner loop training for each task, and the meta-update loss is calculated; adversarial examples are generated using the projective gradient descent method, and the adversarial loss is calculated; the expression for generating adversarial examples is:

[0040]

[0041] In the formula, Indicates adversarial examples, This represents the task of the validation set. This represents the vector that counters the perturbation. Indicates hyperparameters, Represents a symbolic function. Represents the cross-entropy classification loss. This represents the gradient operator. Represents the forward propagation function. This represents the true label of sample x; Based on the meta-update loss and adversarial loss, a total loss function is constructed. Based on this total loss function, the parameters of the multimodal early rumor detection network are updated twice. When the total loss function converges, the shared initialization parameters of the multimodal early rumor detection network are obtained. The expression for the second update is:

[0042] In the formula, This indicates the parameters after the second update. This represents learnable initialization parameters. Represents the total loss function. This represents the gradient operator.

[0043] Furthermore, the expression for the joint loss function is:

[0044] In the formula, This represents the loss weighting coefficient. Represents the forward propagation function. Represents the cross-entropy classification loss. This represents the mean squared error loss. Representation vectors for different modalities. Represents the joint loss function; The expression for the meta-update loss is:

[0045]

[0046] In the formula, express, This represents the loss weighting coefficient. This represents the parameters after k iterations of the inner loop. This represents the classification loss for task i. This represents the mean squared error loss for task i. Indicates the meta-update loss. N Indicates the total number of verification set tasks; The expression for the adversarial loss is:

[0047] In the formula, Represents the cross-entropy classification loss. This represents the gradient operator. express, This represents the true label of sample x. This indicates the number of iterations in the projective gradient descent. Indicates resistance to loss; The expression for the total loss function shown is:

[0048] In the formula, This represents the total loss function.

[0049] Furthermore, based on reinforcement learning algorithms, the process of constructing and training a policy network to obtain a trained policy network is as follows: Step 1: Transform the multimodal early rumor detection problem into a reinforcement learning problem of partially observable Markov decision processes; wherein, the final fused features extracted and fused by the trained multimodal early rumor detection network are used as the state set of reinforcement learning, and waiting, judging as a rumor, and judging as not a rumor are used as the action set of reinforcement learning, and a reward function is defined. Step 2: Construct a policy network and a value network; wherein the policy network is used to output the action probability distribution, and the value network is used to estimate the state value; the policy network and the value network share the parameters in the initially trained multimodal early rumor detection network; Step 3: Obtain the current state of the multimodal early rumor detection network after initial training, input the state into the policy network to obtain the action and the state at the next moment after the action is executed; calculate the reward value using the reward function; construct the state-action-reward experience tuple based on the current state, action, state at the next moment after the action is executed, and reward value, and store it in the experience replay buffer. Step 4: Obtain the state-action-reward experience tuple from the experience replay buffer as input to the value network, and calculate the temporal difference error and generalized advantage estimation; based on the temporal difference error and generalized advantage estimation, construct the shearing probability ratio objective function for near-end policy optimization; Step 5: Update the policy network parameters by maximizing the shearing probability ratio objective function of the near-end policy optimization; construct the mean squared error loss function of the value network, and update the value network parameters by minimizing the mean squared error loss function of the value network. Step 6: Solve the shearing probability ratio objective function and the mean squared error loss function in sequence, update the policy network and the value network until the preset training rounds are reached, and obtain the trained policy network.

[0050] Furthermore, the expression for the objective function of the shear probability ratio is:

[0051] In the formula, The objective function represents the shear probability ratio. Expressing expectations, Represents the probability ratio. This represents the generalized advantage estimation. The clipping threshold is represented by `clip`, and the clipping function is represented by `clip`. The expression for the generalized dominance estimation is as follows:

[0052] In the formula, This represents the generalized advantage estimation. This represents the time difference error at time t. This represents the time difference error at time t+1. Indicates the discount factor. express Time difference error at any moment K represents the decay factor, and K represents the number of expansion steps; The expression for the probability ratio is:

[0053] In the formula, Indicates the current policy in state Select action The probability, This indicates the state of the strategy at the previous moment. Select action probability The expression for the time difference error is:

[0054] In the formula, Indicates the reward value. Indicates the discount factor. Indicates the next state The value function, This represents the value function.

[0055] Furthermore, the policy network includes: a first fully connected layer for extracting the first hidden feature, a first activation function layer for non-linear activation, a second fully connected layer for extracting the second hidden feature, a second activation function layer for non-linear activation, and a first action output layer for outputting reinforcement learning actions, connected in sequence.

[0056] Compared with existing technologies, the beneficial effects of this method are: This invention provides a multimodal early rumor detection method based on meta-adversarial and reinforcement learning, relating to the technical field of social media content analysis. First, it acquires data to be detected from a social media platform. Then, it constructs a multimodal early rumor detection network. The multimodal early rumor detection network is trained using a meta-adversarial algorithm, obtaining shared initialization parameters. A policy network is constructed and trained using a reinforcement learning algorithm, resulting in a pre-trained policy network. For the data to be detected from the social media platform, the shared initialization parameters are used to quickly adapt the multimodal early rumor detection network, extracting fusion features and inputting them into the pre-trained policy network. The network then outputs a rumor timing decision action. If the output is "wait," it continues to receive data to be detected for timing decision-making; otherwise, it outputs the multimodal early rumor detection result. This invention, based on a constructed and trained multimodal early rumor detection network, improves the accuracy and timeliness of early rumor detection. Attached Figure Description

[0057] Figure 1 This is a flowchart illustrating the multimodal early rumor detection method based on meta-adversarial and reinforcement learning proposed in this embodiment of the invention. Figure 2 This diagram illustrates the structure of the multimodal early rumor detection network proposed in this embodiment of the invention. Figure 3A flowchart illustrating the meta-adversarial algorithm proposed in this embodiment of the invention; Figure 4 A flowchart illustrating the reinforcement learning algorithm proposed in this embodiment of the invention; Figure 5 This diagram illustrates the structure of the policy network proposed in this embodiment of the invention. Detailed Implementation

[0058] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the scope of this patent. To better illustrate this embodiment, some parts of the accompanying drawings may be omitted, enlarged, or reduced, and do not represent the actual dimensions; It is understandable to those skilled in the art that some well-known details may be omitted from the accompanying drawings.

[0059] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0060] The positional relationships depicted in the accompanying drawings are for illustrative purposes only and should not be construed as limiting this patent. Example 1 This embodiment proposes a multimodal early rumor detection method based on meta-adversarial and reinforcement learning, such as Figure 1 The flowchart shown illustrates the method, and the method proposed in this embodiment generally includes the following steps: S1: Obtain the data to be detected from social media platforms; S2: Construct a multimodal early rumor detection network; the multimodal early rumor detection network includes: a multimodal feature extraction module, a multimodal enhancement module, and a cross-modal interaction module; S3: Train a multimodal early rumor detection network based on the meta-adversarial algorithm and obtain the shared initialization parameters of the multimodal early rumor detection network; S4: Based on the reinforcement learning algorithm, construct and train a policy network to obtain a trained policy network; the policy network is used to output rumor time-series decision actions based on the fusion features extracted by the multimodal early rumor detection network; S5: For the data to be detected on social media platforms, the shared initialization parameters are used to quickly adapt the multimodal early rumor detection network; S6: Input the fusion features extracted by the adapted multimodal early rumor detection network into the trained policy network, and output the rumor time-series decision action; S7: If the output rumor timing decision action is to wait, continue to receive the data to be detected from the social media platform and return to step S6; otherwise, output the multimodal early rumor detection result based on the adapted multimodal early rumor detection network.

[0061] In this embodiment, as Figure 2 The diagram shows the structure of the multimodal early rumor detection network. The multimodal feature extraction module includes: a first text feature extraction unit, a first image feature extraction unit, and a first graph structure feature extraction unit. The multimodal enhancement module includes: a first multi-head self-attention mechanism unit and a first channel attention (SE) unit connected in sequence; The cross-modal interaction module includes: a first interaction attention mechanism unit and a first fusion output unit connected in sequence; The outputs of the first text feature extraction unit, the first image feature extraction unit, and the first graph structure feature extraction unit are all connected to the input of the first multi-head self-attention mechanism unit; the output of the first channel attention (SE) unit is connected to the input of the first interactive attention mechanism unit.

[0062] In this embodiment, BERT is used to encode the source post text features, and CNN is used for further processing of redundant information; VGG16 model is used to process the output image features; and Graph Attention Network (GAT) is used to extract feature representations from user interaction data.

[0063] In this embodiment, in the first text feature extraction unit: The data to be detected on social media platforms includes: text data. ,in, This represents the Nth word in the text, where N represents the number of words, and T represents the text data; text-related image data. ,in, This indicates that the text is associated with the image data. Represents the Mth image; interactive data ,in, Indicates interactive data, This represents the Hth interaction action; In this embodiment, text data (including video titles, text descriptions, comment texts, etc.), text-related image data (including short video keyframes, user avatars, and homepage images) and interaction data (including user forwarding links, comment interaction graphs, user social relationship graphs, etc.) are included.

[0064] In the first text feature extraction unit: the text data is encoded using a preset BERT encoder to obtain the text representation of the text data, expressed as:

[0065] In the formula, Text representation, Indicates the BERT encoder; Based on the text representation of the text data, local feature extraction is performed using a pre-defined convolutional neural network to obtain the text dimension features, expressed as:

[0066]

[0067]

[0068] In the formula, This represents the activation function. This represents the output feature of the i-th convolution operation. Represents the weight matrix. Represents the paranoia vector. Represents the input features of the convolution operation. This indicates the operation of taking the maximum value in the time dimension. This represents the output features of the convolution operation. This represents the output characteristics after the pooling operation. Represents the residual embedding vector. This represents the output feature after the Nth pooling operation. Represents text-dimensional features; In the first image feature extraction unit: image feature extraction is performed using a preset VGG16 model to obtain image dimensional features, expressed as:

[0069] In the formula, Represents image dimensional features, Represents the VGG16 model; In the first graph structure feature extraction unit: a graph structure is constructed based on the interaction data; the graph structure is then processed using a pre-defined graph attention network to extract interaction features, resulting in graph structure dimensional features, expressed as:

[0070]

[0071]

[0072]

[0073] In the formula, This represents the original features of the i-th node in the graph structure. This represents the learnable weight matrix. Let represent the feature vector of the i-th node after the linear transformation. This represents the activation function. This indicates a splicing operation. This represents the feature vector of the j-th node after the linear transformation. This represents the original attention scores for nodes i and j. This represents the original attention score between node i and its neighbor node k. Represents an exponential function. Let i represent the set of neighboring nodes. The normalized attention score reflects the importance of neighbor node j to node i. Represents a non-linear activation function. Represents a node i The graph structure dimension features.

[0074] Multi-head self-attention processing is applied to the features of the three dimensions of text, image and structure, and the important feature dimensions are enhanced by the channel recalibration mechanism to obtain the enhanced single-modal representation.

[0075] The first multi-head self-attention mechanism unit includes: a first linear layer, a second linear layer, a third linear layer, a first scaled dot product attention mechanism layer, and a first concatenation layer (concat); the outputs of the first linear layer, the second linear layer, and the third linear layer are all connected to the input of the first scaled dot product attention mechanism layer; the output of the first scaled dot product attention mechanism layer is connected to the input of the first concatenation layer.

[0076] In this embodiment, in the first multi-head self-attention mechanism unit: multi-head self-attention operations are performed on the text dimension features, image dimension features, and graph structure dimension features respectively to generate initial enhanced features; the initial enhanced features include: text dimension enhanced features, initial image dimension enhanced features, and initial graph structure dimension enhanced features; The multi-head self-attention operation involves generating a query vector Q, a key vector K, and a value vector V from the input features through three linear transformations, expressed as:

[0077]

[0078]

[0079] In the formula, Q represents the query vector, K represents the key matrix, and V represents the value matrix. Represents the attention weight matrix. Represents input features; The query vector matrix Q, key matrix K, and value matrix V are partitioned along the feature dimension into multi-head attention segments. The dot product is calculated and scaled to obtain the attention score for each head, expressed as:

[0080] In the formula, Indicates the scaling factor. This represents the head attention score. Represents the normalized exponential function, This represents the query vector for the i-th attention head. This represents the key matrix of the i-th attention head. This represents the value matrix of the i-th attention head; The outputs of all attention heads are concatenated and then linearly projected to obtain the multi-head attention output. The output and input features of the multi-head attention operation are then subjected to a residual connection operation, followed by layer normalization to obtain the initial enhanced features, expressed as:

[0081] In the formula, Indicates the initial enhanced features, MultiHead This indicates a bullish self-attention strategy. Presentation layer normalization operation, This indicates a random path discard operation; The first channel attention SE unit and the second channel attention SE unit include: a fourth linear layer, a first channel chunk layer, and a first activation function layer connected in sequence; a fifth linear layer, a first feedforward network layer (FFN), and a first layer normalization layer connected in sequence; the output of the first channel chunk layer and the output of the first activation function layer are element-wise multiplied and then connected to the input of the fifth linear layer.

[0082] In the first channel attention SE unit: The initial enhancement features are segmented along the channel dimension to obtain the first initial enhancement sub-feature and the second initial enhancement sub-feature, expressed as follows:

[0083] In the formula, Indicates the first initial enhancer feature. This represents the second initial enhancer feature. This indicates a channel segmentation operation; for example, Indicates the supported value. Acting as a gate, Indicates the supported value. Act as a gate Activate the first initial enhancer feature to obtain the activated first initial enhancer feature; perform element-wise multiplication of the activated first initial enhancer feature and the second initial enhancer feature to generate the gated feature; the expression is as follows;

[0084] In the formula, Indicates the first initial enhancer feature. Indicates the activation function; Performing a linear transformation on the gated features yields the transformed gated features, expressed as:

[0085] In the formula, Indicates gating features, Represents the weight matrix. Represents the bias vector; The transformed gated features and the first multi-head self-attention mechanism unit are input into a preset feedforward network, and residual connection operations are performed to obtain the enhanced features, expressed as:

[0086]

[0087] In this embodiment, three types of bidirectional interaction mappings—text-image, text-structure, and image-structure—are constructed based on multi-head cross-modal attention to characterize the correlation between different modalities. The three types of interaction features are dynamically integrated through a learnable weighted fusion network to generate a global joint representation of events.

[0088] In this embodiment, in the first interactive attention mechanism unit: By utilizing any two of the text-dimensional enhancement features, image-dimensional enhancement features, and graph structure-dimensional enhancement features, a multi-head self-attention operation is performed to generate image-text fusion features, text-image fusion features, text-graph structure fusion features, graph structure-text fusion features, graph structure-image fusion features, and image-graph structure fusion features. For example, taking text and image features as examples, the two modalities are mapped to a shared space as input to the cross-attention mechanism.

[0089]

[0090]

[0091]

[0092] In the formula, Q, K, and V represent the query, key, and value vectors, respectively. These represent the attention weight matrix.

[0093] The output is calculated based on the multi-head attention mechanism formula, generating image-text fusion features. By analyzing input text, images, and social graph features, five other cross-modal features are generated. These six cross-modal fusion features are input into a learnable weighted fusion network to achieve dynamic integration. This network dynamically integrates cross-modal features by learning the importance weights of each feature.

[0094] The image-text fusion feature, text-image fusion feature, text-graph structure fusion feature, graph structure-text fusion feature, graph structure-image fusion feature, and image-graph structure fusion feature are fused together to obtain the final fused feature, which is expressed as follows:

[0095] In the formula, Indicates the final fusion characteristics, Indicates the weight value. Indicates the fusion parameters; The formula for calculating the weight value is as follows:

[0096] In the formula,

[0097] In the first fusion output unit: based on the final fusion features, a pre-defined multilayer perceptron classifier is used to obtain the rumor detection result, expressed as:

[0098]

[0099]

[0100] In the formula, This represents the features after the first convolution operation. This represents the activation function. This represents a regularization operation. This represents the features after the second convolution operation. , , Represents the weight matrix. This represents the final two-dimensional output vector, which shows the probability distribution of the category (rumor, non-rumor). The predicted category is determined by the index corresponding to the maximum value in this vector.

[0101] This invention employs a cross-modal triple feature fusion mechanism to jointly model textual semantics, visual cues, and user interaction structure information, significantly improving information utilization and overcoming the information loss problem in early rumor identification inherent in single-modal methods. Simultaneously, this invention introduces a meta-adversarial learning framework, enabling the model to possess stronger generalization ability and robustness when handling few-sample rumors, novel propagation patterns, and adversarial forgery, effectively improving adaptability in real-world, complex social environments. Furthermore, this invention uses a reinforcement learning strategy to optimize the rumor judgment timing, overcoming the limitations of traditional static classification methods. This allows the model to autonomously select the optimal detection time in dynamic scenarios with streaming data, achieving more efficient early warning. The overall technical solution possesses good cross-platform versatility and can be widely applied to fake information detection tasks on various social media platforms such as Douyin, Weibo, WeChat, and Twitter, demonstrating significant practical value and promotional potential.

[0102] Example 2 In this embodiment, the training steps of a multimodal early rumor detection method based on meta-adversarial and reinforcement learning proposed in Embodiment 1 are described in detail.

[0103] In this embodiment, as Figure 3 The flowchart shown illustrates the steps for training a multimodal early rumor detection network based on the meta-adversarial algorithm and obtaining the shared initialization parameters of the multimodal early rumor detection network: Based on data from social media platforms, each social media event is designated as a task, resulting in N tasks; each task is then divided into a support set. (For in-task learning) and validation set (For meta-update); the meta-adversarial algorithm performs multi-task training with inner and outer loops; In the inner loop training, for each task in the support set, the parameters of the multimodal early rumor detection network are updated using gradient descent with a joint loss function. The update expression is:

[0104]

[0105]

[0106]

[0107]

[0108] In the formula, This represents learnable initialization parameters. This represents the learning rate obtained through rapid updates. Indicates the learning rate. This represents the parameter updated after the (k+1)th iteration; In the outer loop training, for each task on the validation set, the parameters of the multimodal early rumor detection network are updated using the inner loop training for each task, and the meta-update loss is calculated; adversarial examples are generated using the projective gradient descent method, and the adversarial loss is calculated; the expression for generating adversarial examples is:

[0109]

[0110] In the formula, Indicates adversarial examples, This represents the task of the validation set. This represents the vector that counters the perturbation. This represents a hyperparameter used to constrain the disturbance amplitude; The sign function is represented by its gradient, which determines its direction. Represents the cross-entropy classification loss. This represents the gradient operator. Represents the forward propagation function. This represents the true label of sample x; Adversarial examples are generated using the projective gradient descent method, introducing perturbations into the validation set samples. .

[0111] Based on the meta-update loss and adversarial loss, a total loss function is constructed. Based on this total loss function, the parameters of the multimodal early rumor detection network are updated twice. When the total loss function converges, the shared initialization parameters of the multimodal early rumor detection network are obtained. The expression for the second update is:

[0112] In the formula, This indicates the parameters after the second update. This represents learnable initialization parameters. Represents the total loss function. This represents the gradient operator.

[0113] The expression for the joint loss function is:

[0114] In the formula, This represents the loss weighting coefficient. Represents the forward propagation function. Represents the cross-entropy classification loss. This represents the mean squared error loss. Representation vectors for different modalities. Represents the joint loss function; The expression for the meta-update loss is:

[0115]

[0116] In the formula, express, This represents the loss weighting coefficient. This represents the parameters after k iterations of the inner loop. This represents the classification loss for task i. This represents the mean squared error loss for task i. Indicates the meta-update loss. N Indicates the total number of verification set tasks; The expression for the adversarial loss is:

[0117] In the formula, Represents the cross-entropy classification loss. This represents the gradient operator. express, This represents the true label of sample x. This indicates the number of iterations in the projective gradient descent. Indicates resistance to loss; The expression for the total loss function shown is:

[0118] In the formula, This represents the total loss function.

[0119] In this embodiment, events are constructed as meta-learning tasks. MAML is used to quickly adapt to the support set and update shared initialization parameters on the query set, enabling the transfer of new events. Simultaneously, PGD adversarial training is introduced, jointly optimizing the adversarial loss and classification loss to improve the model's robustness and generalization ability. The meta-adversarial training process includes four steps: dataset sampling, rapid support set update, multi-step adversarial update, and meta-parameter update.

[0120] Each event is treated as an independent meta-task. Through the second-order gradient optimization process of MAML, a support set and a query set are constructed, supporting intra-set updates (intra-task adaptation): quickly learning event-specific patterns; and inter-task updates (outside the query set): optimizing general initialization parameters so that the model can converge quickly in new events. Meta-learning enables the model to adapt quickly with few samples. Adversarial perturbations such as PGD are introduced in the meta-learning stage to generate perturbation samples in text, images, or structures; the training loss is jointly composed of task classification loss and adversarial loss, improving the model's stability against complex adversarial examples such as malicious forged media and manipulated expressions.

[0121] Example 2 In this embodiment, the steps of constructing and training a policy network to obtain a trained policy network are described in detail for the reinforcement learning-based algorithm of the multimodal early rumor detection method based on meta-adversarial and reinforcement learning proposed in Embodiment 1.

[0122] In this embodiment, early rumor detection is constructed as a partially observable Markov decision process (POMDP), with fused multimodal features as states and "wait" and "determine" as actions. The early detection target is optimized and modeled through a reward function that balances prediction accuracy and detection timeliness.

[0123] In this embodiment, as Figure 4 The flowchart shown illustrates the process of constructing and training a policy network based on a reinforcement learning algorithm to obtain a trained policy network: Step 1: Transform the multimodal early rumor detection problem into a reinforcement learning problem of partially observable Markov decision processes; wherein, the final fused features extracted and fused by the trained multimodal early rumor detection network are used as the state set of reinforcement learning, and waiting, judging as a rumor, and judging as not a rumor are used as the action set of reinforcement learning, and a reward function is defined. The temporal decision-making problem of early rumor detection is modeled as a partially observable Markov Decision Process (POMDP), which includes state S (fused multimodal features and historical observations), action A (wait, determine as rumor, determine as not rumor), reward R (balancing accuracy and advance notice, encouraging early and reliable identification of rumors), state transition T, and observation O. POMDP enables the model to make dynamic decisions based on incomplete information.

[0124] The POMDP representation is adopted because in real-world social media environments, event propagation data often arrives gradually in a dynamic and incremental manner. The information the model can acquire at any given time is incomplete, making a "partially observable" decision framework suitable. The system state is represented by fused multimodal features, including text encoding, image encoding, propagation structure features, and time step information. The action set includes "wait," "determine as rumor," and "determine as nonrumor." The observation set comes from gradually arriving social media data over time. The state transition function describes the changes in state features at the next time step under different action choices. The reward function comprehensively considers prediction accuracy and early detection, optimizing early detection by penalizing excessive waiting and rewarding early correct classification.

[0125] A partially observable Markov decision process consists of a set of states, a set of actions, a set of observations, a state transition function, and a reward function. , Represents the observation space Wherein, the state set: the system state at any time t is represented by the fused multimodal features and time information, expressed as:

[0126] The action set implies either continuing observation (waiting) or making a judgment at the current moment (judging it as a rumor / judging it as not a rumor). The expression for the action set is:

[0127] Observation set: Observations are the visible information that an agent can obtain at time t, typically consisting of several recently arrived replies / comments, accompanying images, or user metadata (i.e., a portion of the historical observation sequence). There is partial observability between observations and the true state.

[0128] The state transition function describes the action How a state is demonstrated under different actions, i.e., conditional probability. State transitions reflect the evolution of event representations under different actions. When the "wait" action is taken, the environment will change as new comments / shares / media arrive. Updated based on new observations; when a decision action is taken, the episode terminates or enters an evaluation / archiving state, expressed as:

[0129] The reward function is used to balance detection accuracy and early detection, and its expression is:

[0130] In the formula, This represents the weighting coefficient. The accuracy index for judging the action is set as follows (1 indicates consistency with the actual label, and 0 indicates inconsistency; for waiting actions, 0 or a small negative value can be used). This represents the delay cost from the event's occurrence to the current time t (which can be defined as the number of steps or the time length), with λ,γ>0 as weighting coefficients to balance accuracy and lead time. This reward can be extended to a weighted combination considering confidence level, false positive costs, or platform costs.

[0131] The system state is represented by fused multimodal features. Actions include waiting and two types of decision behaviors. Observations are derived from historical observation sequences. State transitions reflect the evolution of event representation under different actions. The reward function simultaneously measures detection accuracy and early detection to maximize the effect of early detection.

[0132] Step 2: Construct a policy network and a value network; wherein the policy network is used to output the action probability distribution, and the value network is used to estimate the state value; the policy network and the value network share the parameters in the initially trained multimodal early rumor detection network; Step 3: Obtain the current state of the multimodal early rumor detection network after initial training, input the state into the policy network to obtain the action and the state at the next moment after the action is executed; calculate the reward value using the reward function; construct the state-action-reward experience tuple based on the current state, action, state at the next moment after the action is executed, and reward value, and store it in the experience replay buffer. Step 4: Obtain the state-action-reward experience tuple from the experience replay buffer as input to the value network, and calculate the temporal difference error and generalized advantage estimation; based on the temporal difference error and generalized advantage estimation, construct the shearing probability ratio objective function for near-end policy optimization; Step 5: Update the policy network parameters by maximizing the shearing probability ratio objective function of the near-end policy optimization; construct the mean squared error loss function of the value network, and update the value network parameters by minimizing the mean squared error loss function of the value network. Step 6: Solve the objective function of shearing probability ratio and the mean squared error loss function in sequence, update the policy network and the value network until the preset training rounds are reached, obtain the optimal policy network parameters and the optimal value network parameters, and obtain the trained policy network.

[0133] The policy network and value network are trained using Proximal Policy Optimization (PPO). By pruning the objective function to stably update the policy, the model learns when to wait for more information and when to output a decision immediately, thereby achieving better early detection results.

[0134] After completing the POMDP modeling, the Proximal Policy Optimization (PPO) algorithm was used to train the policy network and the value network. Since early rumor detection is a strongly temporal decision problem, the dynamic changes in data and the delayed nature of rewards are quite obvious. Therefore, the stability and pruning update mechanism of PPO can effectively prevent policy collapse and improve training convergence efficiency.

[0135] Trajectory sequences are collected step-by-step in the POMDP environment, including state, action, reward, and next-time state information. A probability ratio function, advantage function, and temporal difference error are then constructed. Policy gradient optimization using a clipped objective strictly controls the update step size, preventing large policy shifts that could lead to training instability. A value network is used to evaluate the long-term benefits of the current state, providing a benchmark for advantage estimation.

[0136] The policy network and value network are trained using the Proximal Policy Optimization (PPO) algorithm. By constructing a shearing probability ratio objective function and combining advantage estimation and temporal difference error, the policy is stably updated in the dynamic data stream, thereby learning the optimal early decision timing.

[0137] In this embodiment, the expression for the objective function of the shear probability ratio is:

[0138] In the formula, The objective function represents the shear probability ratio. Expressing expectations, Represents the probability ratio. This represents the generalized advantage estimation. The clipping threshold is represented by `clip`, and the clipping function is represented by `clip`. The expression for the generalized dominance estimation is as follows:

[0139] In the formula, This represents the generalized advantage estimation. This represents the time difference error at time t. This represents the time difference error at time t+1. Indicates the discount factor. express Time difference error at any moment K represents the decay factor, and K represents the number of expansion steps; The expression for the probability ratio is:

[0140] In the formula, Indicates the current policy in state Select action The probability, This indicates the state of the strategy at the previous moment. Select action probability The expression for the time difference error is:

[0141] In the formula, Indicates the reward value. Indicates the discount factor. Indicates the next state The value function, This represents the value function.

[0142] In this embodiment, as Figure 5The structure diagram shown indicates that the policy network includes: a first fully connected layer for extracting the first hidden feature, a first activation function layer for non-linear activation, a second fully connected layer for extracting the second hidden feature, a second activation function layer for non-linear activation, and a first action output layer for outputting reinforcement learning actions, connected in sequence.

[0143] In the policy network: the final fused features extracted and fused by the multimodal early rumor detection network are received, and the final fused features are subjected to a first linear transformation to obtain a first hidden feature; the first hidden feature is subjected to a second linear transformation to obtain a second hidden feature; the second hidden feature is normalized to output the reinforcement learning action.

[0144] The expression for the first linear transformation is:

[0145] In the formula, This represents the first hidden feature. This represents the activation function. Indicates learnable parameters, Indicates the final fusion features; The expression for the second linear transformation is:

[0146] In the formula, This represents the second hidden feature. This represents the activation function. , Indicates learnable parameters; The expression for the normalization operation is:

[0147] In the formula, , Indicates scientific parameters, This represents the activation function. Represents the policy function. Indicates the action at the current moment. It indicates the state at the current moment.

[0148] During the inference phase, shared initialization parameters obtained through meta-adversarial learning are used to quickly adapt to new events. Subsequently, the trained policy network selects either a wait or decision action based on the current state. If the policy chooses a decision action, the final rumor identification result is output; otherwise, it continues to receive subsequent information and update the state to achieve adaptive early rumor warning. The initial parameters obtained through meta-adversarial learning are used during the inference phase. A rapid adaptation is performed, followed by the reinforcement learning policy network executing actions based on the current state; the expression is: For new events that arrive, initialize the model parameters based on a small number of early samples (support set). Perform a small number of gradient updates to obtain task-specific parameters. The update expression is:

[0149] In the formula, This indicates the updated parameters. Represents the original parameters. Indicates the learning rate. This represents the gradient operator. This represents the joint loss function. The fast adaptation step typically consists of only 1–5 gradient steps to ensure real-time performance and manageable computational costs.

[0150]

[0151] In the formula, Indicates the action selected at the current time step. Represents the policy network.

[0152] If the strategy selects the "determine" action, the final classification result is output, achieving adaptive early warning of rumors; if the "wait" action is selected, subsequent observations are received and the status is updated.

[0153] After the model completes meta-adversarial training and reinforcement learning policy training, it enters the inference phase for practical applications. During inference, the shared initialization parameters obtained from meta-adversarial learning are first used to quickly adapt the model to the current event, allowing the model to fine-tune according to the specific characteristics of the event. This adaptation process requires only a small number of samples and gradient steps, thus enabling rapid transfer between different events.

[0154] Subsequently, the adapted parameters are input into the policy network, which then executes a decision based on the current fusion state characteristics, choosing between "wait" or "determine". If the policy network outputs "wait", the system will continue to receive subsequent propagation data and re-execute the policy decision based on the new state; if it outputs "determine", it will immediately provide the final prediction result of whether it is a rumor or not, achieving adaptive early rumor warning.

[0155] Leveraging the rapid adaptability of the meta-learning stage and the temporal judgment capability of the PPO reinforcement learning strategy stage, this invention can dynamically and intelligently select the optimal detection timing during the inference stage, significantly improving the detection lead while ensuring accuracy, and achieving robust and efficient multimodal early rumor recognition.

[0156] The embodiments described are merely examples to clearly illustrate the present invention and are not intended to limit the implementation of the invention. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively describe all possible implementations. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A multimodal early rumor detection method based on meta-adversarial and reinforcement learning, characterized in that, Includes the following steps: S1: Obtain the data to be detected from social media platforms; S2: Construct a multimodal early rumor detection network; The multimodal early rumor detection network includes: a multimodal feature extraction module, a multimodal enhancement module, and a cross-modal interaction module; S3: Train a multimodal early rumor detection network based on the meta-adversarial algorithm and obtain the shared initialization parameters of the multimodal early rumor detection network; S4: Based on the reinforcement learning algorithm, construct and train a policy network to obtain a trained policy network; the policy network is used to output rumor time-series decision actions based on the fusion features extracted by the multimodal early rumor detection network; S5: For the data to be detected on social media platforms, the shared initialization parameters are used to quickly adapt the multimodal early rumor detection network; S6: Input the fusion features extracted by the adapted multimodal early rumor detection network into the trained policy network, and output the rumor time-series decision action; S7: If the output rumor timing decision action is to wait, continue to receive the data to be detected from the social media platform and return to step S6; otherwise, output the multimodal early rumor detection result based on the adapted multimodal early rumor detection network.

2. The multimodal early rumor detection method based on meta-adversarial and reinforcement learning according to claim 1, characterized in that, The multimodal feature extraction module includes: a first text feature extraction unit, a first image feature extraction unit, and a first graph structure feature extraction unit; The multimodal enhancement module includes: a first multi-head self-attention mechanism unit and a first channel attention (SE) unit connected in sequence; The cross-modal interaction module includes: a first interaction attention mechanism unit and a first fusion output unit connected in sequence; The outputs of the first text feature extraction unit, the first image feature extraction unit, and the first graph structure feature extraction unit are all connected to the input of the first multi-head self-attention mechanism unit; the output of the first channel attention (SE) unit is connected to the input of the first interactive attention mechanism unit.

3. The multimodal early rumor detection method based on meta-adversarial and reinforcement learning according to claim 2, characterized in that, In the first text feature extraction unit: The data to be detected on social media platforms includes: text data. ,in, This represents the Nth word in the text. N Indicates the number of lexical units. T Represents text data; text-linked image data ,in, This indicates that the text is associated with the image data. Represents the Mth image; interactive data ,in, Represents interactive data, This represents the Hth interaction action; In the first text feature extraction unit: the text data is encoded using a preset BERT encoder to obtain the text representation of the text data, expressed as: In the formula, Text representation, Indicates the BERT encoder; Based on the text representation of the text data, local feature extraction is performed using a pre-defined convolutional neural network to obtain the text dimension features, expressed as: In the formula, This represents the activation function. This represents the output feature of the i-th convolution operation. Represents the weight matrix. Represents the paranoia vector. Represents the input features of the convolution operation. This indicates the operation of taking the maximum value in the time dimension. This represents the output features of the convolution operation. This represents the output characteristics after the pooling operation. Represents the residual embedding vector. This represents the output feature after the Nth pooling operation. Represents text-dimensional features; In the first image feature extraction unit: image feature extraction is performed using a preset VGG16 model to obtain image dimensional features, expressed as: In the formula, Represents image dimensional features. Represents the VGG16 model; In the first graph structure feature extraction unit: a graph structure is constructed based on the interaction data; the graph structure is then processed using a pre-defined graph attention network to extract interaction features, resulting in graph structure dimensional features, expressed as: In the formula, This represents the original features of the i-th node in the graph structure. This represents the learnable weight matrix. Let represent the feature vector of the i-th node after the linear transformation. This represents the activation function. This indicates a splicing operation. This represents the feature vector of the j-th node after the linear transformation. This represents the original attention scores for nodes i and j. This represents the original attention score between node i and its neighbor node k. Represents an exponential function. Let i represent the set of neighboring nodes. This represents the normalized attention score. Represents a non-linear activation function. Represents a node i The graph structure dimension features.

4. The multimodal early rumor detection method based on meta-adversarial and reinforcement learning according to claim 2, characterized in that, In the first multi-head self-attention mechanism unit: multi-head self-attention operation is performed on text dimension features, image dimension features and graph structure dimension features respectively to generate initial enhanced features; The initial enhancement features include: text dimension enhancement features, initial image dimension enhancement features, and initial graph structure dimension enhancement features; The multi-head self-attention operation involves generating a query vector Q, a key vector K, and a value vector V from the input features through three linear transformations, expressed as: In the formula, Q represents the query vector, K represents the key matrix, and V represents the value matrix. Represents the attention weight matrix. Represents input features; The query vector matrix Q, key matrix K, and value matrix V are partitioned along the feature dimension into multi-head attention segments. The dot product is calculated and scaled to obtain the attention score for each head, expressed as: In the formula, Indicates the scaling factor. This represents the head attention score. Represents the normalized exponential function, This represents the query vector for the i-th attention head. This represents the key matrix of the i-th attention head. This represents the value matrix of the i-th attention head; The outputs of all attention heads are concatenated and then linearly projected to obtain the multi-head attention output. The output and input features of the multi-head attention operation are then subjected to a residual connection operation, followed by layer normalization to obtain the initial enhanced features, expressed as: In the formula, Indicates the initial enhancement features, MultiHead This indicates a bullish self-attention strategy. Presentation layer normalization operation, This indicates a random path discard operation; In the first channel attention SE unit: The initial enhancement features are segmented along the channel dimension to obtain the first initial enhancement sub-feature and the second initial enhancement sub-feature, expressed as follows: In the formula, Indicates the first initial enhancer feature. This represents the second initial enhancer feature. This indicates a channel segmentation operation; Activate the first initial enhancer feature to obtain the activated first initial enhancer feature; perform element-wise multiplication of the activated first initial enhancer feature and the second initial enhancer feature to generate the gated feature; the expression is as follows; In the formula, Indicates the first initial enhancer feature. Indicates the activation function; Performing a linear transformation on the gated features yields the transformed gated features, expressed as: In the formula, Indicates gating features, Represents the weight matrix. Represents the bias vector; The transformed gated features and the first multi-head self-attention mechanism unit are input into a preset feedforward network, and residual connection operations are performed to obtain the enhanced features, expressed as: In the formula, Indicates enhanced features, This represents a feedforward network.

5. A multimodal early rumor detection method based on meta-adversarial and reinforcement learning according to claim 2, characterized in that, In the first interactive attention mechanism unit: By utilizing any two of the text-dimensional enhancement features, image-dimensional enhancement features, and graph structure-dimensional enhancement features, a multi-head self-attention operation is performed to generate image-text fusion features, text-image fusion features, text-graph structure fusion features, graph structure-text fusion features, graph structure-image fusion features, and image-graph structure fusion features. The image-text fusion feature, text-image fusion feature, text-graph structure fusion feature, graph structure-text fusion feature, graph structure-image fusion feature, and image-graph structure fusion feature are fused together to obtain the final fused feature, which is expressed as follows: In the formula, Indicates the final fusion characteristics, Indicates the weight value. Indicates the fusion parameters; In the first fusion output unit: based on the final fusion features, a pre-defined multilayer perceptron classifier is used to obtain the rumor detection result, expressed as: In the formula, This represents the features after the first convolution operation. This represents the activation function. This represents a regularization operation. This represents the features after the second convolution operation. , , Represents the weight matrix. This represents the final two-dimensional output vector.

6. The multimodal early rumor detection method based on meta-adversarial and reinforcement learning according to claim 1, characterized in that, The steps for training a multimodal early rumor detection network based on the meta-adversarial algorithm and obtaining the shared initialization parameters of the multimodal early rumor detection network are as follows: Based on data from social media platforms, each social media event is set as a task, and N tasks are constructed. Each task is divided into a support set and a validation set; the meta-adversarial algorithm performs multi-task training with inner and outer loops. In the inner loop training, for each task in the support set, the parameters of the multimodal early rumor detection network are updated using gradient descent with a joint loss function. The update expression is: In the formula, This represents learnable initialization parameters. This represents the learning rate obtained through rapid updates. Indicates the learning rate. This represents the parameter updated after the (k+1)th iteration; In the outer loop training, for each task on the validation set, the parameters of the multimodal early rumor detection network are updated using the inner loop training for each task, and the meta-update loss is calculated; adversarial examples are generated using the projective gradient descent method, and the adversarial loss is calculated; the expression for generating adversarial examples is: In the formula, Indicates adversarial examples, This represents the task of the validation set. This represents the vector that counters the perturbation. Indicates hyperparameters, Represents a symbolic function. Represents the cross-entropy classification loss. This represents the gradient operator. Represents the forward propagation function. This represents the true label of sample x; Based on the meta-update loss and adversarial loss, a total loss function is constructed. Based on this total loss function, the parameters of the multimodal early rumor detection network are updated twice. When the total loss function converges, the shared initialization parameters of the multimodal early rumor detection network are obtained. The expression for the second update is: In the formula, This indicates the parameters after the second update. This represents learnable initialization parameters. Represents the total loss function. This represents the gradient operator.

7. The multimodal early rumor detection method based on meta-adversarial and reinforcement learning according to claim 6, characterized in that, The expression for the joint loss function is: In the formula, This represents the loss weighting coefficient. Represents the forward propagation function. Represents the cross-entropy classification loss. This represents the mean squared error loss. Representation vectors for different modes, Represents the joint loss function; The expression for the meta-update loss is: In the formula, express, This represents the loss weighting coefficient. This represents the parameters after k iterations of the inner loop. This represents the classification loss for task i. This represents the mean squared error loss for task i. Indicates the meta-update loss. N Indicates the total number of verification set tasks; The expression for the adversarial loss is: In the formula, Represents the cross-entropy classification loss. This represents the gradient operator. express, This represents the true label of sample x. This indicates the number of iterations in the projective gradient descent. Indicates resistance to loss; The expression for the total loss function shown is: In the formula, This represents the total loss function.

8. The multimodal early rumor detection method based on meta-adversarial and reinforcement learning according to claim 1, characterized in that, The process of constructing and training a policy network based on reinforcement learning algorithms to obtain a well-trained policy network is as follows: Step 1: Transform the multimodal early rumor detection problem into a reinforcement learning problem of partially observable Markov decision process; wherein, the final fused features extracted and fused by the multimodal early rumor detection network are used as the state set of reinforcement learning, and waiting, judging as a rumor, and judging as not a rumor are used as the action set of reinforcement learning, and a reward function is defined; Step 2: Construct a policy network and a value network; wherein the policy network is used to output the action probability distribution, and the value network is used to estimate the state value; the policy network and the value network share the parameters in the initially trained multimodal early rumor detection network; Step 3: Obtain the current state of the multimodal early rumor detection network after initial training, input the state into the policy network to obtain the action and the state at the next moment after the action is executed; calculate the reward value using the reward function; construct the state-action-reward experience tuple based on the current state, action, state at the next moment after the action is executed, and reward value, and store it in the experience replay buffer. Step 4: Obtain the state-action-reward experience tuple from the experience replay buffer as input to the value network, and calculate the temporal difference error and generalized advantage estimation; based on the temporal difference error and generalized advantage estimation, construct the shearing probability ratio objective function for near-end policy optimization; Step 5: Update the policy network parameters by maximizing the shearing probability ratio objective function of the near-end policy optimization; construct the mean squared error loss function of the value network, and update the value network parameters by minimizing the mean squared error loss function of the value network. Step 6: Solve the shearing probability ratio objective function and the mean squared error loss function in sequence, update the policy network and the value network until the preset training rounds are reached, and obtain the trained policy network.

9. A multimodal early rumor detection method based on meta-adversarial and reinforcement learning according to claim 8, characterized in that, The expression for the objective function of the shear probability ratio is: In the formula, The objective function represents the shear probability ratio. Expressing expectations, Represents the probability ratio. This represents the generalized advantage estimation. Indicates the shear threshold. clip Represents the shearing function; The expression for the generalized dominance estimation is as follows: In the formula, This represents the generalized advantage estimation. This represents the time difference error at time t. This represents the time difference error at time t+1. Indicates the discount factor. express Time difference error at any moment Indicates the attenuation factor. K Indicates the number of steps to unfold; The expression for the probability ratio is: In the formula, Indicates the current policy in state Select action The probability, This indicates the state of the strategy at the previous moment. Select action probability The expression for the time difference error is: In the formula, Indicates the reward value. Indicates the discount factor. Indicates the next state The value function, This represents the value function.

10. A multimodal early rumor detection method based on meta-adversarial and reinforcement learning according to claim 8, characterized in that, The policy network includes: a first fully connected layer for extracting the first hidden feature, a first activation function layer for non-linear activation, a second fully connected layer for extracting the second hidden feature, a second activation function layer for non-linear activation, and a first action output layer for outputting reinforcement learning actions, connected in sequence.

Images