A multi-modal rumor detection method based on modal alternation optimization

Abstract

The present application relates to the technical field of multi-modal pattern recognition, and particularly relates to a multi-modal rumor detection method based on modal alternation optimization, comprising constructing and training a multi-modal rumor detection model based on modal alternation optimization by using a training data set DS, the multi-modal rumor detection model adopts a phased training strategy, including only updating the parameters of the neural network path of the target optimization mode in a specific training phase, freezing the corresponding network parameters of the remaining modes, and dynamically adjusting the hyperparameters to regulate the training progress of different modes, and when switching between the specific training phase and the non-specific training phase, the historical task key parameters in the multi-modal fusion classifier are fixed through parameter importance evaluation; inputting the sample to be detected into the trained multi-modal rumor detection model, and outputting the authenticity probability distribution of the corresponding sample. The present application can solve the modal inertia problem existing in the existing multi-modal rumor detection technology, so as to realize the balanced optimization of the contribution degree of different modes.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal pattern recognition, and in particular to a multimodal rumor detection method based on modal alternation optimization. Background Art

[0002] In an era of information overload, the rapid adoption of social media and the internet has greatly accelerated the spread of information, but it has also exacerbated the spread of rumors. According to a research report from the Massachusetts Institute of Technology Media Lab, on Twitter, false information spreads six times faster than true information, and fake news is 70% more likely to be retweeted than true news. This rapid spread of rumors not only disrupts daily life but can also cause social panic, cause economic losses, and even threaten national security. The harmful effects of rumor spreading are particularly pronounced during emergencies or public health crises. Therefore, effectively detecting and suppressing the spread of rumors has become a critical issue that needs to be addressed.

[0003] Currently, multimodal rumor detection methods that combine multiple information sources such as text, images, and videos are gaining widespread attention in both industry and academia. However, these methods commonly suffer from the problem of modality laziness (ML). This is because in multimodal learning, certain modalities can become overly dominant, causing the features of other modalities to be neglected or degraded, resulting in "pseudo-multimodal" decisions. Once the dominant modality's information is biased or missing, the detection accuracy of the entire model plummets, severely weakening the reliability and robustness of rumor detection. Summary of the Invention

[0004] The purpose of the present invention is to propose a multimodal rumor detection method based on modal alternation optimization to solve the modal inertia problem existing in the existing multimodal rumor detection technology proposed in the background technology, thereby achieving balanced optimization of the contribution of different modalities.

[0005] To achieve this object, the present invention adopts the following technical solutions:

[0006] A multimodal rumor detection method based on modal alternation optimization includes the following steps:

[0007] Step A: Obtain a multimodal dataset published on social media platforms, including images, text, and social information, annotate the authenticity of the multimodal dataset, and perform data preprocessing operations to form a training dataset DS;

[0008] Step B: Construct and use the training dataset DS to train a multimodal rumor detection model with alternating modality optimization. The multimodal rumor detection model adopts a staged training strategy, including only optimizing the target modality M in a specific training stage. tThe neural network pathway of the training model is updated, the corresponding network parameters of the remaining modalities are frozen, and the hyperparameters are dynamically adjusted to control the training progress of different modalities. When switching between specific training phases and non-specific training phases, the key parameters of historical tasks in the multimodal fusion classifier are fixed through parameter importance evaluation.

[0009] Step C: Input the sample to be detected into the trained multimodal rumor detection model and output the authenticity probability distribution of the corresponding sample.

[0010] Preferably, the step B specifically includes the following steps:

[0011] Step B1: Determine the target optimization mode M for the current training stage t t , read the image data I in the training data set DS t , text data T t and social data t As the input of the multimodal rumor detection model, the multimodal rumor detection model includes an image encoder DeiT, a text encoder BERT and a sentiment analyzer ALBERT, wherein the image encoder DeiT is used to extract image data I t Image features in The text encoder BERT is used to extract text data T t Text features in

[0012] Step B2: The sentiment analyzer ALBERT first analyzes the text data T t Emotional characteristics Then count the text data T t The number of special symbols in the text forms the statistical features of the text Emotional characteristics Text statistical features and social data t Splicing to get social features

[0013] Step B3: Deeply fuse image features using cross-modal attention mechanism Text features and social characteristics Three modal features get multimodal features Use multimodal fusion classifier to classify multimodal features Perform classification to obtain multimodal classification results Three independent image classifiers, text classifiers and social classifiers are used to classify image features respectively. Text features and social characteristics The three modal features are classified to obtain the single modal classification result and

[0014] Step B4: Based on the multimodal classification results Unimodal classification results and Calculating classification loss and Evaluate the importance of each parameter in the multimodal fusion classifier to the historical task and determine the regularization term of the loss function accordingly Combined classification loss and the loss function regularization term Get the final loss function L t ;

[0015] Step B5: Based on the multimodal classification results Unimodal classification results and Dynamically calculate the target optimization mode M t The learning rate α t , for the target optimization mode M t The neural network pathway of the modalities is updated, and the corresponding network parameters of the remaining modalities are frozen. After the update is completed, the next training phase is entered to repeat the above steps B1-B4.

[0016] Preferably, the step B1 specifically includes the following steps:

[0017] Step B11: Determine the target optimization mode M of the current training stage t through modular operation t , which is expressed as follows:

[0018]

[0019] Where T represents the total number of training steps, mod represents the modulo operation, I, T, and S represent the image modality, text modality, and social modality, respectively;

[0020] Step B12: Read a batch of multimodal data from the training dataset DS as the input of the multimodal rumor detection model in the current training phase, including image data I t , text data T t and social data t , where social data S t This includes the number of users’ followers, followings, verification status, number of post likes, reposts, comments, and geographic location;

[0021] Step B13: Load the pre-trained image encoder DeiT and convert the image data I t Send it to the image encoder DeiT, and output image features after processing It is expressed as follows:

[0022]

[0023] Step B14: Translate the text data T t The word is divided into subword units according to the WordPiece algorithm and mapped into a sequence of integer IDs, which are represented as follows:

[0024]

[0025]

[0026] in is the WordPiece word segmentation function, represents sequence concatenation, L=512 represents the maximum input length of the BERT model, Truncate(·,L) represents truncating the sequence to the first L-2 words, |S| represents the length of the sentence, V -1 (·) represents the inverse mapping function of the BERT model vocabulary, and its corresponding integer ID can be obtained through the subword lookup table;

[0027] When the ID sequence length is greater than L, it is truncated to L bits. Otherwise, it is padded with 0 elements to L bits. The processed sequence is input into the pre-trained text encoder BERT, and the text features are output after processing. It is expressed as follows:

[0028]

[0029] Where pad_id represents the filler, Indicates sequence splicing, X 1:L It means to truncate bits 1 to L of X, and |X| represents the length of X.

[0030] Preferably, the step B2 specifically includes the following steps:

[0031] Step B21: Referring to step B14, use the pre-trained sentiment analyzer ALBERT to analyze the text data T t Emotional characteristics It is expressed as follows:

[0032]

[0033] Step B22: Define a special symbol set S, which is expressed as follows:

[0034] S={! ,? ,@,#, / ,url,(),"",...} (8)

[0035] Among them, !, ?, @, #, / , url, (), "" and ... represent exclamation point, question mark, user mention, pound sign, slash, URL, bracket, double quote and ellipsis respectively;

[0036] Initialize character counter Traverse the text data T t Each character t in i , if t i If it is in the set S, the corresponding counter is incremented by one, as shown below:

[0037] C[t i ]+=1If t i ∈S (9)

[0038] Vectorize the character counter C to obtain text statistical features

[0039] Step B23: Emotional features Text statistical features and social data t Splicing to get social features It is expressed as follows:

[0040]

[0041] Where Concat(·) represents the tensor concatenation operation.

[0042] Preferably, the step B3 specifically includes the following steps:

[0043] Step B31: Generate image features Text features and social characteristics The query, key and value of each modality i∈{I,T,S} are defined as follows:

[0044]

[0045] Where j represents other modalities different from modality i, Q, K and V represent query vector, key vector and value vector respectively, and W q 、W k and W v The projection matrices representing the query vector, key vector, and value vector respectively;

[0046] Step B32: Obtain the enhanced features E of the three modalities i I 、E T and E S :

[0047] Calculate the attention score of modality i∈{I,T,S}, taking image modality I as an example, it is expressed as follows:

[0048]

[0049] where d K represents the key vector dimension, softmax represents the normalization of the dot product result, α I Indicates the degree of attention of image modality I to each feature in modalities T and S, Represents tensor concatenation;

[0050] Extract the contextual information of other modalities and connect it with the feature residual of the original modality. Taking image modality I as an example, it is expressed as follows:

[0051]

[0052] E I =V I +C I (14)

[0053] Among them C I V represents the context information extracted from the image modality I by focusing on the important features of the text modality T and the social modality S, I represents the characteristics of the original mode I, E I represents the enhanced feature representation of modality I;

[0054] Step B33: Enhance the features E of the three modalities i I 、E T and E S Splice and project to the target dimension to obtain the final multimodal fusion features It is expressed as follows:

[0055]

[0056] Where W o represents the projection matrix, b o Indicates bias, Represents tensor concatenation;

[0057] Step B34: Use the multimodal fusion classifier to classify the multimodal fusion features The classification is as follows:

[0058]

[0059] Among them, MLP M Represents a multimodal fusion classifier, which is composed of a multi-layer perceptron. Softmax represents converting the classifier results into a probability distribution. represents the probability distribution of rumor authenticity obtained by multimodal fusion feature classification at the current training stage t;

[0060] Step B35: Use three independent classifiers to classify and The three modal features are classified to obtain the single modal classification result and It is expressed as follows:

[0061]

[0062] Among them, MLP I 、MLP T and MLP S Represents independent classifiers for image, text, and social modes, respectively. They are composed of multi-layer perceptrons. Softmax represents the conversion of classifier results into probability distributions. and They represent the probability distribution of rumor authenticity obtained by using three types of unimodal feature classification at the current training stage t.

[0063] Preferably, the step B4 specifically includes the following steps:

[0064] Step B41: Use the cross entropy loss function to calculate the multimodal classification results separately Unimodal classification results and The loss value is expressed as follows:

[0065]

[0066] Where N represents the number of samples, R represents the probability distribution of the rumor's authenticity calculated in the previous steps, represents the true label, and Respectively represent the multimodal classification results at the current training stage t Multimodal classification loss, unimodal classification results and The corresponding loss value;

[0067] Step B42: Calculate the multimodal fusion classifier MLP M The Fisher information matrix F of the parameters i , which is expressed as follows:

[0068]

[0069] where θ i Represents the multimodal fusion classifier MLP M The i-th parameter, L t-1 Represents the multimodal fusion classifier MLP M The loss function in the previous training phase is, Represents the loss function for the parameter θi The gradient of, D represents the data set of the previous training stage, N represents the number of samples in the data set D, F i represents the parameter θ i The corresponding Fisher information matrix;

[0070] Step B43: Using the Fisher information matrix F i Calculating the regularization term of the loss function in the multimodal rumor detection model It is expressed as follows:

[0071]

[0072] where θ i Represents the multimodal fusion classifier MLP M The i-th parameter in the current training stage, Represents the multimodal fusion classifier MLP M The i-th parameter in the previous training phase, λ, represents the hyperparameter that balances the learning in the old and new training phases;

[0073] Step B44: Combine multimodal classification losses Objective to optimize modality classification loss and regularization term Get the final loss function L of the current training stage t t , which is expressed as follows:

[0074]

[0075] Where α is the balance and The hyperparameters of the two losses, the target optimization mode classification loss in the current training stage t And the target optimization mode M t Calculated and determined by formula (1).

[0076] Preferably, the step B5 specifically includes the following steps:

[0077] Step B51: Based on the unimodal classification results and Dynamically calculate the target optimization mode M of the current training stage t t The learning rate α t , which is expressed as follows:

[0078]

[0079]

[0080] where α base =0.001 represents the baseline learning rate, Denotes the mode M defined by formula (28) t The classification accuracy of Indicates non-M t The accuracy of the modality, Represents mode M t The total number of samples, 1(A) is the indicator function, the value is 1 when A is true, otherwise it is 0, Indicates that the i-th sample is correctly classified;

[0081] Step B52: After blocking the gradient of the non-target modality neural network pathway, perform directed backpropagation, that is, return the gradient signal only through the target modality pathway and adjust its exclusive parameter W t Perform update iterations, as follows:

[0082] W t =SGD(W t-1 ,L t ,α t ) (29)

[0083] Where SGD means using stochastic gradient descent to update parameters, W t-1 Indicates the parameters updated in the previous training phase;

[0084] Step B53: Enter the next training phase and repeat steps B1 to B5 until a preset stop condition is met.

[0085] One of the above technical solutions has the following beneficial effects:

[0086] (1) To address the modal inertia problem in multimodal rumor detection, this paper innovatively proposes an alternating gradient update mechanism for modality-specific parameters. By establishing a cyclic modal optimization sequence, it ensures that each modal encoder obtains balanced optimization opportunities. This mechanism effectively overcomes the representation degradation problem caused by gradient coupling in traditional joint training, allowing the discriminative features of the three modalities of image, text, and social to be fully integrated into the multimodal decision space, thereby improving the overall performance of the multimodal rumor detection model.

[0087] (2) The present invention designs an adaptive learning rate adjustment strategy based on classification accuracy comparison. By monitoring the performance differences of independent classifiers for each modality in real time, a learning rate reinforcement mechanism is implemented for the weaker modality. This negative feedback regulation method breaks through the limitations of the traditional fixed learning rate, forms a dynamic balance between modalities during the parameter update process, promotes the collaborative optimization of cross-modal representations, and further alleviates the problem of modal inertia.

[0088] (3) The present invention introduces an incremental regularization method based on the Fisher information matrix to construct a flexible constraint mechanism when switching between modal training phases. This method quantitatively evaluates the importance of multimodal fusion classifier parameters to historical training tasks and imposes flexible constraints on key parameters when optimizing the current modality, thereby effectively preventing the multimodal fusion classifier from forgetting the acquired knowledge when switching between training phases. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] Figure 1 This is a schematic flow chart of the steps of a multimodal rumor detection method based on modal alternation optimization of the present invention;

[0090] Figure 2 This is a structural diagram of a multimodal rumor detection model and a schematic diagram of a phased training process in a multimodal rumor detection method based on modal alternation optimization of the present invention. DETAILED DESCRIPTION

[0091] The technical solution of the present invention will be further described below with reference to the accompanying drawings and through specific implementation methods.

[0092] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "axial", "radial", "circumferential" and the like to indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the present invention.

[0093] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature specified as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "plurality" means two or more.

[0094] In the description of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood broadly. For example, they may refer to fixed connections, detachable connections, or integral connections; they may refer to direct connections, indirect connections through an intermediary, or internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on the specific circumstances.

[0095] like Figure 1As shown, this embodiment provides a multimodal rumor detection method based on modal alternation optimization, comprising the following steps:

[0096] Step A: Obtain a multimodal dataset published on social media platforms, including images, text, and social information, annotate the authenticity of the multimodal dataset, and perform data preprocessing operations to form a training dataset DS;

[0097] Step B: Construct and use the training dataset DS to train a multimodal rumor detection model with alternating modality optimization. The multimodal rumor detection model adopts a staged training strategy, including only optimizing the target modality M in a specific training stage. t The neural network pathway of the training model is updated, the corresponding network parameters of the remaining modalities are frozen, and the hyperparameters are dynamically adjusted to control the training progress of different modalities. When switching between specific training phases and non-specific training phases, the key parameters of historical tasks in the multimodal fusion classifier are fixed through parameter importance evaluation.

[0098] Step C: Input the sample to be detected into the trained multimodal rumor detection model and output the authenticity probability distribution of the corresponding sample.

[0099] Figure 2 This is a structural diagram of the multimodal rumor detection model based on modal alternation optimization and a schematic diagram of the staged training process in this embodiment.

[0100] To further illustrate, the specific implementation steps of step B are as follows:

[0101] Step B1: Determine the target optimization mode M for the current training stage t t , read the image data I in the training data set DS t , text data T t and social data t As the input of the multimodal rumor detection model, the multimodal rumor detection model includes an image encoder DeiT, a text encoder BERT and a sentiment analyzer ALBERT, wherein the image encoder DeiT is used to extract image data I t Image features in The text encoder BERT is used to extract text data T t Text features in

[0102] In this embodiment, step B1 specifically includes the following steps:

[0103] Step B11: Determine the target optimization mode M of the current training stage t through modular operation t , which is expressed as follows:

[0104]

[0105] Where T represents the total number of training steps, mod represents the modulo operation, I, T, and S represent the image modality, text modality, and social modality, respectively;

[0106] Step B12: Read a batch of multimodal data from the training dataset DS as the input of the multimodal rumor detection model in the current training phase, including image data I t , text data T t and social data t , where social data S t This includes the number of users’ followers, followings, verification status, number of post likes, reposts, comments, and geographic location;

[0107] Step B13: Load the pre-trained image encoder DeiT and convert the image data I t Send it to the image encoder DeiT, and output image features after processing It is expressed as follows:

[0108]

[0109] Step B14: Translate the text data T t The word is divided into subword units according to the WordPiece algorithm and mapped into a sequence of integer IDs, which are represented as follows:

[0110]

[0111]

[0112] in is the WordPiece word segmentation function, represents sequence concatenation, L=512 represents the maximum input length of the BERT model, Truncate(·,L) represents truncating the sequence to the first L-2 words, |S| represents the length of the sentence, V -1 (·) represents the inverse mapping function of the BERT model vocabulary, and its corresponding integer ID can be obtained through the subword lookup table;

[0113] When the ID sequence length is greater than L, it is truncated to L bits, otherwise it is padded with 0 elements to L bits. The processed sequence is input into the pre-trained text encoder BERT, and the text feature FT is output after processing. t , which is expressed as follows:

[0114]

[0115]

[0116] Where pad_id represents the filler, Indicates sequence splicing, X 1:L It means to truncate bits 1 to L of X, and |X| represents the length of X.

[0117] Step B2: The sentiment analyzer ALBERT first analyzes the text data T t Emotional characteristics Then count the text data T t The number of special symbols in the text forms the statistical features of the text Emotional characteristics Text statistical features and social data t Splicing to get social features

[0118] In this embodiment, step B2 specifically includes the following steps:

[0119] Step B21: Referring to step B14, use the pre-trained sentiment analyzer ALBERT to analyze the text data T t Emotional characteristics It is expressed as follows:

[0120]

[0121] Step B22: Define a special symbol set S, which is expressed as follows:

[0122]

[0123] Among them, !, ?, @, #, / , url, (), "" and ... represent exclamation point, question mark, user mention, pound sign, slash, URL, bracket, double quote and ellipsis respectively;

[0124] Initialize character counter Traverse the text data T t Each character t in i , if t i If it is in the set S, the corresponding counter is incremented by one, as shown below:

[0125] C[t i ]+=1If t i ∈S (9)

[0126] Vectorize the character counter C to obtain text statistical features

[0127] Step B23: Emotional features Text statistical features and social data t Splicing to get social features It is expressed as follows:

[0128]

[0129] Among them Conca t (·) represents a tensor concatenation operation.

[0130] Step B3: Deeply fuse image features using cross-modal attention mechanism Text features and social characteristics Three modal features get multimodal features Use multimodal fusion classifier to classify multimodal features Perform classification to obtain multimodal classification results Three independent image classifiers, text classifiers and social classifiers are used to classify image features respectively. Text features and social characteristics The three modal features are classified to obtain the single modal classification result and

[0131] In this embodiment, step B3 specifically includes the following steps:

[0132] Step B31: Generate image features Text features and social characteristics The query, key and value of each modality i∈{I,T,S} are defined as follows:

[0133]

[0134] Where j represents other modalities different from modality i, Q, K and V represent query vector, key vector and value vector respectively, and W q 、W k and W v The projection matrices representing the query vector, key vector, and value vector respectively;

[0135] Step B32: Obtain the enhanced features E of the three modalities i I 、E T and E S :

[0136] Calculate the attention score of modality i∈{I,T,S}, taking image modality I as an example, it is expressed as follows:

[0137]

[0138] where d k represents the key vector dimension, softmax represents the normalization of the dot product result, α Irepresents the attention degree of the image modality I to each feature in modalities T and S, represents tensor concatenation;

[0139] extract the context information of other modalities and residual connect with the original modality feature, taking the image modality I as an example, represented as follows:

[0140]

[0141] where C I represents the context information extracted by the image modality I by paying attention to the important features of the text modality T and the social modality S, V I represents the feature of the original modality I, E I represents the feature representation of the enhanced modality I;

[0142] Step B33: Concatenate the three modality i enhanced features E I , E T and E S and project to the target dimension to obtain the final multi-modal fusion feature represented as follows:

[0143]

[0144] where W o represents the projection matrix, b o represents the bias, represents tensor concatenation;

[0145] Step B34: Use the multi-modal fusion classifier to classify the multi-modal fusion feature represented as follows:

[0146]

[0147] where MLP M represents the multi-modal fusion classifier composed of multiple layers of perceptron, and softmax represents converting the classifier result into a probability distribution, represents the rumor truth probability distribution obtained by classifying the multi-modal fusion feature at the current training stage t;

[0148] Step B35: Use three modal independent classifiers to classify and three modal features to obtain single modal classification results and represented as follows:

[0149]

[0150] where MLP I , MLPT and MLP S Represents independent classifiers for image, text, and social modes, respectively. They are composed of multi-layer perceptrons. Softmax represents the conversion of classifier results into probability distributions. and They represent the probability distribution of rumor authenticity obtained by using three types of unimodal feature classification at the current training stage t.

[0151] Step B4: Based on the multimodal classification results Unimodal classification results and Calculating classification loss and Evaluate the importance of each parameter in the multimodal fusion classifier to the historical task and determine the regularization term of the loss function accordingly Combined classification loss and loss function regularization term Get the final loss function L t ;

[0152] In this embodiment, step B4 specifically includes the following steps:

[0153] Step B41: Use the cross entropy loss function to calculate the multimodal classification results separately Unimodal classification results and The loss value is expressed as follows:

[0154]

[0155]

[0156]

[0157]

[0158] Where N represents the number of samples, R represents the probability distribution of the rumor's authenticity calculated in the previous steps, represents the true label, and Respectively represent the multimodal classification results at the current training stage t Multimodal classification loss, unimodal classification results and The corresponding loss value;

[0159] Step B42: Calculate the multimodal fusion classifier MLP M The Fisher information matrix F of the parameters i , which is expressed as follows:

[0160]

[0161] where θ i Represents the multimodal fusion classifier MLP M The i-th parameter, L t-1 Represents the multimodal fusion classifier MLP M The loss function in the previous training phase is, Represents the loss function for the parameter θ i The gradient of, D represents the data set of the previous training stage, N represents the number of samples in the data set D, F i represents the parameter θ i The corresponding Fisher information matrix;

[0162] Step B43: Using the Fisher information matrix F i Calculating the regularization term of the loss function in the multimodal rumor detection model It is expressed as follows:

[0163]

[0164] where θ i Represents the multimodal fusion classifier MLP M The i-th parameter in the current training stage, Represents the multimodal fusion classifier MLP M The i-th parameter in the previous training phase, λ, represents the hyperparameter that balances the learning in the old and new training phases;

[0165] Step B44: Combine multimodal classification losses Objective to optimize modality classification loss and regularization term Get the final loss function L of the current training stage t t , which is expressed as follows:

[0166]

[0167] Where α is the balance and The hyperparameters of the two losses, the target optimization mode classification loss in the current training stage t And the target optimization mode M t Calculated and determined by formula (1).

[0168] Step B5: Based on the multimodal classification results Unimodal classification results and Dynamically calculate the target optimization mode M t The learning rate α t , for the target optimization mode M tThe neural network pathway of the modalities is updated, and the corresponding network parameters of the remaining modalities are frozen. After the update is completed, the next training phase is entered to repeat the above steps B1-B4.

[0169] In this embodiment, step B5 specifically includes the following steps:

[0170] Step B51: Based on the unimodal classification results and Dynamically calculate the target optimization mode M of the current training stage t t The learning rate α t , which is expressed as follows:

[0171]

[0172] where α base =0.001 represents the baseline learning rate, Denotes the mode M defined by formula (28) t The classification accuracy of Indicates non-M t The accuracy of the modality, Represents mode M t The total number of samples, 1(A) is the indicator function, the value is 1 when A is true, otherwise it is 0, Indicates that the i-th sample is correctly classified;

[0173] Step B52: After blocking the gradient of the non-target modality neural network pathway, perform directed backpropagation, that is, return the gradient signal only through the target modality pathway and adjust its exclusive parameter W t Perform update iterations, as follows:

[0174] W t =SGD(W t-1 ,L t ,α t ) (29)

[0175] Where SGD means using stochastic gradient descent to update parameters, W t-1 Indicates the parameters updated in the previous training phase;

[0176] Step B53: Enter the next training phase and repeat steps B1 to B5 until a preset stop condition is met.

[0177] The technical principles of the present invention have been described above with reference to specific embodiments. These descriptions are intended solely to illustrate the principles of the present invention and are not to be construed in any way as limiting the scope of protection of the present invention. Based on the explanations herein, those skilled in the art will be able to devise other specific embodiments of the present invention without inventive effort, and such equivalent variations or substitutions are intended to be encompassed within the scope of the claims of this application.

Claims

1. A multimodal rumor detection method based on modal alternation optimization, characterized in that: The following steps are involved: Step A: Obtain a multimodal dataset published on social media platforms, including images, text, and social information, annotate the authenticity of the multimodal dataset, and perform data preprocessing operations to form a training dataset DS; Step B: Construct and use the training dataset DS to train a multimodal rumor detection model with modality alternating optimization. The multimodal rumor detection model adopts a staged training strategy, which includes: Deeply fuse image features using cross-modal attention mechanism , text features and social characteristics Three modal features get multimodal features , use multimodal fusion classifier to classify multimodal features Perform classification to obtain multimodal classification results , and use three independent modal image classifiers, text classifiers and social classifiers to classify image features respectively , text features and social characteristics The three modal features are classified to obtain the single modal classification result 、 and ; Based on the unimodal classification results 、 and Dynamically calculate the current training stage Target optimization mode The learning rate , which is expressed as follows: in represents the baseline learning rate, represents the mode defined by formula (28) The classification accuracy of Indicates non The accuracy of the modality, Indicates modality The total number of samples, is the indicator function, when If true, the value is 1, otherwise it is 0. Indicates the samples are correctly classified; After blocking the gradient of the non-target modality neural network pathway, directed backpropagation is performed, that is, the gradient signal is only returned through the target modality pathway, and its exclusive parameters are Perform update iterations, as follows: in Indicates that the parameters are updated using stochastic gradient descent. Indicates the parameters updated in the previous training phase; Enter the next training phase and repeat the above steps until the preset stopping condition is met; Step C: Input the sample to be detected into the trained multimodal rumor detection model and output the authenticity probability distribution of the corresponding sample.

2. A multimodal rumor detection method based on modal alternation optimization according to claim 1, characterized in that: The step B specifically comprises the following steps: Step B1: Determine the current training stage The target optimization mode , read the image data in the training dataset DS , text data and social data As the input of the multimodal rumor detection model, the multimodal rumor detection model includes an image encoder DeiT, a text encoder BERT and a sentiment analyzer ALBERT, wherein the image encoder DeiT is used to extract image data Image features in , the text encoder BERT is used to extract text data Text features in ; Step B2: The sentiment analyzer ALBERT first analyzes the text data Emotional characteristics , and then count the text data The number of special symbols in the text forms the statistical features of the text , the emotional features , text statistical features and social data Splicing to get social features ; Step B3: Deeply fuse image features using cross-modal attention mechanism , text features and social characteristics Three modal features get multimodal features , use multimodal fusion classifier to classify multimodal features Perform classification to obtain multimodal classification results , and use three independent modal image classifiers, text classifiers and social classifiers to classify image features respectively , text features and social characteristics The three modal features are classified to obtain the single modal classification result 、 and ; Step B4: Based on the multimodal classification results , unimodal classification results 、 and Calculating classification loss 、 、 and , evaluate the importance of each parameter in the multimodal fusion classifier to the historical task and determine the regularization term of the loss function accordingly , combined with classification loss 、 、 、 and loss function regularization term Get the final loss function ; Step B5: Based on the multimodal classification results , unimodal classification results 、 and Dynamically calculate target optimization mode The learning rate , for the target optimization mode The neural network pathway of the modalities is updated, and the corresponding network parameters of the remaining modalities are frozen. After the update is completed, the next training phase is entered to repeat the above steps B1-B4.

3. A multimodal rumor detection method based on modal alternation optimization according to claim 2, characterized in that: The step B1 specifically includes the following steps: Step B11: Determine the current training stage through modular operation The target optimization mode , which is expressed as follows: in represents the total number of training steps, Represents the modulo operation, They represent image modality, text modality, and social modality respectively; Step B12: Read a batch of multimodal data from the training dataset DS as the input of the multimodal rumor detection model in the current training phase, including image data , text data and social data , including social data This includes the number of users’ followers, followings, verification status, number of post likes, reposts, comments, and geographic location; Step B13: Load the pre-trained image encoder DeiT and convert the image data Send it to the image encoder DeiT, and output image features after processing , which is expressed as follows: Step B14: Convert text data The word is divided into subword units according to the WordPiece algorithm and mapped into a sequence of integer IDs, which are represented as follows: in is the WordPiece word segmentation function, Indicates sequence splicing, Indicates the maximum input length of the BERT model, Indicates truncating the sequence to the beginning words, Indicates the length of the sentence, Represents the inverse mapping function of the BERT model vocabulary, which can be obtained by looking up the subword table to obtain its corresponding integer ID; When the ID sequence length is greater than When truncated to Otherwise, fill with 0 elements. The processed sequence is input into the pre-trained text encoder BERT, and the text features are output after processing. , which is expressed as follows: in Indicates filler, Indicates sequence splicing, Indicates interception 1 to Bit, express length.

4. The multimodal rumor detection method based on modal alternation optimization according to claim 3 is characterized in that: The step B2 specifically includes the following steps: Step B21: Refer to step B14 and use the pre-trained sentiment analyzer ALBERT to analyze the text data Emotional characteristics , which is expressed as follows: Step B22: Define a set of special symbols , which is expressed as follows: in Represented as exclamation point, question mark, user mention symbol, pound sign, slash, URL, brackets, double quotes and ellipsis respectively; Initialize character counter , traverse the text data Each character in ,like In a collection The corresponding counter is incremented by one, as shown below: Set the character counter Vectorization to obtain text statistical features ; Step B23: Emotional features , text statistical features and social data Splicing to get social features , which is expressed as follows: in Represents a tensor concatenation operation.

5. The multimodal rumor detection method based on modal alternation optimization according to claim 4 is characterized in that: The step B3 specifically includes the following steps: Step B31: Generate image features , text features and social characteristics The query, key and value for each modality ,definition: in Representation and Modality Different other modes, represent query vector, key vector and value vector respectively, 、 and The projection matrices representing the query vector, key vector, and value vector respectively; Step B32: Get three modalities Enhanced features : Calculation mode The attention score of image modality I is expressed as follows: in represents the key vector dimension, Indicates that the dot product result is normalized. Indicates the degree of attention of image modality I to each feature in modalities T and S, Represents tensor concatenation; Extract the context information of other modalities and connect it with the residual feature of the original modality to form an image modality For example, it is expressed as follows: in represents the contextual information extracted by the image modality I by focusing on the important features of the text modality T and the social modality S, represents the characteristics of the original mode I, represents the enhanced feature representation of modality I; Step B33: Combine the three modes Enhanced features Splice and project to the target dimension to obtain the final multimodal fusion features , which is expressed as follows: in represents the projection matrix, Indicates bias, Represents tensor concatenation; Step B34: Use the multimodal fusion classifier to classify the multimodal fusion features The classification is as follows: in Represents a multimodal fusion classifier, which is composed of a multi-layer perceptron. Softmax represents converting the classifier results into a probability distribution. Indicates the current training stage The probability distribution of rumor authenticity obtained by multimodal fusion feature classification; Step B35: Use three independent classifiers to classify 、 and The three modal features are classified to obtain the single modal classification result 、 and , which is expressed as follows: in 、 and Represents independent classifiers for image, text, and social modes, respectively. They are composed of multi-layer perceptrons. Softmax represents the conversion of classifier results into probability distributions. 、 and Represents the current training stage The probability distribution of rumor authenticity obtained by using three unimodal features for classification.

6. A multimodal rumor detection method based on modal alternation optimization according to claim 5, characterized in that: The step B4 specifically includes the following steps: Step B41: Use the cross entropy loss function to calculate the multimodal classification results separately , unimodal classification results 、 and The loss value is expressed as follows: in represents the number of samples, represents the probability distribution of rumor authenticity calculated in the above steps, represents the true label, 、 、 and Represents the current training stage Multimodal classification results Multimodal classification loss, unimodal classification results 、 and The corresponding loss value; Step B42: Calculate the multimodal fusion classifier The Fisher information matrix of the parameters , which is expressed as follows: in Represents a multimodal fusion classifier No. parameters, Represents a multimodal fusion classifier The loss function in the previous training phase is, Represents the loss function for parameters The gradient, represents the data set of the previous training stage, Representation dataset The number of samples in Representation parameters The corresponding Fisher information matrix; Step B43: Using the Fisher Information Matrix Calculating the regularization term of the loss function in the multimodal rumor detection model , which is expressed as follows: in Represents a multimodal fusion classifier In the current training phase parameters, Represents a multimodal fusion classifier In the previous training phase parameters, represents the hyperparameters learned during the balancing of the old and new training phases; Step B44: Combine multimodal classification losses , target optimization modality classification loss and regularization term Get the current training stage The final loss function , which is expressed as follows: in For balance and Hyperparameters of the two losses, current training stage The objective is to optimize the modality classification loss , and the target optimization mode Calculated and determined by formula (1).

Images