Training method and apparatus for fake information detection model

Abstract

This invention provides a training method and apparatus for a fake information detection model. The method includes: acquiring image data, text data, and user comment data of target social information; extracting image feature representations, text feature representations, and user comment feature representations; obtaining text-image context information and text-user comment context information based on the image feature representations, text feature representations, and user comment feature representations, respectively; encoding the two context information sets separately; concatenating the two encoded context information sets to obtain target features; and training a deep learning network based on the target features and a cross-entropy loss function to obtain a fake information detection model. The method of this invention, by introducing user comment data, provides more valuable clues and supplementary information for subsequent models, and by retrieving and selecting key content from the information to be detected, thereby improving the detection performance of the fake information detection model.

Description

Technical Field

[0001] This invention relates to the field of machine learning technology, and in particular to a training method and apparatus for a fake information detection model. Background Technology

[0002] With the widespread popularity and increased convenience of social media platforms, and the continuous growth of the user base, social media websites have become hubs for various information. As a result, some false information maliciously distorts and fabricates facts. How to efficiently and automatically identify massive amounts of social media information has significant research and application value.

[0003] In related technologies, deep network models are usually used to detect fake information. However, in actual processing, the model splices images and text features, ignoring the fine-grained intrinsic relationships between and within multimodal information. It lacks the retrieval and screening of key information, resulting in a lack of mutual guidance and information interaction between different modal information. This leads to insufficient learning of multimodal information by the model, which in turn results in low detection performance. Summary of the Invention

[0004] This invention provides a training method and apparatus for a fake information detection model, which addresses the shortcomings of existing deep network models that do not learn multimodal information sufficiently and have low performance in detecting fake information, thereby improving the detection performance of the fake information detection model.

[0005] This invention provides a training method for a fake information detection model, comprising:

[0006] Acquire image data, text data, and user comment data corresponding to the target social information;

[0007] Extract image feature representations from the image data, extract text feature representations from the text data, and extract user comment feature representations from the user comment data;

[0008] Multimodal processing is performed on the image feature representation and the text feature representation to obtain text-image context information; multimodal processing is performed on the text feature representation and the user comment feature representation to obtain text-user comment context information.

[0009] The text-image context information and the text-user comment context information are encoded separately, and the encoded text-image context information and the encoded text-user comment context information are concatenated to obtain the target features. A deep learning network is then trained based on the target features and the cross-entropy loss function to obtain a fake information detection model.

[0010] According to the training method of the fake information detection model provided by the present invention, the step of extracting image feature representation from the image data includes:

[0011] The image feature representation is obtained by extracting average pooling convolution features from multiple regions of the image data using a visual coding network. The visual coding network is constructed based on the ResNet50 model and a self-attention mechanism.

[0012] The extraction of text feature representations from the text data includes:

[0013] The text feature representation is obtained by encoding multiple words in the text data using a text encoding network, wherein the text encoding network is built based on the pre-trained model BERT.

[0014] The extraction of user comment feature representations from the user comment data includes:

[0015] The user comment data is encoded using a comment encoding network to obtain the user comment feature representation. The comment encoding network is constructed based on the BERT model.

[0016] According to the training method of the fake information detection model provided by the present invention, the step of performing multimodal processing on the image feature representation and the text feature representation to obtain text-image context information, and performing multimodal processing on the text feature representation and the user comment feature representation to obtain text-user comment context information, includes:

[0017] Multimodal processing is performed on the image feature representation and the text feature representation based on the Gather-Transformer network to obtain the text-image context information; multimodal processing is also performed on the text feature representation and the user comment feature representation based on the Gather-Transformer network to obtain the text-user comment context information.

[0018] Among them, the dual-cooperative Gather-Transformer network is constructed based on the dual-cooperative mechanism, the self-attention mechanism, and two Transformer networks.

[0019] According to the training method of the fake information detection model provided by the present invention, the two Transformer networks include a text-comment Transformer network and a text-image Transformer network;

[0020] The multimodal processing of the image feature representation and the text feature representation based on the Gather-Transformer network to obtain the text-image context information includes:

[0021] The text-image Transformer network is used to calculate the similarity score between the image feature representation and the text feature representation to obtain the first similarity matrix;

[0022] The Gather mechanism is used to select the K elements with the highest weights among the elements in the first similarity matrix to obtain the first linear sequence, and the text-image context information is obtained based on the first linear sequence.

[0023] The multimodal processing of the text feature representation and the user comment feature representation based on the Gather-Transformer network to obtain the text-user comment context information includes:

[0024] The similarity score between the text feature representation and the user comment feature representation is calculated based on the text-comment Transformer network to obtain a second similarity matrix;

[0025] The Gather mechanism is used to select the K elements with the highest weights among the elements in the second similarity matrix to obtain a second linear sequence, and the text-image context information is obtained based on the second linear sequence, where K is a positive integer.

[0026] According to a training method for a fake information detection model provided by the present invention, the encoding of the text-image context information and the text-user comment context information respectively includes:

[0027] The text-image context information is encoded using the following formula to obtain the text-image encoded information:

[0028]

[0029] Where Q1 belongs to text-image context information, K1 and V1 belong to text-image encoding information, and P TV For the text-image context information, P TC Encode the information for the text-image;

[0030] The text-user comment context information is encoded using the following formula to obtain the text-user comment encoding information:

[0031]

[0032] Wherein, Q2 belongs to the text-image encoding information, and K1 and V1 belong to the text-image context information.

[0033] According to the training method of the fake information detection model provided by the present invention, the step of training a deep learning network based on the target features and the cross-entropy loss function to obtain the fake information detection model includes:

[0034] The target features are input into a fully connected layer to obtain the predicted probability;

[0035]

[0036] Where softmax(·) represents the activation function. Let F be the predicted probability. f Let W be the target feature, W be the weight, and b be the bias;

[0037] The target loss value is calculated based on the predicted probability and the cross-entropy loss function, and the training parameters of the deep learning network are optimized based on the target loss value to obtain the fake information detection model; wherein, the cross-entropy loss function includes:

[0038]

[0039] Where Loss is the target loss value, and N is the number of target social information items. Let y be the predicted probability of the nth social message. n This represents the true label of the nth social message.

[0040] The present invention also provides a training device for a fake information detection model, comprising:

[0041] The data acquisition module is used to acquire image data, text data, and user comment data corresponding to the target social information;

[0042] The feature extraction module is used to extract image feature representations from the image data, extract text feature representations from the text data, and extract user comment feature representations from the user comment data;

[0043] A multimodal processing module is used to perform multimodal processing on the image feature representation and the text feature representation to obtain text-image context information, and to perform multimodal processing on the text feature representation and the user comment feature representation to obtain text-user comment context information;

[0044] The training module is used to encode the text-image context information and the text-user comment context information respectively, and to concatenate the encoded text-image context information and the encoded text-user comment context information to obtain target features. Based on the target features and the cross-entropy loss function, a deep learning network is trained to obtain a fake information detection model.

[0045] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement a training method for a fake information detection model as described above.

[0046] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a training method for a fake information detection model as described above.

[0047] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements a training method for any of the above-described fake information detection models.

[0048] The training method and apparatus for the fake information detection model provided by this invention introduce user comment data on top of the image and text data corresponding to the target social information, providing more valuable clues and supplementary information for subsequent models; by extracting image feature representation, text feature representation, and user comment feature representation, text-image context information and text-user comment context information are obtained, which can improve the model's extraction of multimodal key features by retrieving and selecting key content in the information to be detected; by encoding the text-image context information and text-user comment context information respectively, and concatenating the encoded information, the target features are obtained; finally, a deep learning network is trained based on the target features and the cross-entropy loss function to obtain the fake information detection model, which can improve the detection performance of the fake information detection model. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0050] Figure 1 This is one of the flowcharts illustrating the training method for the false information detection model provided by the present invention;

[0051] Figure 2 This is the second flowchart illustrating the training method of the false information detection model provided by this invention;

[0052] Figure 3 This is a schematic diagram of the structure of the training device for the false information detection model provided by the present invention;

[0053] Figure 4This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0055] The following is combined with Figures 1-3 The present invention describes a training method and apparatus for a fake information detection model.

[0056] Figure 1 This is one of the flowcharts illustrating the training method for the fake information detection model provided by this invention, such as... Figure 1 As shown, the training method for this fake information detection model includes the following steps:

[0057] Step 110: Obtain the image data, text data, and user comment data corresponding to the target social information.

[0058] In this step, the target social information can be social data collected based on user needs, such as data crawled from forums, online communities, or other internet communities.

[0059] In this embodiment, image data can be an attached image included in one or more social messages, text data can be text content included in one or more social messages, and user comment data can be comment information from some or all users in one or more social messages.

[0060] In this embodiment, the method of selecting some important users from multiple social information can be determined based on the user's check-in level, number of views, and number of interactions (including the number of likes or replies). The higher the check-in level, number of views, or number of interactions, the more important the user's comment data is.

[0061] In this embodiment, while using the text and visual content of social media posts, corresponding important user comments are introduced to provide more valuable clues and supplementary information for the subsequent misinformation detection model, thereby improving the model's detection capability.

[0062] Step 120: Extract image feature representations from image data, extract text feature representations from text data, and extract user comment feature representations from user comment data.

[0063] In this embodiment, the image, text, and corresponding important user comments of a given social media post are input into a pre-trained deep learning network to obtain the corresponding image feature representation, text feature representation, and user comment feature representation.

[0064] In this embodiment, corresponding feature representations can be extracted from the raw data using different types of encoders. For example, image feature representations can be extracted from image data using a visual encoding network, text feature representations can be extracted from text data using a text encoding network, and user comment feature representations can be extracted from user comment data using a comment encoding network.

[0065] Step 130: Perform multimodal processing on the image feature representation and text feature representation to obtain text-image context information; perform multimodal processing on the text feature representation and user comment feature representation to obtain text-user comment context information.

[0066] In this step, the neural network constructed through the dual-cooperative mechanism can perform multimodal processing on image feature representation and text feature representation to obtain the corresponding text-image context information, and perform multimodal processing on text feature representation and user comment feature representation to obtain the corresponding text-user comment context information.

[0067] In this embodiment, the neural network includes a dual collaborative Gather Transformer network, which can effectively integrate textual, visual, and comment features from social media posts to fully capture and fuse high-order complementary information.

[0068] In this embodiment, the self-attention mechanism is indispensable in the Gather Transformer network model. It forms the core of the encoder-decoder architecture in sequence-to-sequence processing and is beneficial for establishing long-range dependencies between input and output.

[0069] For example, for an input sequence of length L The formula for the single-head self-attention mechanism is as follows:

[0070]

[0071] Where the product of softmax(·) and V represents the weighted combination of the attention scores obtained from the interaction between the query value Q and the key value K on the input; the scaling factor in the denominator It can effectively suppress the growth of dot product amplitude, thereby stabilizing the measurement standard.

[0072] In this embodiment, a multimodal Gather mechanism is added to enhance the multi-head attention architecture of the traditional Transformer network, enabling the retrieval and capture of key parts of modal information and a full understanding and extraction of fine-grained interrelationships between modalities.

[0073] Step 140: Encode the text-image context information and the text-user comment context information respectively, and concatenate the encoded text-image context information and the encoded text-user comment context information to obtain the target features. Then, train a deep learning network based on the target features and the cross-entropy loss function to obtain the fake information detection model.

[0074] In this embodiment, the two contextual information can be encoded by two symmetrically designed Cross-Attention networks to obtain the encoded contextual information.

[0075] For example, the first Cross-Attention network and the second Cross-Attention network adopt a symmetric network design structure. The input of the first Cross-Attention network includes text-image context information and text-user comment context information.

[0076] In this embodiment, the text-image context information and the text-user comment context information are further encoded through the Cross-Attention mechanism, so that they can fully cooperate, interact and enhance each other.

[0077] In this embodiment, after further encoding the two context information, the encoded context information is concatenated to obtain a multimodal representation, namely the target feature, which has a stronger representational ability.

[0078] For example, using two target features F output by two Cross-Attention modules ca1 and F ca Then, F is assembled using the following formula. ca1 and F ca2 Perform a connection to obtain the target feature F. f :

[0079] F f =Concat(F ca1 +F ca2 );

[0080] Among them, F f For the target feature, Concat is the concatenation function.

[0081] In this embodiment, after obtaining the target features, the target features are used as input to the softmax classifier, and the predicted probability of the corresponding social information is output. The cross-entropy loss function value of the deep learning network is calculated based on the predicted probability. When the cross-entropy loss converges, the trained fake information detection model is obtained.

[0082] The training method for the fake information detection model provided in this invention introduces user comment data on top of the image and text data corresponding to the target social information, providing more valuable clues and supplementary information for subsequent models. By extracting image feature representations, text feature representations, and user comment feature representations, text-image context information and text-user comment context information are obtained. This enables the model to improve the extraction of multimodal key features by retrieving and selecting key content in the information to be detected. By encoding the text-image context information and the text-user comment context information respectively, and concatenating the encoded information, the target features are obtained. Finally, a deep learning network is trained based on the target features and the cross-entropy loss function to obtain the fake information detection model, which can improve the detection performance of the fake information detection model.

[0083] In some embodiments, extracting image feature representations from image data includes: extracting average pooling convolution features from multiple regions in the image data using a visual encoding network, wherein the visual encoding network is constructed based on the ResNet50 model and a self-attention mechanism; extracting text feature representations from text data includes: encoding multiple words in the text data using a text encoding network, wherein the text encoding network is constructed based on the pre-trained model BERT; and extracting user comment feature representations from comment data includes: encoding multiple comment contents in the user comment data using a comment encoding network, wherein the comment encoding network is constructed based on the BERT model.

[0084] In this embodiment, the input to the encoding network used for feature extraction is set to multimodal news P = {T, V, C}, where T represents text content, V represents visual content, and C represents the corresponding user comment data.

[0085] In this embodiment, the pre-trained model BERT is used to extract text features for text data and user comment data; for image data, ResNet50 is used to obtain image feature representations, which can better obtain fine-grained image feature representations, text feature representations, and user comment feature representations of social media posts.

[0086] The following sections will explain how to obtain image feature representations using visual encoding networks, text feature representations using text encoding networks, and user comment feature representations using comment encoding networks:

[0087] (1) Obtain text feature representation using a text encoding network; given text content T, model T as a word sequence T = {w1, w2, ..., w...} m (where m represents the number of words in the text), the processed text features are represented as follows:

[0088] P T ={P t1 ,P t2 ,...P tm};

[0089] Among them, P ti w represents the i-th word i Features. Word representation method P ti It is calculated by the pre-trained BERT model.

[0090] P T ={P t1 ,P t2 ,...P tm} = BERT(W);

[0091] in, This refers to the hidden state of the output layer corresponding to the label in BERT, d t The dimension of word embedding.

[0092] (2) Obtain image feature representation using a visual encoding network; for a given visual content V, use a bottom-up attention pre-trained model ResNet50 to extract region features. The output of the pre-trained model is a set of region features:

[0093] P V ={P v1 ,P v2 ,...P vn};

[0094] Where n represents the number of regions in the image, and each P vj The average pooled convolutional feature is defined as the j-th region; during pre-training, the pre-trained model parameters remain fixed, and the operation of the penultimate pooling layer in the visual feature extractor can be represented as follows when processing a given visual content V:

[0095] P V ={P v1 ,P v2 ,...P vn} = ResNet50(V);

[0096] in, d v Represented as the dimension of the image embedding layer;

[0097] In this embodiment, by additionally incorporating a 2D convolutional layer, the embedding dimension d is increased. v Adjust to d t To meet the requirements of the task.

[0098] (3) Obtain user comment feature representation using a comment coding network; for a given comment content C, use the BERT model described above to extract comment text features; the given comment content is represented as C = {c1,...,c...} o (o represents the number of comments), after conversion, the comment feature representation is as follows:

[0099]

[0100] Each of them Corresponding to the h-th comment C h Features, comment features The following is calculated using the trained BERT:

[0101]

[0102] in, This is the pooling output of the hidden layer in BERT. t This indicates the dimension of the word embedding.

[0103] The training method for the fake information detection model provided in this invention uses a visual encoding network to extract average pooling convolution features from multiple regions in image data to obtain image feature representations, a text encoding network to encode multiple words in text data to obtain text feature representations, and a comment encoding network to encode multiple comment contents in user comment data to obtain user comment feature representations. This method can obtain fine-grained image and text information from social information, improving the model's accuracy in extracting multimodal key features.

[0104] In some embodiments, multimodal processing is performed on image feature representations and text feature representations to obtain text-image context information, and multimodal processing is performed on text feature representations and user comment feature representations to obtain text-user comment context information. This includes: multimodal processing of image feature representations and text feature representations based on a Gather-Transformer network to obtain text-image context information; and multimodal processing of text feature representations and user comment feature representations based on a Gather-Transformer network to obtain text-user comment context information; wherein, the dual-cooperative Gather-Transformer network is constructed based on a dual-cooperative mechanism, a self-attention mechanism, and two Transformer networks.

[0105] In this embodiment, the Gather Transformer network designed using a dual-cooperative mechanism can increase mutual guidance and collaboration among multimodal information, thereby increasing the interaction capability among multimodal information and improving the understanding of multimodal information. Moreover, the dual-cooperative Gather Transformer network improved by the Gather mechanism can fully explore and capture the fine-grained intrinsic connections within text-image context information and text-comment context information, and fuse higher-order complementary information to effectively filter out key parts of multimodal information.

[0106] In this embodiment, the two Transformer networks include a text-comment Transformer network and a text-image Transformer network. Multimodal processing of image and text feature representations using the Gather-Transformer network to obtain text-image context information includes: calculating similarity scores between image and text feature representations using the text-image Transformer network to obtain a first similarity matrix; selecting the K elements with the highest weights among all elements in the first similarity matrix using the Gather mechanism to obtain a first linear sequence, and obtaining text-image context information based on the first linear sequence. Multimodal processing of text and user comment feature representations using the Gather-Transformer network to obtain text-user comment context information includes: calculating similarity scores between text and user comment feature representations using the text-comment Transformer network to obtain a second similarity matrix; selecting the K elements with the highest weights among all elements in the second similarity matrix using the Gather mechanism to obtain a second linear sequence, and obtaining text-image context information based on the second linear sequence, where K is a positive integer.

[0107] In this embodiment, a text-image Transformer network is used to capture and retrieve fine-grained intrinsic relationships between text and image modal information. The specific process is as follows:

[0108] (1) The image features P obtained after pre-training V and text features P T Input is fed into a text-image Transformer network; in the self-attention mechanism, for multimodal input... and The first similarity matrix S1 is calculated using the following formula:

[0109]

[0110] Among them, S i,j P representsT The i-th word in the middle and P V The similarity score between the j-th regions.

[0111] (2) Expand the first similarity matrix S1 into a linear sequence s with N elements (N is the product of n and m). ’ , is represented as:

[0112] s ’ =(s0,s1,…,s N-1 );

[0113] (3) By introducing the Gather mechanism, linear sequences s are screened. ’ The K elements with the highest weights are used to obtain a new linear sequence s. g , is represented as:

[0114] s g =Gather(s);

[0115] (4) Finally, sequence s g Restored to the similarity matrix S after processing by the Gather mechanism g This allows you to obtain text-image context information.

[0116] In this embodiment, the modified single-head multimodal self-attention mechanism in the text-image Transformer network is defined as follows:

[0117]

[0118] in, and These represent different linear transformations that project the input onto the query, key, and value; (symbols) This indicates the processing operations of the Gather mechanism.

[0119] In this embodiment, a text-comment Transformer network is used to capture and retrieve fine-grained intrinsic relationships between text and user comment modal information. The specific process is as follows:

[0120] (1) The user review features P obtained after pre-training Ch and text features P T The inputs are fed into a text-comment Transformer network; in the self-attention mechanism, for multimodal inputs... and The second similarity matrix S2 is calculated using the following formula:

[0121]

[0122] Among them, Sq,p P represents T The qth word in the middle is related to P C Similarity score between the p-th comments;

[0123] (2) Expand the second similarity matrix S2 into a linear sequence, and by introducing the Gather mechanism, filter the K elements with the highest element weights in the linear sequence to obtain a new linear sequence. Finally, the new linear sequence is restored to the similarity matrix after the Gather mechanism. The steps to obtain the text-user comment context information are the same as the steps (2)-(4) above for capturing and retrieving the fine-grained intrinsic relationship between text and image modal information through the text-image Transformer network. This embodiment will not repeat them.

[0124] In this embodiment, the modified single-head multimodal self-attention mechanism in the text-comment Transformer network is defined as follows:

[0125]

[0126] in, and These represent different linear transformations that project the input onto the query, key, and value; (symbols) This indicates the processing operations of the Gather mechanism.

[0127] The training method for the fake information detection model provided in this invention performs multimodal processing on image feature representations and text feature representations through a Gather-Transformer network to obtain text-image context information; it also performs multimodal processing on text feature representations and user comment feature representations through a Gather-Transformer network to obtain text-user comment context information. This method can capture and retrieve fine-grained intrinsic relationships between text and image, as well as between text and user comment modal information, fully capturing and fusing high-order complementary information, and improving the model's ability to understand multimodal information.

[0128] In some embodiments, text-image context information and text-user comment context information are encoded separately, including: encoding the text-image context information using the following formula to obtain text-image encoded information:

[0129]

[0130] Where Q1 belongs to text-image context information, K1 and V1 belong to text-image encoding information, and P TV For text-image contextual information, P TCText-image encoding information; the text-user comment context information is encoded using the following formula to obtain the text-user comment encoded information:

[0131]

[0132] Q2 belongs to text-image encoding information, while K1 and V1 belong to text-image context information.

[0133] In this embodiment, let the output of the text-image Gather Transformer be P. TV The output of the text-comment GatherTransformer is P. Tc Further processing of P using two Cross-Attention networks TV and P Tc Encode them to be fully collaborative and interactive in order to explore the inherent relationships between text, images, and comments.

[0134] The training method for the fake information detection model provided in this invention, by encoding text-image context information and text-user comment context information, can fully obtain the inherent relationship between text, images and comments, providing data support for the target features obtained subsequently through feature concatenation.

[0135] In some embodiments, training a deep learning network based on target features and a cross-entropy loss function to obtain a false information detection model includes: inputting target features into a fully connected layer to obtain predicted probabilities;

[0136]

[0137] Where softmax(·) represents the activation function. To predict the probability, F f Let W be the target feature, W be the weight, and b be the bias. The target loss value is calculated based on the predicted probability and the cross-entropy loss function. The training parameters of the deep learning network are then optimized based on this target loss value to obtain the fake information detection model. The cross-entropy loss function includes:

[0138]

[0139] Where Loss is the target loss value, and N is the amount of target social information. Let y be the predicted probability of the nth social message. n This represents the true label of the nth social message.

[0140] In this embodiment, after acquiring the target features, the target features F are... fInput a softmax classifier (e.g., a fake information detector) to obtain the predicted probability that the corresponding social information is fake information, and use the cross-entropy loss function to calculate the loss value corresponding to the social information.

[0141] In this embodiment, two publicly available datasets, ReCOVery and MMCoVaR, were used for experiments during the actual training process. ReCOVery contains news posts about the novel coronavirus covering multimodal information from January 2020 to May 2020; MMCoVaR contains multimodal social media posts related to specific novel coronavirus vaccines, with post editing times from February 2020 to May 2021. The two datasets were first divided into training and test sets in an 8:2 ratio. For the multimodal embedding of social media posts, a pre-trained ResNet was used to extract visual features, and a pre-trained BERT model was used to extract text features from post text and comments. The image embedding dimension was 2048, and the text embedding dimension was 768.

[0142] In this embodiment, to adapt to the model, a 2D convolutional layer is introduced to convert the visual region feature dimension from 2048 to 768. The model is trained using an adaptive moment estimator (Adam) for 200 epochs with a learning rate of 0.001 and a mini-batch size of 64. Since the fake information detection task is a binary classification task, accuracy is generally used as the main evaluation metric. However, its reliability may decrease under imbalanced sample data. Therefore, in the experiment, precision, recall, and weighted F1 score are added as supplementary evaluation metrics to address the problem caused by data imbalance.

[0143] The training method for the fake information detection model provided in this embodiment of the invention obtains the prediction probability by inputting the target features into a fully connected layer, calculates the target loss value based on the prediction probability and the cross-entropy loss function, and optimizes the training parameters of the deep learning network based on the target loss value to obtain the fake information detection model, thereby improving the prediction performance of the fake information detection model.

[0144] Figure 2 This is the second flowchart illustrating the training method for the fake information detection model provided by this invention. Figure 2In the illustrated embodiment, the dual-cooperative Gather Transformer network consists of two modules: a text-image Transformer network and a text-comment Transformer network. First, the acquired image or video data is processed through feature extraction to obtain image feature representations. Text data is also processed through feature extraction to obtain text feature representations, and user comment data is processed through feature extraction to obtain user comment feature representations. Then, the image feature representations are used as input 1 and the text feature representations as input 2, respectively, and fed into the text-image Transformer network for multimodal processing to obtain text-image context information. Similarly, the user comment feature representations are used as input 1 and the text feature representations as input 2, respectively, and fed into the text-comment Transformer network for multimodal processing to obtain text-comment context information. Next, two feedforward neural networks combined with a Cross-Attention mechanism are used to encode the two context information sets. Finally, the two encoded context information sets are concatenated using a Concat function to obtain the target feature y.

[0145] The training apparatus for the fake information detection model provided by the present invention will be described below. The training apparatus for the fake information detection model described below can be referred to in correspondence with the training method for the fake information detection model described above.

[0146] Figure 3 This is a schematic diagram of the training device for the fake information detection model provided by the present invention, as shown below. Figure 3 As shown, the training device for the fake information detection model includes a data acquisition module 310, a feature extraction module 320, a multimodal processing module 330, and a training module 340.

[0147] Data acquisition module 310 is used to acquire image data, text data and user comment data corresponding to the target social information;

[0148] The feature extraction module 320 is used to extract image feature representations from image data, text feature representations from text data, and user comment feature representations from user comment data;

[0149] The multimodal processing module 330 is used to perform multimodal processing on image feature representation and text feature representation to obtain text-image context information, and to perform multimodal processing on text feature representation and user comment feature representation to obtain text-user comment context information;

[0150] Training module 340 is used to encode text-image context information and text-user comment context information respectively, and concatenate the encoded text-image context information and the encoded text-user comment context information to obtain target features. Based on the target features and the cross-entropy loss function, a deep learning network is trained to obtain a fake information detection model.

[0151] The training method for the fake information detection model provided in this invention introduces user comment data on top of the image and text data corresponding to the target social information, providing more valuable clues and supplementary information for subsequent models. By extracting image feature representations, text feature representations, and user comment feature representations, text-image context information and text-user comment context information are obtained. This enables the model to improve the extraction of multimodal key features by retrieving and selecting key content in the information to be detected. By encoding the text-image context information and the text-user comment context information respectively, and concatenating the encoded information, the target features are obtained. Finally, a deep learning network is trained based on the target features and the cross-entropy loss function to obtain the fake information detection model, which can improve the detection performance of the fake information detection model.

[0152] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 4 As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other through the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute a training method for a fake information detection model. This method includes: acquiring image data, text data, and user comment data corresponding to the target social information; extracting image feature representations from the image data, text feature representations from the text data, and user comment feature representations from the user comment data; performing multimodal processing on the image feature representations and text feature representations to obtain text-image context information, and performing multimodal processing on the text feature representations and user comment feature representations to obtain text-user comment context information; encoding the text-image context information and the text-user comment context information respectively, and concatenating the encoded text-image context information and the encoded text-user comment context information to obtain target features; and training a deep learning network based on the target features and the cross-entropy loss function to obtain a fake information detection model.

[0153] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0154] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the training method of the fake information detection model provided by the above methods. The method includes: acquiring image data, text data, and user comment data corresponding to the target social information; extracting image feature representations from the image data, extracting text feature representations from the text data, and extracting user comment feature representations from the user comment data; performing multimodal processing on the image feature representations and text feature representations to obtain text-image context information, performing multimodal processing on the text feature representations and user comment feature representations to obtain text-user comment context information; encoding the text-image context information and the text-user comment context information respectively, and concatenating the encoded text-image context information and the encoded text-user comment context information to obtain target features, and training a deep learning network based on the target features and the cross-entropy loss function to obtain a fake information detection model.

[0155] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements a training method for the fake information detection model provided by the above methods. The method includes: acquiring image data, text data, and user comment data corresponding to target social information; extracting image feature representations from the image data, extracting text feature representations from the text data, and extracting user comment feature representations from the user comment data; performing multimodal processing on the image feature representations and text feature representations to obtain text-image context information, performing multimodal processing on the text feature representations and user comment feature representations to obtain text-user comment context information; encoding the text-image context information and the text-user comment context information respectively, and concatenating the encoded text-image context information and the encoded text-user comment context information to obtain target features; and training a deep learning network based on the target features and a cross-entropy loss function to obtain a fake information detection model.

[0156] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0157] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0158] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A training method for a fake information detection model, characterized in that, include: Acquire image data, text data, and user comment data corresponding to the target social information; Extract image feature representations from the image data, extract text feature representations from the text data, and extract user comment feature representations from the user comment data; Multimodal processing is performed on the image feature representation and the text feature representation to obtain text-image context information; multimodal processing is performed on the text feature representation and the user comment feature representation to obtain text-user comment context information. The text-image context information and the text-user comment context information are encoded separately, and the encoded text-image context information and the encoded text-user comment context information are concatenated to obtain the target features. A deep learning network is trained based on the target features and the cross-entropy loss function to obtain a fake information detection model. The multimodal processing of the image feature representation and the text feature representation to obtain text-image context information, and the multimodal processing of the text feature representation and the user comment feature representation to obtain text-user comment context information, include: Multimodal processing is performed on the image feature representation and the text feature representation based on the Gather-Transformer network to obtain the text-image context information; multimodal processing is also performed on the text feature representation and the user comment feature representation based on the Gather-Transformer network to obtain the text-user comment context information. Among them, the dual-cooperative Gather-Transformer network is constructed based on the dual-cooperative mechanism, the self-attention mechanism, and two Transformer networks; The two Transformer networks include a text-comment Transformer network and a text-image Transformer network; The multimodal processing of the image feature representation and the text feature representation based on the Gather-Transformer network to obtain the text-image context information includes: The text-image Transformer network is used to calculate the similarity score between the image feature representation and the text feature representation to obtain the first similarity matrix; The Gather mechanism is used to select the K elements with the highest weights among the elements in the first similarity matrix to obtain the first linear sequence, and the text-image context information is obtained based on the first linear sequence. The multimodal processing of the text feature representation and the user comment feature representation based on the Gather-Transformer network to obtain the text-user comment context information includes: The similarity score between the text feature representation and the user comment feature representation is calculated based on the text-comment Transformer network to obtain a second similarity matrix; The Gather mechanism is used to select the K elements with the highest weights among the elements in the second similarity matrix to obtain a second linear sequence, and the text-image context information is obtained based on the second linear sequence, where K is a positive integer.

2. The training method for the fake information detection model according to claim 1, characterized in that, The step of extracting image feature representations from the image data includes: The image feature representation is obtained by extracting average pooling convolution features from multiple regions of the image data using a visual coding network. The visual coding network is constructed based on the ResNet50 model and a self-attention mechanism. The extraction of text feature representations from the text data includes: The text feature representation is obtained by encoding multiple words in the text data using a text encoding network, wherein the text encoding network is built based on the pre-trained model BERT. The extraction of user comment feature representations from the user comment data includes: The user comment data is encoded using a comment encoding network to obtain the user comment feature representation. The comment encoding network is constructed based on the BERT model.

3. The training method for the fake information detection model according to claim 1, characterized in that, The encoding of the text-image context information and the text-user comment context information includes: The text-image context information is encoded using the following formula to obtain the text-image encoded information: ； in, Belongs to text-image context information. It belongs to text-image encoded information. For the text-image context information, Encode the text-image information; The text-user comment context information is encoded using the following formula to obtain the text-user comment encoding information: ； in, Belongs to the text-image encoded information, This belongs to the text-image context information.

4. The training method for the fake information detection model according to claim 1, characterized in that, The step of training a deep learning network based on the target features and the cross-entropy loss function to obtain a fake information detection model includes: The target features are input into a fully connected layer to obtain the predicted probability; ； in, This represents the activation function. The predicted probability, For the target feature, W As weight, b For bias; The target loss value is calculated based on the predicted probability and the cross-entropy loss function, and the training parameters of the deep learning network are optimized based on the target loss value to obtain the fake information detection model; wherein, the cross-entropy loss function includes: ； in, Let N be the target loss value, and N be the number of target social information items. For the first n Predicted probability of individual social information For the first n A genuine label for social information.

5. A training apparatus for a fake information detection model, employing the training method for the fake information detection model as described in claim 1, characterized in that, include: The data acquisition module is used to acquire image data, text data, and user comment data corresponding to the target social information; The feature extraction module is used to extract image feature representations from the image data, extract text feature representations from the text data, and extract user comment feature representations from the user comment data; A multimodal processing module is used to perform multimodal processing on the image feature representation and the text feature representation to obtain text-image context information, and to perform multimodal processing on the text feature representation and the user comment feature representation to obtain text-user comment context information; The training module is used to encode the text-image context information and the text-user comment context information respectively, and to concatenate the encoded text-image context information and the encoded text-user comment context information to obtain target features. Based on the target features and the cross-entropy loss function, a deep learning network is trained to obtain a fake information detection model.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the training method for the fake information detection model as described in any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the training method of the fake information detection model as described in any one of claims 1 to 4.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the training method of the fake information detection model as described in any one of claims 1 to 4.

Images