False information detection method based on multimodal context hierarchical and step-by-step alignment

Abstract

The present invention discloses a false information detection method and system based on multimodal context hierarchical step-by-step alignment, obtains images and texts in social media blogs; establishes a local alignment module with bidirectional cross-modal attention, and infers the matching relationship at the fragment level; establishes a global context fusion entity feature extraction module, uses a multi-head self-attention mechanism to help local objects learn more contextual semantics, and enhances the contextual representation of image and text features; designs an adaptive weight filtering module to perform image-text entity feature alignment, and integrates all weighted features to achieve global matching and suppress overall semantic deviation according to the similarity of the two modal enhancement features; inputs the overall similarity of the two different levels of fusion entity features into a binary classifier to obtain the result of false information detection. By integrating context information, the present invention can retain the complete semantic information of text and image and improve the accuracy of false information detection.

Description

Technical Field

[0001] The present invention relates to a false information detection method and system based on multimodal context hierarchical and step-by-step alignment, specifically a method for detecting the matching of image and text information in social media blog posts, belonging to the technical field of false news detection. Background Art

[0002] With the increasing importance of social media, many bloggers resort to exaggeration to increase the spread of their blog posts. However, this type of blog posts are often full of misleading information. Existing technologies use various algorithms, such as keyword detection, user behavior analysis, and machine learning models to detect false information in social media, but these methods have certain defects. First, keyword detection methods may produce false positives or false negatives depending on the context; user behavior analysis is limited by data acquisition and is easily restricted by privacy protection policies; although machine learning models have been proven to be effective in recent years, traditional machine learning methods often cannot handle subtle inconsistencies between image and text information well, which leads to their limited ability to detect fine-grained deceptive information.

[0003] For example, an example in the PHEME dataset shows a social media post that distorts factual information. In this example, the image shows a scene of a "woman holding a child", and the accompanying text description is "This woman has given birth to 14 children." It can be found that these two different forms of information have significant inconsistencies in describing the same topic "children". Further, from the exaggerated text description, it can be inferred that this blog post contains false information. Therefore, it is crucial to perform fine image-text context matching to ensure correct information alignment and thus improve the accuracy of identifying false information. Summary of the invention

[0004] Purpose of the invention: In view of the two problems existing in the existing fine-grained matching method, that is, some information words in the text cannot strictly correspond to a certain area in the image and ignore the context information, and the matching network over-emphasizes the fine-grained alignment, which will mislead the overall understanding of the image and ignore the alignment of the context, the present invention provides a false information detection method and system based on multimodal context hierarchical and step-by-step alignment. First, the image and text features in the social media blog are extracted independently, the image area features are extracted by the Faster R-CNN network based on ResNet-101, and the text vocabulary features are extracted by the Bi-GRU network. Secondly, in order to strictly correspond to the image area and the text vocabulary, the similarity of the image area and the text area is calculated using a bidirectional cross-attention mechanism to obtain the local fragment-level matching results between the visual area and the text vocabulary. Then, in order to avoid the matching network over-emphasizing fine-grainedness and ignoring the alignment of the context, the multi-head self-attention mechanism is used to enhance the basic features of the image and text in combination with the context information, and the enhanced features of the image and text are obtained. Through the adaptive weight filtering module, the global context matching results of the image and text are obtained. Finally, a fully connected layer with a softmax function is used to add the result vectors of the local and global stages and classify them into two results: true information or false information.

[0005] Technical solution: A false information detection method based on multimodal context hierarchical and step-by-step alignment, comprising the following steps:

[0006] (1) Extraction of basic features of images and texts: Using the object detection model and language representation model, the features of images and texts in social media posts are extracted to obtain basic feature vectors of visual areas and text words respectively.

[0007] (2) Image and text weighted feature extraction: for the image and text basic feature vectors extracted in step (1), a bidirectional cross-attention mechanism is used to align the same segment in different modalities, and then the text vocabulary is used as a reference to obtain the weighted sum of the basic feature vectors of the image region describing the vocabulary as the weighted feature vector of the vocabulary; similarly, the image region is used as a reference to obtain the weighted sum of the basic feature vectors of the text vocabulary describing the region as the weighted feature vector of the region;

[0008] (3) Image-text feature local matching: for the image and text basic feature vectors extracted in step (1) and the image and text weighted feature vectors extracted in step (2), similarity matching is used to obtain local segment-level matching results;

[0009] (4) Image and text enhancement feature extraction: For the image and text basic feature vectors extracted in step (1), a multi-head self-attention mechanism is used to explore the contextual relationship between the features, and image and text enhancement feature vectors that incorporate contextual information are obtained respectively;

[0010] (5) Image-text enhancement feature matching: for the image and text enhancement features extracted in step (4), an adaptive weight filtering module is used to obtain a global correlation matching result;

[0011] (6) Social media blog post detection: the local segment-level matching result obtained in step (3) and the global correlation matching result obtained in step (5) are input into a binary classifier. The fully connected layer with a softmax function in the binary classifier is used to project the sum vector of the two-level result vectors into the target space of true information and false information, and the detection result of social media blog posts is obtained.

[0012] In (1), an image X and a text Y are defined. First, the basic features of the image region are extracted using Faster R-CNN based on Resnet-101, and then the basic features of the text vocabulary are extracted using the Bi-GRU network.

[0013] Furthermore, in (1), the specific steps of extracting image region features and text vocabulary features are as follows:

[0014] (1.1) Extraction of basic features of image regions: For an image X, Faster R-CNN is selected to extract image region features, and the first m regions are selected to represent X according to the scores of the image regions. Then, a pre-trained ResNet-101 is used to extract a feature vector f for each image region. i , i∈[1,m]. Finally, the fully connected layer is used to convert it into a 1024-dimensional feature vector, and finally the feature vector is represented as a set of regional features {x1,x2,...,x m} as the basic features of the image region, X∈R d ×m As shown in formula (1):

[0015] x i =W x f i +b x (1)

[0016] (1.2) Extraction of basic features of text vocabulary: For a sentence Y, first encode each word into a one-hot vector g t , t∈[1,n], and through the parameter matrix W g and the bias vector b g Embedded into a 300-dimensional vector o t As shown in formula (2):

[0017] o t =W g g t+b g (2)

[0018] Next, we use Bi-GRU to capture contextual information in the sentence from both the front and back directions and integrate it into the text embedding, and then get the final word feature y by averaging t As the basic features of text vocabulary, as shown in formula (3):

[0019]

[0020] In (2), the basic feature vector of the image region extracted in step (1.1) and the basic feature vector of the text vocabulary extracted in step (1.2) are used to obtain a multimodal weighted feature vector of the image region and the text vocabulary using a bidirectional cross-attention mechanism.

[0021] Furthermore, in (2), the specific steps of extracting the weighted feature vectors of the image region and text vocabulary are as follows:

[0022] (2.1) Calculate the similarity matrix between the image region and the text vocabulary: as shown in formula (4):

[0023] A=(W x X)(W y Y) T (4)

[0024] Where W x and W y is a learnable parameter, X={x1,x2,...,x m} and Y={y1,y2,...,y n} are the basic features of the image region and the basic features of the text vocabulary extracted in step (1.1) and step (1.2), respectively, A∈R m×n is the image region-text vocabulary similarity matrix, A it represents the semantic similarity between the ith region and the tth word.

[0025] (2.2) Image region weighted feature extraction: Take a certain image region as a reference, assign weights to n words, and then obtain a weighted feature based on the image region x by weighted combination of the n words. i The text vocabulary features y i * , the process is shown in formula (5):

[0026]

[0027] Where λ is the temperature parameter of the softmax function.

[0028] (2.3) Text vocabulary weighted feature extraction: Similarly, a text vocabulary is used as a benchmark to obtain a feature based on the text vocabulary y. t The corresponding image region feature x t * , the process is shown in formula (6):

[0029]

[0030] In (3), the basic feature vectors of the image and text extracted in step (1) and the weighted feature vectors of the image and text extracted in step (2) are matched at the segment level, and on this basis, the segment level matching scores for the complete image and the complete text are obtained.

[0031] Furthermore, in (3), the specific steps for calculating the segment-level similarity are as follows:

[0032] (3.1) Calculate the fragment-level matching score related to the image region: For the image region x i The text vocabulary feature y corresponding to the region extracted from (2.2) i * , calculate the similarity As shown in formula (7):

[0033]

[0034] (3.2) Calculate the segment-level matching score related to the text vocabulary: Similarly, for the text vocabulary y t And the image region feature x corresponding to the word extracted in (2.3) t * , calculate the similarity As shown in formula (8):

[0035]

[0036] (3.3) Calculate the segment-level matching score S of the complete image and the complete text segment (X, Y), as shown in formula (9):

[0037]

[0038] where η is a hyperparameter used to balance the region-dependent segment-level matching score S x (x i ,y i * ) and the vocabulary-related segment-level matching score S y (y t ,x t * )’s contribution.

[0039] In (4), the basic feature vectors of the image and text extracted in step (1) are used to explore the contextual relationship between the features using a multi-head self-attention mechanism to obtain enhanced feature vectors of the image and text that incorporate contextual information. The specific steps are as follows:

[0040] (4.1) Image enhancement feature extraction: For an image X, a set of regional features X = {x1, x2, ..., x m}, using the multi-head self-attention mechanism to explore the contextual relationship between regional features. Specifically, we first define the query Q X , key K X Sum value Val X , as shown in formula (10):

[0041] Q X =XW Q ,K X =XW K ,Val X =XW Val (10)

[0042] Where W Q ,W K ,W Val is a learnable feature mapping matrix. Further, the value Val is calculated by formula (11) X The weighted sum of:

[0043]

[0044] where d k is the scaling factor. The weight matrix Q X K X T is a square matrix that encodes the relationship between each visual feature and all other features through the inner product layer. The relationship between different objects is then encoded using formula (12):

[0045]

[0046] Where j represents the jth head, The W in the formula represents a learnable feature mapping matrix, and the superscript Q X Indicates that this matrix is ​​for image query, where Q represents query, x represents image, and subscript j represents the jth head. Therefore, Refers to the query weight matrix specifically used to process the image input on the jth head in the multi-head attention mechanism. Similarly, and denote the key weight matrix and value weight matrix specifically used to process the image input on the jth head.

[0047] Finally, the i-th original visual region feature x i It is transformed into enhanced features x by combining global objects and local guided objects. ip , as shown in formula (13):

[0048]

[0049] Where W p is a learnable weight matrix, and h is the number of heads. Then for an image, the set of regional features X = {x1, x2, ..., x m The enhanced feature of} can be expressed as X reinforcement ={x 1p ,x 2p ,...,x mp}.

[0050] (4.2) Text Enhanced Feature Extraction: Similarly, for a piece of text Y, the text features Y extracted in step (1.2) are Y = {y1, y2, ..., y n}, perform the same process as step (4.1) to obtain the text enhancement feature Y reinforcement ={y 1p ,y 2p ,...,y np}.

[0051] (4.3) Weighted Enhanced Feature Extraction: Similarly, for the weighted text feature X extracted in step (2), * ={x1 * ,x2 * ,...,x n *} and image features Y * ={y1 * ,y2 * ,...,y m *}Perform the same process as in step (4.1) to obtain the enhanced weighted features, which are represented as X reinforcement * ={x 1p * ,x 2p * ,...,x np *},Y reinforcement * ={y 1p * ,y 2p * ,...,y mp*}.

[0052] In the above (5), the image and text enhancement features extracted in step (4) are filtered using an adaptive weight filtering module to obtain a global correlation matching result. The specific steps are as follows:

[0053] (5.1) Calculate the global matching score from image to text: Substitute the enhanced basic feature pair (X reinforcement ,Y reinforcement ) and enhanced weighted feature pairs (Y reinforcement * ,X reinforcement * ) as input, the global semantics is generated by weighted fusion of regional features, and then the global semantic similarity between the two modalities is calculated to obtain the global matching score s in the image-to-text direction x→y , as shown in formula (14):

[0054]

[0055] where w x =cos(x lp ,y lp * ) is the adaptive weight, which represents the enhancement of basic features x lp and enhanced weighted features y lp * The importance of.

[0056] (5.2) Calculate the global matching score from text to image: Similarly, enhance the basic feature pair (X reinforcement ,Y reinforcement ) and enhanced weighted feature pairs (Y reinforcement * ,X reinforcement * ) as input, the global semantics is generated by weighted fusion of regional features, and then the global semantic similarity between the two modalities is calculated to obtain the global matching score s in the image-to-text direction y→x , as shown in formula (15):

[0057]

[0058] Similar to (5.1), w y = cos(y lp ,x lp * ) is the adaptive weight, which means enhancing the basic feature y lp and the enhanced weighted features x lp * The importance of.

[0059] (5.3) Calculate the global matching score S of the complete image and the complete text global (X, Y), as shown in formula (16):

[0060] S global (X,Y)=s x→y +s y→x (16)

[0061] In (6), the local segment-level matching result obtained in step (3) and the global matching result obtained in step (5) are input into a binary classifier, and the sum vector of the result vectors of the two levels is projected into two target spaces of true information and false information using a fully connected layer with a softmax function in the binary classifier, and the detection result of the social media blog post is obtained. The specific steps are as follows:

[0062] (6.1) After obtaining the local region-word matching results and the global matching results, a fully connected layer with a softmax function is used to transform S segment (X,Y) and S global (X, Y) is projected into the target space with only two categories (real or fake), and its probability distribution is obtained as shown in formula (17):

[0063]

[0064] Where p = [p0, p1] represents the predicted probability vector, p0 and p1 represent the probabilities of the predicted results being 0 (real) and 1 (fake), respectively. W is the weight matrix, and b is the bias term.

[0065] (6.2) For each blog post, the goal is to minimize the binary cross entropy loss function to distinguish whether the blog post is true information or false information. The loss function is shown in formula (18):

[0066] L p = -[ylnp0+(1-y)lnp1] (18)

[0067] Where y∈{0,1} is the authenticity label of each blog post, y=0 means that the blog post is true information, and y=1 means that the blog post contains false information.

[0068] A false information detection system based on multimodal context hierarchical and step-by-step alignment, including five modules: image region feature extraction, text vocabulary feature extraction, local alignment module, global context alignment module, and social media blog post detection;

[0069] The image region feature extraction module first uses Faster R-CNN to detect salient regions, then uses the pre-trained ResNet-101 to extract a feature vector for each image region, and finally uses a fully connected layer to convert it into a feature vector to obtain an image region feature vector;

[0070] The text vocabulary feature extraction module first encodes each word into a one-hot vector and embeds it into the vector o through the parameter matrix and the bias vector. t Then, Bi-GRU is used to capture the contextual information in the sentence from both the front and back directions and integrate it into the text embedding to obtain the text vocabulary feature vector.

[0071] The local alignment module first obtains the fragment-level cross-modal weighted feature vector through a bidirectional cross-attention mechanism, then calculates the region-related fragment-level matching score and the vocabulary-related fragment-level matching score, and finally averages all region-based fragment-level matching scores and vocabulary-based fragment-level matching scores to obtain the final fragment-level matching score of the complete image and text.

[0072] The global context alignment module first uses a multi-head self-attention mechanism to obtain the contextual relationship between fragment-level features and obtain an enhanced feature vector that incorporates contextual information. It then calculates the global matching scores in the image-to-text and text-to-image directions respectively, and finally takes the sum of the global matching scores in the above two directions as the global matching score of the complete image and text.

[0073] The social media blog post detection module inputs the segment-level similarity and global similarity between the image and the text into a fully connected layer with a softmax function to obtain a detection result of whether the social media blog post is true information or false information.

[0074] The implementation process of the system is the same as the implementation process of the method, and will not be repeated here.

[0075] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the false information detection method based on multimodal context hierarchical and step-by-step alignment as described above is implemented.

[0076] A computer-readable storage medium stores a computer program for executing the false information detection method based on multimodal context hierarchical and step-by-step alignment as described above.

[0077] Beneficial effects: For false information that uses exaggeration to fabricate events, the model ignores the importance of contextual information when modeling multi-modal fine-grained correspondences and the model cannot capture the alignment of different modal contexts. The present invention adopts a two-stage strategy to detect false information. In the first stage, a local alignment module with bidirectional cross-modal attention is established to infer the fragment-level matching relationship by adding the region-vocabulary similarities in different directions. In the second stage, a global context alignment module is established, which uses a multi-head self-attention mechanism to help local objects learn more contextual semantics and enhance the contextual representation of vision and text. Then, through the adaptive weight filtering module, according to the similarity of the two modal enhancement features, global matching is achieved by integrating all weighted features to suppress overall semantic deviation. Finally, the two different levels of result vectors are integrated into a classifier with a fully connected layer with a softmax function to classify the blog post into two results: true information or false information. The false information detection model obtained by the method of the present invention can effectively obtain accurate detection results. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 A method framework diagram of an embodiment of the present invention;

[0079] Figure 2 This is a flow chart of the bidirectional cross-attention mechanism of an embodiment of the present invention. DETAILED DESCRIPTION

[0080] The present invention is further explained below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, various equivalent forms of modifications to the present invention by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0081] like Figure 1 As shown, the false information detection method based on multimodal context hierarchical and step-by-step alignment disclosed in the embodiment of the present invention specifically includes the following steps:

[0082] (1) Extraction of basic features of images and texts: Define an image X and a text Y. First, use Faster R-CNN based on Resnet-101 to extract basic features of the image region, and then use the Bi-GRU network to extract basic features of text vocabulary. The specific steps for extracting image region features and text vocabulary features are as follows:

[0083] (1.1) Extraction of basic features of image regions: For an image X, Faster R-CNN is used to extract image region features, and the first m regions are selected to represent X based on their scores. Then, a pre-trained ResNet-101 is used to extract a feature vector f for each image region. i, i∈[1,m]. Finally, the fully connected layer is used to convert it into a 1024-dimensional feature vector, which is finally represented as a set of regional features {x1,x2,...,x m}, X∈R d×m As shown in formula (1):

[0084] x i =W x f i +b x (1)

[0085] (1.2) Extraction of basic features of text vocabulary: For a sentence Y, first encode each word into a one-hot vector g t , t∈[1,n], and through the parameter matrix W g and the bias vector b g Embedded into a 300-dimensional vector o t As shown in formula (2):

[0086] o t =W g g t +b g (2)

[0087] Next, we use Bi-GRU to capture contextual information in the sentence from both the front and back directions and integrate it into the text embedding, and then get the final word feature y by averaging t , as shown in formula (3):

[0088]

[0089] (2) Image and text weighted feature extraction: The basic feature vector of the image region extracted in step (1.1) and the basic feature vector of the text vocabulary extracted in step (1.2) are used to obtain the multimodal weighted feature vector of the image region and the text vocabulary using a bidirectional cross attention mechanism, where the bidirectional cross attention mechanism is as follows: Figure 2 The specific steps for extracting weighted feature vectors of image regions and text words are as follows:

[0090] (2.1) Calculate the similarity matrix between the image region and the text vocabulary: as shown in formula (4):

[0091] A=(W x X)(W y Y) T (4)

[0092] Where W x and W y is a learnable parameter, A∈R m×n is the image region-text vocabulary similarity matrix, A itrepresents the semantic similarity between the ith region and the tth word.

[0093] (2.2) Image region weighted feature extraction: Take a certain image region as a reference, assign weights to n words, and then obtain a weighted combination of these n words based on the image region x. i The text vocabulary features y i * , the process is shown in formula (5):

[0094]

[0095] Where λ is the temperature parameter of the softmax function.

[0096] (2.3) Text vocabulary weighted feature extraction: Similarly, a text vocabulary is used as a benchmark to obtain a feature based on the text vocabulary y. t The corresponding image region feature x t * , the process is shown in formula (6):

[0097]

[0098] (3) Local matching of image-text features: Perform segment-level matching on the image and text basic feature vectors extracted in step (1) and the image and text weighted feature vectors extracted in step (2), and obtain segment-level matching scores for the complete image and complete text on this basis. The specific steps for calculating segment-level similarity are as follows:

[0099] (3.1) Calculate the fragment-level matching score related to the image region: For the image region x i The text vocabulary feature y corresponding to the region extracted from (2.2) i * , calculate the similarity As shown in formula (7):

[0100]

[0101] (3.2) Calculate the segment-level matching score related to the text vocabulary: Similarly, for the text vocabulary y t And the image region feature x corresponding to the word extracted in (2.3) t * , calculate the similarity As shown in formula (8):

[0102]

[0103] (3.3) Calculate the segment-level matching score S of the complete image and the complete text segment(X, Y), as shown in formula (9):

[0104]

[0105] where η is a hyperparameter used to balance the region-dependent segment-level matching score S x (x i ,y i * ) and the vocabulary-related segment-level matching score S y (y t ,x t * )’s contribution.

[0106] (4) Image and text enhanced feature extraction: The image and text basic feature vectors extracted in step (1) are used to explore the contextual relationship between features using a multi-head self-attention mechanism to obtain image and text enhanced feature vectors that incorporate contextual information. The specific steps are as follows:

[0107] (4.1) Image enhancement feature extraction: For an image X, a set of regional features X = {x1, x2, ..., x m}, using the multi-head self-attention mechanism to explore the contextual relationship between regional features. Specifically, we first define the query Q X , key K X Sum value Val X , as shown in formula (10):

[0108] Q X =XW Q ,K X =XW K ,Val X =XW Val (10)

[0109] Where W Q ,W K ,W Val is a learnable feature mapping matrix. Further, the value Val is calculated by formula (11) X The weighted sum of:

[0110]

[0111] where d k is the scaling factor. The weight matrix Q X K X T is a square matrix that encodes the relationship between each visual feature and all other features through the inner product layer. The relationship between different objects is then encoded using formula (12):

[0112]

[0113] Where j represents the jth head. Finally, the i-th original visual region feature x i It is transformed into enhanced features by combining global objects and local guide objects, such as x ip As shown in formula (13):

[0114] x ip =Concat(R1+R2+...+R h )W p +x i (13)

[0115] Where W p is a learnable weight matrix, and h is the number of heads. Then for an image, the set of regional features X = {x1, x2, ..., x m The enhanced feature of} can be expressed as X reinforcement ={x 1p ,x 2p ,...,x mp}.

[0116] (4.2) Text Enhanced Feature Extraction: Similarly, for a piece of text Y, the text features extracted in step (1.2) are Y = {y1, y2, ..., y n}Perform the same process as step (4.1) to obtain the text enhancement feature Y reinforcement ={y 1p ,y 2p ,...,y np}.

[0117] (4.3) Weighted Enhanced Feature Extraction: Similarly, for the weighted text feature X extracted in step (2), * ={x1 * ,x2 * ,...,x n *} and image features Y * ={y1 * ,y2 * ,...,y m *}Perform the same process as in step (4.1) to obtain the enhanced weighted features, which are represented as X reinforcement * ={x 1p * ,x 2p * ,...,x np *},Y reinforcement* ={y 1p * ,y 2p * ,...,y mp *}.

[0118] (5) Image-text enhanced feature matching: The image and text enhanced features extracted in step (4) are used to obtain global correlation matching results using an adaptive weight filtering module. The specific steps are as follows:

[0119] (5.1) Calculate the global matching score from image to text: Substitute the enhanced basic feature pair (X reinforcement ,Y reinforcement ) and enhanced weighted feature pairs (Y reinforcement * ,X reinforcement * ) as input, the global semantics is generated by weighted fusion of regional features, and then the global semantic similarity between the two modalities is calculated to obtain the global matching score s in the image-to-text direction x→y , as shown in formula (14):

[0120]

[0121] where w x =cos(x lp ,y lp * ) is the adaptive weight, which represents the enhancement of basic features x lp and enhanced weighted features y lp * The importance of.

[0122] (5.2) Calculate the global matching score from text to image: Similarly, enhance the basic feature pair (X reinforcement ,Y reinforcement ) and enhanced weighted feature pairs (Y reinforcement * ,X reinforcement * ) as input, the global semantics is generated by weighted fusion of regional features, and then the global semantic similarity between the two modalities is calculated to obtain the global matching score s in the image-to-text direction y→x , as shown in formula (15):

[0123]

[0124] (5.3) Calculate the global matching score S of the complete image and the complete text global (X, Y), as shown in formula (16):

[0125] S global (X,Y)=s x→y +s y→x (16)

[0126] (6) Social media blog post detection: The local segment-level matching results obtained in step (3) and the global matching results obtained in step (5) are input into a binary classifier. The fully connected layer with a softmax function in the binary classifier is used to project the sum vector of the two-level result vectors into the two target spaces of true information and false information, and the detection results of social media blog posts are obtained. The specific steps are as follows:

[0127] (6.1) After obtaining the local region-word matching results and the global matching results, a fully connected layer with a softmax function is used to transform S segment (X,Y) and S global (X, Y) is projected into the target space with only two categories (real or fake), and its probability distribution is obtained as shown in formula (17):

[0128]

[0129] Where p = [p0, p1] represents the predicted probability vector, p0 and p1 represent the probabilities of the predicted results being 0 (real) and 1 (fake), respectively. W is the weight matrix, and b is the bias term.

[0130] (6.2) For each blog post, the goal is to minimize the binary cross entropy loss function, as shown in formula (18):

[0131] L p = -[ylogp0+(1-y)logp1] (18)

[0132] Where y∈{0,1} represents the true value.

[0133] A false information detection system based on multimodal context hierarchical and step-by-step alignment, including five modules: image region feature extraction, text vocabulary feature extraction, local alignment module, global context alignment module, and social media blog post detection;

[0134] The image region feature extraction module first uses Faster R-CNN to detect salient regions, then uses the pre-trained ResNet-101 to extract a feature vector for each image region, and finally uses a fully connected layer to convert it into a 1024-dimensional feature vector to obtain the image region feature vector;

[0135] The text vocabulary feature extraction module first encodes each word into a one-hot vector and embeds it into a 300-dimensional vector through a parameter matrix and a bias vector. Then, Bi-GRU is used to capture the contextual information in the sentence from both the front and back directions and integrate it into the text embedding to obtain the text vocabulary feature vector.

[0136] The local alignment module first obtains the fragment-level cross-modal weighted feature vector through a bidirectional cross-attention mechanism, then calculates the region-related fragment-level matching score and the vocabulary-related fragment-level matching score, and finally averages all region-based fragment-level matching scores and vocabulary-based fragment-level matching scores to obtain the final fragment-level matching score of the complete image and text.

[0137] The global context alignment module first uses a multi-head self-attention mechanism to obtain the contextual relationship between fragment-level features and obtain an enhanced feature vector that incorporates contextual information. It then calculates the global matching scores in the image-to-text and text-to-image directions respectively. Finally, the sum of the global matching scores in the above two directions is taken as the global matching score of the complete image and text.

[0138] The social media post detection module inputs the segment-level similarity and global similarity between image and text into a fully connected layer with a softmax function to obtain the detection result of whether the social media post is true or false information.

[0139] Obviously, those skilled in the art should understand that the various steps of the false information detection method based on multimodal context hierarchical step-by-step alignment or the various modules of the false information detection system based on multimodal context hierarchical step-by-step alignment of the above-mentioned embodiments of the present invention can be implemented with a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, optionally, they can be implemented with program codes executable by the computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in an order different from that herein, or they can be made into individual integrated circuit modules respectively, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present invention are not limited to any specific combination of hardware and software.

[0140] The parameter settings and experimental evaluation criteria are as follows:

[0141] A parameter settings:

[0142] The following are several parameters that affect this model: Epoch, Batch Size, Network Learning Rate, and Learnable Linear Projection Function Hidden Layer Parameter η.

[0143] Table 1 Model training parameter settings

[0144]

[0145]

[0146] BEvaluation index:

[0147] The evaluation indicators involved are a combination of indicators uniformly used by existing methods, including: Accuracy, Precision, Recall and F1-Score.

[0148] According to the above embodiments, in the case of false information that uses exaggeration to fabricate events, the importance of context information is ignored when the model models the multi-modal fine-grained correspondence relationship, and the model cannot capture the alignment of different modal contexts.

[0149] A two-step detection strategy is proposed. The present invention uses a local alignment module to learn complementary information between multiple modalities, which can handle the problem that some information in the text cannot strictly correspond to the image area. The present invention uses a global context alignment module to achieve global matching of images and texts, which can solve the problem that the matching network is misled by image semantics and ignores context alignment, and improve the accuracy of false information detection.

Claims

1. A false information detection method based on multimodal context hierarchical and step-by-step alignment, characterized in that: The steps include: (1) Extraction of basic features of images and texts: Using the object detection model and language representation model, the features of images and texts in social media posts are extracted to obtain basic feature vectors of visual areas and text words respectively. (2) Image and text weighted feature extraction: for the image and text basic feature vectors extracted in step (1), a bidirectional cross-attention mechanism is used to align the same segment in different modalities, and then the text vocabulary is used as a reference to obtain the weighted sum of the basic feature vectors of the image region describing the vocabulary as the weighted feature vector of the vocabulary; similarly, the image region is used as a reference to obtain the weighted sum of the basic feature vectors of the text vocabulary describing the region as the weighted feature vector of the region; (3) Image-text feature local matching: for the image and text basic feature vectors extracted in step (1) and the image and text weighted feature vectors extracted in step (2), similarity matching is used to obtain local segment-level matching results; (4) Image and text enhancement feature extraction: For the image and text basic feature vectors extracted in step (1), a multi-head self-attention mechanism is used to explore the contextual relationship between the features, and image and text enhancement feature vectors that incorporate contextual information are obtained respectively; (5) Image-text enhancement feature matching: for the image and text enhancement features extracted in step (4), an adaptive weight filtering module is used to obtain a global correlation matching result; (6) Social media blog post detection: the local segment-level matching result obtained in step (3) and the global correlation matching result obtained in step (5) are input into a binary classifier, and the sum vector of the result vectors of the two levels is projected into the target space of true information and false information using the fully connected layer with a softmax function in the binary classifier, and the detection result of the social media blog post is obtained; In (1), an image X and a text Y are defined, and the basic features of the image region are first extracted, and then the basic features of the text vocabulary are extracted; The specific steps of extracting image region features and text vocabulary features are as follows: (1.1) Extraction of basic features of image regions: For an image X, Faster R-CNN is used to extract image region features, and the first m regions are selected to represent X according to the scores of the image regions; then a pre-trained ResNet-101 is used to extract a feature vector f for each image region. i , i∈[1,m]; finally, the feature vector is transformed by the fully connected layer, and finally the feature vector is represented as a set of regional features {x1,x2,...,x m } as the basic features of the image region; (1.2) Extraction of basic features of text vocabulary: For a sentence Y, first encode each word into a one-hot vector g t , t∈[1,n], and through the parameter matrix W g and the bias vector b g Embedded into vector o t middle; Next, we use Bi-GRU to capture contextual information in the sentence from both the front and back directions and integrate it into the text embedding, and then get the final word feature y by averaging t As a basic feature of text vocabulary; In (2), the basic feature vector of the image region and the basic feature vector of the text vocabulary are used to obtain a multimodal weighted feature vector of the image region and the text vocabulary using a bidirectional cross attention mechanism; The specific steps for extracting weighted feature vectors of image regions and text words are as follows: (2.1) Calculate the similarity matrix between image regions and text words; (2.2) Image region weighted feature extraction: Take a certain image region as a reference, assign weights to n words, and then obtain a weighted feature based on the image region x by weighted combination of the n words. i The text vocabulary features y i * ; ( 2.3) Text vocabulary weighted feature extraction: Similarly, a text vocabulary is used as a benchmark to obtain a feature based on the text vocabulary y t The corresponding image region feature x t * .

2. According to claim 1, the false information detection method based on multimodal context hierarchical and step-by-step alignment is characterized in that: In (1), an image X and a text Y are defined. First, the basic features of the image region are extracted using Faster R-CNN based on Resnet-101, and then the basic features of the text vocabulary are extracted using the Bi-GRU network. The specific steps of extracting image region features and text vocabulary features are as follows: (1.1) Extraction of basic features of image regions: For an image X, Faster R-CNN is used to extract image region features, and the first m regions are selected to represent X according to the scores of the image regions; then a pre-trained ResNet-101 is used to extract a feature vector f for each image region. i , i∈[1,m]; finally, the fully connected layer is used to convert it into a 1024-dimensional feature vector, and finally the feature vector is represented as a set of regional features {x1,x2,...,x m } as the basic features of the image region, X∈R d×m , as shown in formula (1): x i =W x f i +b x (1) (1.2) Extraction of basic features of text vocabulary: For a sentence Y, first encode each word into a one-hot vector g t , t∈[1,n], and through the parameter matrix W g and the bias vector b g Embedded into a 300-dimensional vector o t As shown in formula (2): o t =W g g t +b g (2) Next, we use Bi-GRU to capture contextual information in the sentence from both the front and back directions and integrate it into the text embedding, and then get the final word feature y by averaging t As the basic features of text vocabulary, as shown in formula (3):

3. The false information detection method based on multimodal context hierarchical and step-by-step alignment according to claim 2 is characterized in that: In (2), the basic feature vector of the image region and the basic feature vector of the text vocabulary are used to obtain a multimodal weighted feature vector of the image region and the text vocabulary using a bidirectional cross attention mechanism; The specific steps for extracting weighted feature vectors of image regions and text words are as follows: (2.1) Calculate the similarity matrix between the image region and the text vocabulary: as shown in formula (4): A=(W x X)(W y Y) T (4) Where W x and W y is a learnable parameter, X={x1,x2,...,x m } and Y={y1,y2,...,y n } are the basic features of the image region and the basic features of the text vocabulary extracted in step (1.1) and step (1.2), respectively, A∈R m×n is the image region-text vocabulary similarity matrix, A it Represents the semantic similarity between the i-th region and the t-th word; (2.2) Image region weighted feature extraction: Take a certain image region as a reference, assign weights to n words, and then obtain a weighted feature based on the image region x by weighted combination of the n words. i The text vocabulary features y i * , the process is shown in formula (5): Where λ is the temperature parameter of the softmax function; (2.3) Text vocabulary weighted feature extraction: Similarly, a text vocabulary is used as a benchmark to obtain a feature based on the text vocabulary y. t The corresponding image region feature x t * , the process is shown in formula (6):

4. The false information detection method based on multimodal context hierarchical and step-by-step alignment according to claim 1 is characterized in that: In the above (3), the basic feature vectors of the image and text extracted in step (1) and the weighted feature vectors of the image and text extracted in step (2) are matched at the segment level, and on this basis, segment-level matching scores for the complete image and the complete text are obtained; The specific steps for calculating segment-level similarity are as follows: (3.1) Calculate the fragment-level matching score associated with the image region: For the image region x i And the text vocabulary feature y corresponding to this region i * , calculate the similarity As shown in formula (7): (3.2) Calculate the segment-level matching score related to the text vocabulary: For the text vocabulary y t And the image region feature x corresponding to the vocabulary t * , calculate the similarity As shown in formula (8): (3.3) Calculate the segment-level matching score S of the complete image and the complete text segment (X, Y), as shown in formula (9): where η is a hyperparameter used to balance the region-dependent segment-level matching score S x (x i ,y i * ) and the vocabulary-related segment-level matching score S y (y t ,x t * )’s contribution.

5. The false information detection method based on multimodal context hierarchical and step-by-step alignment according to claim 1 is characterized in that: In (4), the basic feature vectors of the image and text extracted in step (1) are used to explore the contextual relationship between the features using a multi-head self-attention mechanism to obtain enhanced feature vectors of the image and text that incorporate contextual information. The specific steps are as follows: (4.1) Image enhancement feature extraction: For an image X, a set of regional features X = {x1, x2, ..., x m }, using the multi-head self-attention mechanism to explore the contextual relationship between regional features; first define the query Q X , key K X Sum value Val X , as shown in formula (10): Q X =XW Q ,K X =XW K ,Val X =XW Val (10) Where W Q ,W K ,W Val is a learnable feature mapping matrix; the value Val is calculated by formula (11) X The weighted sum of: where d k is the scaling factor, the weight matrix Q X K X T is a square matrix that encodes the relationship of each visual feature to all other features through the inner product layer; Then the relationship between different objects is encoded using formula (12): Where j represents the jth head, The W in the formula represents a learnable feature mapping matrix, and the superscript Q X Indicates that this matrix is ​​for image query, where Q represents query, x represents image, and subscript j represents the jth head; therefore, Refers to the query weight matrix specifically used to process the image input on the jth head in the multi-head attention mechanism; similarly, and They represent the key weight matrix and value weight matrix specifically used to process the image input on the jth head; Finally, the i-th original visual region feature x i It is transformed into enhanced features x by combining global objects and local guided objects. ip , as shown in formula (13): Where W p is a learnable weight matrix, h is the number of heads; then for an image, the set X based on regional features = {x1, x2, ..., x m The enhanced feature of} can be expressed as X reinforcement ={x 1p ,x 2p ,...,x mp }; (4.2) Text Enhanced Feature Extraction: For a piece of text Y, the text feature Y = {y1, y2, ..., y n }, perform the same process as step (4.1) to obtain the text enhancement feature Y reinforcement ={y 1p ,y 2p ,...,y np }; (4.3) Weighted enhanced feature extraction: For the weighted text feature X extracted in step (2), * ={x1 * ,x2 * ,...,x n * } and image features Y * ={y1 * ,y2 * ,...,y m * }Perform the same process as in step (4.1) to obtain the enhanced weighted features, which are represented as X reinforcement * ={x 1p * ,x 2p * ,...,x np * },Y reinforcement * ={y 1p * ,y 2p * ,...,y mp * }.

6. The false information detection method based on multimodal context hierarchical and step-by-step alignment according to claim 1 is characterized in that: In the above (5), the image and text enhancement features extracted in step (4) are filtered using an adaptive weight filtering module to obtain a global correlation matching result. The specific steps are as follows: (5.1) Calculate the global matching score from image to text: Substitute the enhanced basic feature pair (X reinforcement ,Y reinforcement ) and enhanced weighted feature pairs (Y reinforcement * ,X reinforcement * ) as input, the global semantics is generated by weighted fusion of regional features, and then the global semantic similarity between the two modalities is calculated to obtain the global matching score s in the image-to-text direction x→y , as shown in formula (14): where w x =cos(x lp ,y lp * ) is the adaptive weight, which represents the enhancement of basic features x lp and enhanced weighted features y lp * The importance of (5.2) Calculate the global matching score from text to image: enhance the basic feature pair (X reinforcement ,Y reinforcement ) and enhanced weighted feature pairs (Y reinforcement * ,X reinforcement * ) as input, the global semantics is generated by weighted fusion of regional features, and then the global semantic similarity between the two modalities is calculated to obtain the global matching score s in the image-to-text direction y→x , as shown in formula (15): Similar to (5.1), w y = cos(y lp ,x lp * ) is the adaptive weight, which means enhancing the basic feature y lp and the enhanced weighted features x lp * The importance of (5.3) Calculate the global matching score S of the complete image and the complete text global (X, Y), as shown in formula (16): S global (X,Y)=s x→y +s y→x (16)。 7. The false information detection method based on multimodal context hierarchical and step-by-step alignment according to claim 1 is characterized in that: In (6), the local segment-level matching result obtained in step (3) and the global matching result obtained in step (5) are input into a binary classifier, and the sum vector of the result vectors of the two levels is projected into two target spaces of true information and false information using a fully connected layer with a softmax function in the binary classifier, and the detection result of the social media blog post is obtained. The specific steps are as follows: (6.1) After obtaining the local region-word matching results and the global matching results, a fully connected layer with a softmax function is used to transform S segment (X,Y) and S global (X, Y) is projected into the target space with only two categories, real or fake, and its probability distribution is obtained as shown in formula (17): Where p = [p0, p1] represents the prediction probability vector, p0 and p1 represent the probabilities of the prediction results being 0 and 1 respectively, 0 represents real and 1 represents fake; W is the weight matrix, b is the bias term; (6.2) For each blog post, the goal is to minimize the binary cross entropy loss function to distinguish whether the blog post is true information or false information; the loss function is shown in formula (18): L p =-[ylnp0+(1-y)lnp1] (18) Where y∈{0,1} is the authenticity label of each blog post, y=0 means that the blog post is true information, and y=1 means that the blog post contains false information.

8. A false information detection system based on multimodal context hierarchical and step-by-step alignment for implementing the method of claim 1, characterized in that: It includes five modules: image region feature extraction, text vocabulary feature extraction, local alignment module, global context alignment module, and social media blog post detection; The image region feature extraction module first uses Faster R-CNN to detect salient regions, then uses the pre-trained ResNet-101 to extract a feature vector for each image region, and finally uses a fully connected layer to convert it into a feature vector to obtain an image region feature vector; The text vocabulary feature extraction module first encodes each word into a one-hot vector and embeds it into the vector o through the parameter matrix and the bias vector. t Then, Bi-GRU is used to capture the context information in the sentence from both the front and back directions and integrate it into the text embedding to obtain the text vocabulary feature vector; The local alignment module first obtains the segment-level cross-modal weighted feature vector through a bidirectional cross-attention mechanism, then calculates the segment-level matching score related to the region and the segment-level matching score related to the vocabulary, and finally averages all the segment-level matching scores based on the region and the segment-level matching scores based on the vocabulary to obtain the final segment-level matching score of the complete image and text; The global context alignment module first uses a multi-head self-attention mechanism to obtain the contextual relationship between segment-level features and obtains an enhanced feature vector that incorporates contextual information. It then calculates the global matching scores in the image-to-text and text-to-image directions respectively, and finally takes the sum of the global matching scores in the above two directions as the global matching score of the complete image and text. The social media blog post detection module inputs the segment-level similarity and global similarity between the image and the text into a fully connected layer with a softmax function to obtain a detection result of whether the social media blog post is true information or false information.

9. A computer device, characterized in that: The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the false information detection method based on multimodal context hierarchical and step-by-step alignment as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program for executing the false information detection method based on multimodal context hierarchical and step-by-step alignment as described in any one of claims 1 to 7.

Images