Multi-mode-based method for classifying image text pairs in network public opinions

Abstract

The invention discloses a multi-mode-based image text pair classification method in network public opinions, which comprises the following steps of: crawling image text pair data from a network, preprocessing the data, and extracting a main body part in an image; performing feature extraction on the image data, and performing feature extraction on the text data; performing feature space alignment on the extracted image text features by using a comparative learning method, and adding the image text features generated by using a momentum model as negative samples into comparative learning; inputting the aligned image text data features into a bidirectional feature fusion module to obtain bidirectional fusion features of the image text, and performing deep feature fusion on the bidirectional fusion features to obtain deep fusion features; and finally, inputting the deep fusion feature into a classifier to obtain a classification category of the image text pair. According to the method, the problem that the classification result is inaccurate due to the fact that the existing classification method only classifies texts and cannot overcome noise in image text data is solved.

Description

Technical Field

[0001] The invention belongs to the technical field of computer text image data processing, and relates to a method for classifying image and text pairs in network public opinion based on multimodality. Background Art

[0002] With the rapid development of my country's Internet technology, the number of mobile devices connected to the Internet has also increased exponentially year by year. Internet media has long replaced newspapers, radio and television as the first media. The media that influence social public opinion has also shown a trend of networking, thus forming online public opinion. Online public opinion is the reflection and direct manifestation of public opinion in cyberspace. Specifically, online public opinion refers to the opinions on social hot issues posted by netizens on the Internet with strong tendencies and influence. It usually uses online social platforms as a carrier, and with the development of the Internet, it spreads faster and more widely, and has a huge impact on society.

[0003] On the Internet, image text is a common form of content, such as posts and replies on forum websites, topics and comments on Weibo, most of which are presented in the form of image text pairs. Classifying image text in online public opinion has important practical application value, such as topic detection, public opinion detection and early warning on social media. However, due to the characteristics of high feature dimension, difficult feature extraction, and a lot of noise data of image text pairs on the Internet, the accuracy of traditional single-modal text classification is not high. There are several reasons for this. First, only the text information in the comments is used, while the image part of the comments is ignored; second, most of the text in the comments on the Internet are unstructured or semi-structured short texts, and the vector dimension of the text is high, which will lead to inaccurate classification results; third, the text in the comments contains a large number of code words and terms, and the text model cannot accurately identify the meaning of these code words and terms using only text. Summary of the invention

[0004] The purpose of the present invention is to provide a method for classifying image-text pairs in network public opinion based on multimodality, which solves the problem that the existing classification methods only classify texts and cannot overcome the noise in the image-text data, resulting in inaccurate classification results.

[0005] The technical solution adopted by the present invention is a method for classifying image-text pairs in multimodal network public opinion, which includes the following processes: crawling image-text pair data from the Internet, preprocessing the data, and extracting the main part of the image; extracting features from the image data and extracting features from the text data; aligning the extracted image-text features in feature space using a contrastive learning method, and adding the image-text features generated by a momentum model as negative samples to the contrastive learning; first inputting the aligned image-text data features into a bidirectional feature fusion module to obtain bidirectional fusion features of the image and text, performing deep feature fusion on the bidirectional fusion features to obtain deep fusion features; finally, inputting the deep fusion features into a classifier to obtain the classification category of the image-text pair.

[0006] The present invention is also characterized in that:

[0007] The specific steps include:

[0008] Step 1: crawl image-text pairs on the Internet to form a data set, and preprocess the image data and text data pairs respectively;

[0009] Step 2, constructing a BiFusion-CLIPNet model, which includes an image feature extraction module VIT, a text feature extraction module BERT, a bidirectional fusion module and a classifier. The image feature extraction module VIT is used to extract image features from the image data preprocessed in step 1, and the BERT model is used to extract text features from the text data preprocessed in step 1.

[0010] Step 3: Align the image features and text features extracted in step 2 using contrastive learning method to perform feature alignment of different modalities;

[0011] Step 4, feature fusion of the image text features aligned in step 3;

[0012] Step 5, the features fused in step 4 are predicted and classified through a classifier;

[0013] Step 6: Update the parameters according to the classification results of step 5.

[0014] The specific process of step 1 is:

[0015] Step 1.1: Remove punctuation marks, emoticons, and stop words from the text data crawled from the Internet; then use the Chinese word segmentation model of the BERT model to segment Chinese words and truncate them to within the maximum number of segmented words; map the segmented words to the corresponding id through the Chinese word table to obtain the word embedding representation X T1 , where the maximum number of word segments max_length is expressed using the following formula (1):

[0016]

[0017] Step 1.2, convert the image data crawled in step 1.1 into a binary grayscale image;

[0018] Step 1.3, performing Gaussian smoothing on the grayscale image converted in step 1.2 to eliminate noise pixels in the image;

[0019] Step 1.4, using the Laplace operator to extract the edge of the image after Gaussian smoothing in step 1.3, so as to obtain the edge image of the binary image;

[0020] Step 1.5, obtaining the coordinates of the main part of the image according to the binary edge image obtained in step 1.4;

[0021] Step 1.6: Perform image data enhancement on the main part of the image obtained in step 1.5 to obtain the enhanced image feature X I1 .

[0022] The specific process of step 2 is:

[0023] Step 2.1: The enhanced image obtained in step 1.6 is divided into image blocks by the PatchEmbed class in the timm library, and each divided image block is straightened into a one-dimensional sequence by convolution and flatten operations. The specific process is shown in the following formula (2):

[0024]

[0025] Step 2.2, generate a learnable one-dimensional sequence with the same dimension as the image block sequence obtained in step 2.1 through the torch.nn.Parameter() function, and concatenate it into the image block sequence as [CLS] token. Adding [CLS] token is expressed as the following formula (3):

[0026] X cls =[x cls ;X p ] (3)

[0027] Among them, x cls Indicates the added [CLS] token, X p represents the characteristics of the image block obtained by dimension transformation in formula (2), X cls Indicates the image features after adding [CLS] token;

[0028] Step 2.3, generate a learnable one-dimensional sequence with the same length as the image block sequence through the torch.nn.Parameter() function, and add it to the corresponding image block sequence as position embedding. The added position embedding is expressed as the following formula (4):

[0029] X cls_pos =X cls +x pos +x size (4)

[0030] Among them, X cls_pos represents the image features after adding [CLS] token and position embedding vector, x pos represents the added position embedding vector; x size Represents the image block size, which is calculated using the following formula:

[0031]

[0032] Step 2.4: Add the value X obtained in step 2.3 to cls_pos The sequence is input to each layer of the Transformer encoder of the image feature extraction module VIT. During the propagation process in each layer of the Transformer encoder of the image feature extraction module VIT, each pixel of all channels is first normalized. The normalization process is shown in formula (6), and finally the feature vector y of the image is obtained:

[0033]

[0034] Step 2.5: The features of the image normalized in step 2.4 are passed through the Transformer encoder of the image feature extraction module VIT to obtain the image feature Attention (Q I ,K I ,V I ), the Transformer encoder calculation of each layer of the image feature extraction module VIT is as follows:

[0035] Q I =w q y T (6)

[0036] K I =w k y T (7)

[0037] V I =w v y T (8)

[0038]

[0039] Among them, w q ,w k ,w v is the parameter in the Transformer encoder in the image feature extraction module VIT, and d represents Q I ,K I ,V I Dimensions;

[0040] Step 2.6: embed the words of the text data obtained in step 1.1 into X T1 Input to the embedding layer of the text feature extraction module BERT, and embed the input word X T1 Plus the position embedding y pos , fragment embedding y seg , mark embedding y lab , get the text embedding feature X embed ,Right now:

[0041] X embed =X T1 +y pos +y seg +y lab (10)

[0042] Step 2.7: embed the text feature X obtained in step 2.6 embed , input to the first 6 layers of the text feature extraction module BERT, extract text features, and obtain the text feature Attention (Q T ,K T ,V T ), the calculation process is as follows:

[0043]

[0044] The specific process of step 3 is:

[0045] Step 3.1, the image features obtained in step 2.5 and the text features obtained in step 2.7 are used to form image-text pairs, the image-text pairs are divided into several batches, and the parameters in the BiFusion-CLIPNet model are copied to obtain a momentum model;

[0046] Step 3.2: Input each batch of data constructed in step 3.1 into the momentum model to obtain image features and text features, and extract the [CLS] token added in step 2.2 as a negative sample and add it to the negative sample queue. At the same time, input each batch of data into the BiFusion-CLIPNet model to obtain image features and text features, and extract the [CLS] token added in step 2.2 as a positive sample pair and add it to the positive sample queue.

[0047] Step 3.3, use the normalized Manhattan distance to calculate the similarity S between the texts T (T1, T2), the calculation formula is shown in formula (16):

[0048]

[0049] Among them, t1, T2 represent two different text features extracted by the BiFusion-CLIPNet model, n is the dimension of the feature, and T j Represents the j-th dimension of text feature T;

[0050] Step 3.4, use the Pearson correlation coefficient to determine the similarity S between images I (I1,I2), the calculation formula is shown in formula (17):

[0051]

[0052] Among them, I1, I2 represent two different image features extracted by the BiFusion-CLIPNet model, n is the dimension of the feature, and I k represents the kth dimension of image feature I, Represents the sample mean of all images in the dataset;

[0053] Step 3.5, use extended cosine similarity to calculate the similarity S between the image and text pairs IT (I, T), the calculation formula is shown in formula (18):

[0054]

[0055] Step 3.6, use the contrastive learning method to align the image features and text features in the BiFusion-CLIPNet model. The contrastive learning loss function L ITC as follows:

[0056] y(I,T)=β1S T +β2S I +β3S IT (19)

[0057] p(I m ,T m )=β1S T +β2S I +β3S IT (20)

[0058]

[0059] The specific process of step 4 is:

[0060] Step 4.1: For the image features and text features aligned in step 3: the image features are recorded as X I , X I The global characteristics of The text feature is recorded as X T , X T The global characteristics of

[0061] Step 4.2: Using global image features And the global feature of the text is Calculate the self-attention score and take the weighted sum to get the feature of the text for the image, denoted as F T2I ;

[0062] Step 4.2, taking the global features of the text as Global image features Calculate the self-attention score and take the weighted sum to get the image-to-text feature, denoted as F T2I ;

[0063] Step 4.3: Compare the text obtained in steps 4.1 and 4.2 to the image feature F T2I And the image features F for text I2T Normalize them separately and fuse them through the multi-layer perceptron MLP to obtain the final bidirectional fusion feature F;

[0064] In step 4.4, the bidirectional fusion feature F obtained in step 4.3 is input into the last 6 layers of Transformer encoder of the text feature extraction module BERT to extract deep fusion features and obtain the final image-text fusion feature.

[0065] The specific process of step 5 is:

[0066] Step 5.1, input the bidirectional fusion features extracted in step 4.4 into the classifiers of the BiFusion-CLIPNet model and the momentum model at the same time;

[0067] Step 5.2, in the classifier, first pass the image-text fusion feature obtained in step 4.4 through a linear projection layer to map the feature dimension to a higher-dimensional feature space, then pass it through a ReLU() activation function for nonlinear mapping, and finally pass it through a linear projection layer to map the feature to the category dimension, and finally pass it through the Softmax() function to output the probability distribution of the category prediction of the bidirectional fusion contrast learning model;

[0068] Step 5.3, during training, the model update process is as follows:

[0069] During the training process, each batch of data is input into the momentum model, and after extracting the image-text fusion features, mapping and classification are performed through steps 5.1 and 5.2 to obtain the category prediction probability distribution of the momentum model; combined with the probability distribution of the category prediction of the bidirectional fusion contrast learning model output in step 5.2, the loss function is calculated, the gradient is back-transferred, and the parameters of the BiFusion-CLIPNet model are updated. The loss function L mod The calculation formula is as follows:

[0070]

[0071] Among them, y label Represents the true category probability distribution, p BFC represents the category probability distribution output by the BiFusion-CLIPNet model, p mod represents the category probability distribution output by the momentum model, β represents the weight considering the BiFusion-CLIPNet model and the momentum model, and H represents the cross entropy loss.

[0072] The specific process of step 6 is:

[0073] After updating the model in step 5.3, copy the parameters in the BiFusion-CLIPNet model to the momentum model according to the copy ratio k. The formula is as follows:

[0074] θ m ←kθ m +(1-m)θ (17)

[0075] Among them, θ m represents the parameters of the momentum model, and θ represents the parameters of the BiFusion-CLIPNet model.

[0076] The beneficial effect of the present invention is that, compared with the text classification method that only uses a single mode, the image-text multimodal classification method used in this paper can enable the two modes of image and text to integrate and complement each other, so that the classification can take into account the information of more modes, thereby improving the accuracy and robustness of the classification. And the momentum distillation method is used to make the image text generated by the momentum model participate in the training of the model as a sample, which can better resist the noise existing in the image text crawled from the Internet (the description of the text does not match the image), and has strong robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 It is a flow chart of the image-text pair classification method in multimodal network public opinion of the present invention;

[0078] Figure 2 It is a structural diagram of a bidirectional fusion contrast learning model constructed in the image-text pair classification method in multimodal network public opinion of the present invention;

[0079] Figure 3 It is a comparative learning flow chart adopted in the image-text pair classification method in multimodal network public opinion of the present invention;

[0080] Figure 4 It is a schematic diagram of the structure of a bidirectional fusion module used in a bidirectional fusion contrast learning model in a multimodal image-text pair classification method in network public opinion of the present invention;

[0081] Figure 5(a) to Figure 5(d) It is a pre-trained similarity matrix under different training data in the image-text pair classification method in multimodal network public opinion of the present invention;

[0082] Figure 6 It is a curve of classification training accuracy change under different training data in the image-text pair classification method in multimodal network public opinion of the present invention. DETAILED DESCRIPTION

[0083] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0084] Example 1

[0085] The present invention is based on a multimodal image-text pair classification method in network public opinion, and the process is: crawling image-text pair data from the network, preprocessing the data, and extracting the main part of the image; using the image feature extraction module VIT to extract features from the image data, and using the first 6 layers of the text feature extraction module BERT to extract features from the text data; using the contrast learning method to align the extracted image-text features in the feature space, and using the image-text features generated by the momentum model as negative samples to add to the contrast learning; the aligned image-text data features are first input into the bidirectional feature fusion module to obtain the bidirectional fusion features of the image and text, and then input into the last 6 layers of the text feature extraction module BERT for deep feature fusion to obtain deep fusion features; finally, the fused features are input into the classifier to obtain the classification category of the image-text pair. The present invention not only utilizes the text data on the network, but also integrates the image data, which can improve the accuracy of the classification of image-text pairs on the network; and the pseudo samples generated by the momentum model can better resist the noise of crawling image-text pairs on the network, and have good robustness.

[0086] Example 2

[0087] The present invention is based on a multimodal image-text pair classification method in network public opinion, and the process is as follows: Figure 1 As shown, the specific steps include: Step 1, crawling image-text pairs on the Internet to form a data set, and preprocessing the image-text pairs. The steps of data preprocessing are:

[0088] Step 1.1: Crawl data from the Internet and construct a data set, specifically:

[0089] Punctuation marks, emoticons, and stop words are removed from the text data crawled from the Internet. Then, the Chinese word segmentation model of BERT (Bidirectional Encoder Representations from Transformers) is used to segment Chinese words and truncate them to within the maximum number of word segments. The calculation of the maximum number of word segments is shown in formula (1), where max_length represents the maximum number of word segments calculated at the end, length represents the length of the text in the data set, and n represents the number of data in the data set. The segmented words are mapped to the corresponding id through the Chinese word table to obtain the word embedding representation X. T . Wherein, i=1~n.

[0090]

[0091] Step 1.2, using the traditional image edge extraction method, extract the main part of the image and discard the background part. The data set used in the present invention is crawled from a public opinion website. The image-text pairs in the data set are composed of public opinion images and texts of post comments. After analysis, it can be seen that most images are divided into a main part and a background part (the background is an irrelevant part, such as the floor, wall, grass, etc.), and the boundary between the main part and the background part of the image has a large color change. From the grayscale image of the image, there is an obvious gradient change between the main part and the background part of the image. In order to remove the background information and make the model pay more attention to the main part of the image, the present invention uses the traditional image edge extraction method to extract the contour information (coordinates) of the main part of the image, thereby discarding the useless background information in modal fusion.

[0092] Step 1.2 extracts the main part of the image, specifically:

[0093] Step 1.2.1, convert the image data crawled in step 1.1 into a binary grayscale image, that is, use the cvtColor() function in the OPENCV image processing library to convert the RGB three-channel color image into a binary grayscale image. The principle is to use multiple weighting coefficients to weight the grayscale values ​​of the RGB three channels to obtain the corresponding grayscale image.

[0094] Step 1.2.2, Gaussian smoothing is performed on the grayscale image converted in step 1.2.1. Image data crawled from the Internet will cause the image to have many noise points due to reasons such as shooting equipment. The algorithm used in the present invention to extract the edge of the main part of the image using the Laplace operator is more sensitive to noise. Using this algorithm to extract an image with noise will also extract the edge of the noise, which will cause the edge of the main part of the image to be extracted inaccurately, so the image is first Gaussian smoothed to eliminate the noise pixels in the image. The most important thing about using the Gaussian smoothing algorithm is to determine the weight matrix. The present invention uses a normal distribution curve to distribute weights. When calculating the weight distribution, it is only necessary to use the center point as the origin, and the other points are assigned weights according to their positions on the normal curve to obtain a weighted average. Then, the weight matrix is ​​used to perform a convolution operation on each pixel in the image, smooth the image, remove the noise points in the image, and eliminate the interference of the noise points for the next step of edge extraction.

[0095] Step 1.2.3, use the Laplace operator to extract the edge of the image after Gaussian smoothing in step 1.2.2. Laplace edge extraction belongs to the use of second-order derivatives to extract edges. It is an isotropic edge extraction operator. Isotropic means that using this operator, any boundary and line can be sharpened without directionality. Then use this operator to perform convolution operations on each pixel in the image in the horizontal and vertical directions to smooth the image, thereby obtaining the edge image of the binary image.

[0096] Step 1.2.4, obtain the coordinates of the main part of the image based on the binary edge image obtained in step 1.2.3. Take the upper left as an example, traverse the pixels from the upper left in sequence, and record the pixel coordinates of the first pixel 255. According to this method, the coordinates of the four starting points are obtained, and the coordinates of the four points are minimized to obtain a rectangular selection area. This area is the part containing the main part of the image, and then the image is cropped, the background part of the image is discarded, and the main part of the image is obtained.

[0097] Step 1.3, perform image data enhancement on the main part of the image obtained in step 1.2. Specifically:

[0098] The resolution of the main part of each image is different, so the cropping method is used to fix each photo to the same resolution. At the same time, the transforms in the TorchVision library are used to transform the image to achieve data enhancement, such as the RandomResizedCrop() random cropping method and the RandomHorizontalFlip() random horizontal flip method, so as to enrich the data representation and expand the data set.

[0099] Step 2, extraction of image text features. The present invention constructs a BiFusion-CLIPNet model (bidirectional fusion contrast learning model), which includes an image feature extraction module VIT (Vision Transformer), a text feature extraction module BERT (Bidirectional Encoder Representations from Transformers), a bidirectional cross attention fusion module and a classifier. For image data, the image feature extraction module VIT (Vision Transformer) of the BiFusion-CLIPNet model is used for extraction, and for text data, its text feature extraction module BERT (Bidirectional Encoder Representations from Transformers) model is used for extraction.

[0100] Step 2.1, the image feature extraction module VIT is composed of 12 layers of Transformer encoders, each layer of which includes a self-attention layer, a normalization layer, and a fully connected layer. The image feature extraction module VIT divides a whole image into several small blocks, and straightens each small block into a sequence, thereby turning the image into a sequence and inputting it into the multi-layer Transformer encoder of the image feature extraction module VIT to obtain image features.

[0101] Step 2.1.1: Use the PatchEmbed class in the timm library to divide the enhanced image obtained in step 1.3 into image blocks of different sizes. Because in the dataset, the main part of the image is distributed in different positions of the image, using image blocks of different sizes can better cut out the main part of the image and extract image features with more semantic information. At the same time, each small image block after segmentation is straightened into a one-dimensional sequence through convolution and flatten operations. The dimensional transformation of the image in the above process is shown in formula (2):

[0102]

[0103] Among them, X I1 represents the image features enhanced by step 1.3, and its feature dimension is H×W×C, where H and W represent the image length and width, and C represents the number of channels of the image. Conv2d and flatten represent the operation of converting a two-dimensional image into a one-dimensional sequence. p It represents the one-dimensional sequence after the segmentation of the two-dimensional small image blocks, whose dimension is N×(P 2 ×C), N represents the number of small image blocks, P represents the length of each small image block, where in Represents dimensions of H×W×C and N×(P 2 ×C) belongs to the field of real numbers.

[0104] In step 2.1.2, use the torch.nn.Parameter() function to generate a learnable one-dimensional sequence with the same dimension as the image block sequence obtained in step 2.1.1, and add it to the image block sequence as [CLS] token. Its function is to represent the global features of the image after being encoded by the VIT multi-layer Transformer encoder for the next step of feature fusion. Adding [CLS] token can be expressed as formula (3):

[0105] X cls =[x cls ;X p ] (3)

[0106] Among them, xcls Indicates the added [CLS] token, X p It represents the characteristics of the image block obtained by the dimension transformation of formula-2. cls Indicates the image features after adding [CLS] token.

[0107] Step 2.1.3, because in step 2.1.1, in order to cut out the main part of the image as much as possible, the image is divided into small image blocks of different sizes. In order to provide the size information of the model image block, the present invention adopts area cosine scale coding to provide size information, and its calculation formula is shown in formula (4):

[0108]

[0109] The torch.nn.Parameter() function generates a learnable one-dimensional sequence with the same length as the image block sequence (its elements start from 0 and increase in sequence), and adds it to the corresponding image block sequence as position embedding. Its function is to provide the position information of each image block sequence to the Transformer encoder. The size embedding and the addition of position embedding can be expressed as formula (5), where x pos Represents the added position embedding vector:

[0110] X cls_pos =X cls +x pos +x size (5)

[0111] X cls_pos Represents the image features after adding [CLS] token, position embedding and size embedding vector.

[0112] Step 2.1.4: Substitute the X obtained in step 2.1.3 cls_pos The sequence is input into each layer of the Transformer encoder of the VIT model, and finally the feature vector of the image is obtained. In the process of propagation in each layer of the Transformer encoder of the VIT model, each pixel of all channels is first normalized, and the calculation formula is shown in formula (6):

[0113]

[0114] Among them, X cls_pos represents the vector with position information of the image obtained in step 2.1.3, E(.) represents the calculated mean, Var(.) represents the calculated variance, y represents the normalized image vector finally obtained, and ε is a very small value to prevent division calculation errors when the mean is 0. In the present invention, 10 is taken -5 .

[0115] Step 2.1.5: The features of the image normalized in step 2.1.4 are passed through the Transformer encoder of the image feature extraction module VIT to obtain a feature vector with semantic information. The calculation of each layer of the Transformer encoder of the image feature extraction module VIT is as follows:

[0116] Q I =w q y T (7)

[0117] K I =w k y T (8)

[0118] V I =w v y T (9)

[0119]

[0120] Among them, w q ,w k ,w v are the parameters in the Transformer encoder in the VIT model, multiplied by the transpose (T) of y to get Q I ,K I ,V I Vector, the image feature after attention mechanism is calculated by formula (10), d k Indicates Q T ,K T ,V T The dimension of is used to scale the features.

[0121] Step 2.2, extraction of text features: The text feature extraction module BERT is a pre-trained language representation model, which is different from the traditional unidirectional language model, but a deep bidirectional Transformer encoder (12 layers). The text feature extraction module BERT can generate deep bidirectional language representation. Since the text feature extraction module BERT is pre-trained using MLM (Masked Language Model, masked language modeling task), after obtaining the pre-trained model, only an additional output layer needs to be added for fine-tuning, and it can be used for various downstream tasks, and it has good migration capabilities. The text feature extraction module BERT mainly includes two parts, one is the embedding layer, including position embedding, fragment embedding and tag embedding; the other is an encoder layer stacked by multiple Transformer encoders (12 layers of Transformer encoders are selected in the present invention).

[0122] Step 2.2.1: embed the words in the text data obtained in step 1.1 into X T1 Input into the embedding layer of the text feature extraction module BERT, and add the position embedding y to the input word embedding pos , fragment embedding y seg , mark embedding y lab , get the text embedding feature X embed ,Right now:

[0123] X embed =X T1 +y pos +y seg +y lab (11)

[0124] Step 2.2.2: embed the text feature X obtained in step 2.2.1 embed , input to the first 6 layers of Transformer encoder of the text feature extraction module BERT, extract text features, and obtain text features. The calculation process is as follows:

[0125]

[0126] Among them, w q ,w k ,w v are the parameters in the Transformer encoder in the BERT model, multiplied by X embed The transpose of Q T ,K T ,V T Vector, the text feature after the attention mechanism is calculated by formula (15), d represents Q T ,K T ,V T The dimension of is used to scale the features.

[0127] The information extracted from text data using the first 6 layers of the Transformer encoder of the BERT model can not only capture sentence-level semantic information, but also retain a certain degree of word-level semantic information. This multi-level semantic expression provides rich and detailed features for subsequent classification tasks, which helps to significantly improve classification performance.

[0128] Step 3, alignment of image text features, such as Figure 3As shown, the image features extracted in step 2.1 and the text features extracted in step 2.2 are not aligned in most cases, that is, image-text pairs with the same semantics have nothing to do with each other in the extracted feature space, which will affect the multimodal feature fusion and make the feature fusion encoder unable to extract effective multimodal fusion features, so that the model has no better transferability to downstream tasks.

[0129] In summary, before multimodal feature fusion, the extracted image and text features should be aligned so that the features of image and text pairs with the same semantic information are as close as possible in the feature space, while the features of unrelated image and text pairs are as far away as possible. In this invention, we use the contrastive learning method (the schematic diagram of contrastive learning is shown in Figure 3 ) to align the image-text pair features, thereby providing better image-text pair features for feature fusion.

[0130] Step 3.1, data batch and momentum model construction: The image features obtained in step 2.1 and the text features obtained in step 2.2 form an image-text pair, and the image-text pair is divided into several batches. At the same time, the parameters of the BiFusion-CLIPNet model in step 2 are copied to obtain a momentum model (the momentum model is to copy the parameters of the BiFusion-CLIPNet model, that is, to copy a BiFusion-CLIPNet model, but when updating, this momentum model will always lag behind the BiFusion-CLIPNet model, that is, the parameters of the momentum model are the parameters of the last update of the BiFusion-CLIPNet model). In the experiment, the parameter update of the momentum model constructed above will always lag behind the original model in each training iteration. This is because the data is unevenly distributed when crawled from the Internet. Adding the features obtained by the momentum model to the loss of calculating contrastive learning will slow down the trend of updating the parameters of the BiFusion-CLIPNet model, preventing the BiFusion-CLIPNet model from being turbulent due to the imbalance of data crawled from the Internet during the training process.

[0131] Step 3.2, generation of positive and negative samples: Input each batch of data constructed in step 3.1 into the momentum model to obtain image features and text features, and extract the [CLS] token added in step 2.1.2 as a negative sample and add it to the negative sample queue. At the same time, input each batch of data into the BiFusion-CLIPNet model to obtain image features and text features, and extract the [CLS] token added in step 2.1.2 as a positive sample pair and add it to the positive sample queue.

[0132] Step 3.3, calculating the similarity between the image and the text. In the present invention, multi-dimensional similarity is calculated to evaluate the similarity between the image and the text.

[0133] Step 3.3.1, calculate the similarity between plain texts. Use the normalized Manhattan distance to evaluate the similarity between texts. In the text data in the dataset of the present invention, many features are usually sparse, that is, the values ​​of most features are zero. Manhattan distance can effectively process such sparse data and will not be amplified by the square of non-zero features. Its calculation formula is shown in formula (16):

[0134]

[0135] Among them, t1, T2 represent two different text features extracted by the BiFusion-CLIPNet model, n is the dimension of the feature, and T j Represents the j-th dimension of the text feature T.

[0136] Step 3.3.2, calculate the similarity between pure images. Use the Pearson correlation coefficient to evaluate the similarity between images. The Pearson correlation coefficient has translation invariance and scale invariance, and is adaptive to image translation and selection. Its calculation formula is shown in formula (17):

[0137]

[0138] Among them, I1, I2 represent two different image features extracted by the BiFusion-CLIPNet model, n is the dimension of the feature, and I k represents the kth dimension of image feature I, Represents the sample mean of all images in the dataset.

[0139] Step 3.3.3, calculate the similarity between the image and text pairs. Use the extended cosine similarity to evaluate the similarity between the image and text pairs. The calculation formula is shown in formula (18):

[0140]

[0141] Among them, I and T represent the image and text features extracted by the BiFusion-CLIPNet model, respectively, ∥I∥2 and ∥T∥2 are the L1 norms of image and text features; W is a weighting matrix that can be used to adjust the importance of different dimensions to adapt to more complex multimodal scenarios; λ is the introduced regularization parameter to increase the sparsity of features and reduce the impact of high-dimensional vectors to avoid overfitting.

[0142] Step 3.4, use the contrastive learning method to align the image features and text features in the BiFusion-CLIPNet model, and the contrastive learning loss function L ITC as follows:

[0143] y(I,T)=β1S T +β2S I +β3S IT (19)

[0144] p(I m ,T m )=β1S T +β2S I +β3S IT (20)

[0145]

[0146] In formulas (19) and (20), y(I,T) represents the multidimensional similarity between image and text features in the BiFusion-CLIPNet model, and p(I m ,T m ) represents the multi-dimensional similarity between image and text features in the momentum model, S T , S I , S IT They are the multi-dimensional similarities calculated in step 3.3, and β1, β2, and β3 are the weight parameters of each similarity.

[0147] In formula (21), y i2t represents the multidimensional similarity between image features and text features in the BiFusion-CLIPNet model, y t2i represents the multi-dimensional similarity of text features relative to image features in the BiFusion-CLIPNet model; p i2t represents the cosine similarity of image features relative to text features in the momentum model, p t2i represents the cosine similarity of text features relative to image features in the momentum model, H represents the calculation of cross entropy loss, I, I m Denote the image features extracted by the BiFusion-CLIPNet model and the momentum model, T, T m They represent the image features extracted by the bidirectional fusion contrast learning model and the momentum model respectively. The image features and text features extracted by the BiFusion-CLIPNet model and the momentum model are input into the loss function of formula (22), and the BiFusion-CLIPNet model is back-propagated to update the VIT and BERT in the bidirectional fusion contrast learning model, thereby obtaining the aligned image features and text features.

[0148] Step 4, fusing the image features and text features aligned in step 3 through a bidirectional fusion module: The present invention uses a bidirectional fusion module to fuse image and text features, such as Figure 4 The bidirectional fusion module shown mainly uses the CrossAttention mechanism for feature fusion, and then extracts deep image-text feature fusion through the last 6 layers of the Transformer encoder of the BERT model that extracts text features. In the bidirectional feature fusion module, on the one hand, the image features are taken as Q, the text features are taken as K and V, and the self-attention score is calculated with the global features of the image and the text features, that is, the weight parameters of different word embeddings in the same text feature sequence are calculated with the global features of the image as the query key, and the weighted features of the text for the image are obtained. On the other hand, the text features are taken as Q, the image features are taken as K and V, and the self-attention score is calculated with the global features of the text and the image features, that is, the weight parameters of different image regions in the same image feature sequence are calculated with the global features of the text as the query key, and the weighted features of the image for the text are obtained.

[0149] Step 4.1: For the image features and text features aligned in step 3: the image features are recorded as X I , X I The global characteristics of The text feature is recorded as X T , X T The global characteristics of Global image features And the global feature of the text is Calculate the self-attention score and take the weighted sum to get the feature of the text for the image, denoted as F T2I , the specific steps are as follows:

[0150] Step 4.1.1, use the global features of the aligned images Calculate the self-attention matrix Q T2I The matrix is:

[0151]

[0152] Step 4.1.2, use the aligned text global features as Calculating K in the self-attention mechanism T2I 、V T2I The matrix is:

[0153]

[0154] W in formulas (22) to (24) Q , W K , W V These are all trainable parameters in the attention mechanism.

[0155] Step 4.1.3, Q calculated in Step 4.1.1 and Step 4.1.2 T2I , K T2I The matrix is ​​multiplied to get the self-attention mechanism score α T2I , which is the similarity matrix, where (·) T is transposed, R is the relative position information matrix, and the R matrix contains the relative position information between all positions, which enables the model to consider the relative position information of the input elements when calculating attention. This method allows the model to capture the relationship between different elements more flexibly, rather than just relying on their absolute positions. It can capture the relative relationship of elements in the sequence, rather than just the absolute positions. The formula is shown in formula (25):

[0156] α T2I =Q T2I (K T2I +R) T (25)

[0157] Step 4.1.4, normalize the similarity matrix obtained in step 4.1.3 using the softmax() function to obtain the weight matrix between the input sequences, and scale it using the dimension of the embedded features. The formula is as follows:

[0158]

[0159] Among them, dim represents The dimension of Dimension, weight T2I represents the obtained weight matrix;

[0160] Step 4.1.5, get the weight matrix weight T2I Then with the matrix V T2I Perform weighted summation to obtain the feature F of the text for the image T2I ,Right now:

[0161] F T2I =ΣV T2I ·weight T2I (27)

[0162] Step 4.2, find the feature F of the image for the text I2T , the formula is as follows:

[0163] Step 4.2.1, use the aligned text global features as Calculate the self-attention matrix Q I2T ,Right now:

[0164]

[0165] Step 4.2.2, use the global features of the aligned images Calculating K in the self-attention mechanism I2T 、V I2T The matrix is:

[0166]

[0167] W in formulas (28) to (30) Q , W K , W V These are all trainable parameters in the attention mechanism.

[0168] Step 4.2.3, Q calculated in steps 4.2.1 and 4.2.2 I2T , K I2T The matrix is ​​multiplied to get the self-attention mechanism score α I2T , which is the similarity matrix, where T is the transpose and R is the relative position information matrix, specifically:

[0169] α I2T =Q I2T (K I2T +R) T (31)

[0170] Step 4.2.4, normalize the similarity matrix obtained in step 4.1.3 using the softmax() function to obtain the weight matrix between the input sequences, and scale it using the dimension of the embedded features. The formula is as follows:

[0171]

[0172] Among them, dim represents The dimension of, that is, the dimension of, weight I2T represents the obtained weight matrix.

[0173] Step 4.2.5, get the weight matrix weight I2T Then with the matrix V I2T Perform weighted summation to obtain the feature F of the text for the image I2T ,Right now:

[0174] F I2T =ΣV I2T ·weight I2T (33)

[0175] Step 4.3: Compare the text obtained in steps 4.1 and 4.2 to the image feature F T2I And the image features F for text I2TThey are normalized separately and fused through a multi-layer perceptron MLP to obtain the final bidirectional fusion feature F.

[0176]

[0177] Among them, E(·) represents the mean of the calculated feature, and Var(·) represents the standard deviation of the calculated feature.

[0178] In step 4.4, the bidirectional fusion feature F obtained in step 4.3 is input into the last 6 layers of Transformer encoder of the text feature extraction module BERT to extract deep fusion features and obtain the final image-text fusion feature.

[0179] Step 5: Use the image-text fusion features extracted in step 4.4 to perform prediction and classification. The specific process is as follows:

[0180] In step 5.1, the bidirectional fusion features extracted in step 4.4 are simultaneously input into the classifiers of the BiFusion-CLIPNet model and the momentum model.

[0181] Step 5.2, in the classifier, first pass the image-text fusion features obtained in step 4.4 through a linear projection layer to map the feature dimension to a higher-dimensional feature space, then pass through a ReLU() activation function for nonlinear mapping, and finally pass through a linear projection layer to map the feature to the category dimension, and finally pass through the Softmax() function to output the probability distribution of the category prediction of the bidirectional fusion contrast learning model.

[0182] Step 5.3, during training, the update process of the bidirectional fusion contrastive learning model is as follows: During the training process, the predicted category of the momentum model will be taken into account in the calculation of the loss function. Input each batch of data into the momentum model, extract the image-text fusion features as in steps 3 and 4 (that is, for the momentum model, re-execute steps 3 to 4), and then perform mapping classification through steps 5.1 and 5.2 to obtain the category prediction probability distribution of the momentum model; combine the probability distribution of the category prediction of the bidirectional fusion contrastive learning model output in step 5.2, calculate the loss function, perform gradient backpropagation, update the parameters of the BiFusion-CLIPNet model, and the loss function L mod The calculation formula is as follows:

[0183]

[0184] Among them, y label Represents the true category probability distribution, p BFC represents the category probability distribution output by the BiFusion-CLIPNet model, p modrepresents the category probability distribution output by the momentum model, β represents the weight of the considered model and the momentum model, and H represents the cross entropy loss.

[0185] Step 6, parameter update between the bidirectional fusion contrast learning model and the momentum model during training.

[0186] After each training iteration in step 5.3, the predicted values ​​and true values ​​generated by the bidirectional fusion contrastive learning model and the momentum model are used to calculate the loss function in formula (35), and the parameters of the bidirectional fusion contrastive learning model are gradient updated. At the same time, the updated parameters of the bidirectional fusion contrastive learning model are copied to the momentum model according to a certain ratio. The specific steps are as follows:

[0187] Step 6.1, after updating the BiFusion-CLIPNet model in step 5.3, copy the parameters of the BiFusion-CLIPNet model to the momentum model at a certain copy ratio k, which is expressed as follows:

[0188] θ m ←kθ m +(1-m)θ (36)

[0189] Among them, θ m represents the parameters of the momentum model, and θ represents the parameters of the BiFusion-CLIPNet model;

[0190] In terms of the setting of parameter k, small batches of data are randomly sampled in the dataset for experiments. When the replication ratio k is set to about 0.3, the momentum model can not only update parameters with the bidirectional fusion contrast learning model, but also generate stable samples, thereby improving the classification results.

[0191] Example 3

[0192] 14,000 image-text pairs with the same topic were crawled from a public opinion website and manually annotated to form the data set used in the experiment. For classification annotation, the present invention divides all data into four categories: A, B, C, and D for experimentation. The experimental results are as follows:

[0193] The experimental results of image-text feature alignment in step 3 are as follows: Figure 5(a) to Figure 5(d) The present invention divides the 14,000 data in the data set into 2,000, 8,000, and 14,000 data to conduct experiments, explore the impact of using different amounts of data for training on feature alignment, and use the similarity between paired image and text features as an evaluation indicator. Figure 5a )~ Figure 5d) represent the results of the pre-trained model after 0, 2000, 8000, and 14000 data pre-training respectively. As the amount of data increases, the higher the similarity between paired images and texts, the lower the similarity between unpaired images and texts, and the better the data alignment effect.

[0194] The experimental results of step 5 classification are as follows Figure 6 As shown in Table 1 (A, B, C, and D in Table 1 represent four categories). Different data sizes are used to perform image-text feature alignment in step 3 for classification. As the amount of data increases during feature alignment, the classification accuracy and F1 value also increase.

[0195] Table 1 Classification results of pre-trained models with different data amounts

[0196] Amount of pre-training data A(F1 value) B(F1 value) C(F1 value) D(F1 value) 0 0.742 0.714 0.723 0.395 2000 0.677 0.649 0.693 0.453 8000 0.722 0.687 0.741 0.444 14000 0.713 0.693 0.815 0.557

Claims

1. A method for classifying image-text pairs in multimodal online public opinion, characterized in that: The process includes: crawling image-text pair data from the Internet, preprocessing the data, and extracting the main part of the image; extracting features from the image data and the text data; aligning the extracted image-text features in the feature space using the contrastive learning method, and adding the image-text features generated by the momentum model as negative samples to the contrastive learning; inputting the aligned image-text data features into the bidirectional feature fusion module to obtain the bidirectional fusion features of the image and text, and performing deep feature fusion on the bidirectional fusion features to obtain deep fusion features; Finally, the deep fusion features are input into the classifier to obtain the classification category of the image-text pair.

2. According to claim 1, the image-text pair classification method in multimodal network public opinion is characterized in that: The specific steps include: Step 1: crawl image-text pairs on the Internet to form a data set, and preprocess the image data and text data pairs respectively; Step 2, constructing a BiFusion-CLIPNet model, which includes an image feature extraction module VIT, a text feature extraction module BERT, a bidirectional fusion module and a classifier. The image feature extraction module VIT is used to extract image features from the image data preprocessed in step 1, and the BERT model is used to extract text features from the text data preprocessed in step 1. Step 3: Align the image features and text features extracted in step 2 using contrastive learning method to perform feature alignment of different modalities; Step 4, feature fusion of the image text features aligned in step 3; Step 5, the features fused in step 4 are predicted and classified through a classifier; Step 6: Update the parameters according to the classification results of step 5.

3. The method for classifying image-text pairs in multimodal network public opinion according to claim 2 is characterized in that: The specific process of step 1 is as follows: Step 1.1: Remove punctuation marks, emoticons, and stop words from the text data crawled from the Internet; then use the Chinese word segmentation model of the BERT model to segment Chinese words and truncate them to within the maximum number of segmented words; map the segmented words to the corresponding id through the Chinese word table to obtain the word embedding representation X T1 , where the maximum number of word segments max_length is expressed using the following formula (1): Step 1.2, convert the image data crawled in step 1.1 into a binary grayscale image; Step 1.3, performing Gaussian smoothing on the grayscale image converted in step 1.2 to eliminate noise pixels in the image; Step 1.4, using the Laplace operator to extract the edge of the image after Gaussian smoothing in step 1.3, so as to obtain the edge image of the binary image; Step 1.5, obtaining the coordinates of the main part of the image according to the binary edge image obtained in step 1.4; Step 1.6: Perform image data enhancement on the main part of the image obtained in step 1.5 to obtain the enhanced image feature X I1 .

4. The method for classifying image-text pairs in multimodal network public opinion according to claim 3 is characterized in that: The specific process of step 2 is: Step 2.1: The enhanced image obtained in step 1.6 is divided into image blocks by the PatchEmbed class in the timm library, and each divided image block is straightened into a one-dimensional sequence by convolution and flatten operations. The specific process is shown in the following formula (2): Among them, X I1 represents the image features enhanced by step 1.6, H and W represent the image length and width, C represents the number of channels of the image, Conv2d and flatten represent the operation of converting a two-dimensional image into a one-dimensional sequence, X p It represents the one-dimensional sequence after the segmented two-dimensional image block is converted, and the dimension is N×(P 2 ×C), N represents the number of image blocks, P represents the length of each image block, where Represents dimensions of H×W×C and N×(P 2 ×C) belongs to the real number domain; Step 2.2, generate a learnable one-dimensional sequence with the same dimension as the image block sequence obtained in step 2.1 through the torch.nn.Parameter() function, and concatenate it into the image block sequence as [CLS] token. Adding [CLS] token is expressed as the following formula (3): X cls =[x cls ;X p ] (3) Among them, x cls Indicates the added [CLS] token, X p represents the characteristics of the image block obtained by dimension transformation in formula (2), X cls Indicates the image features after adding [CLS] token; Step 2.3, generate a learnable one-dimensional sequence with the same length as the image block sequence through the torch.nn.Parameter() function, and add it to the corresponding image block sequence as position embedding. The added position embedding is expressed as the following formula (4): X cls_pos =X cls +x pos +x size (4) Among them, X cls_pos represents the image features after adding [CLS] token and position embedding vector, x pos represents the added position embedding vector; x size Represents the image block size, which is calculated using the following formula: Step 2.4: Add the value X obtained in step 2.3 to cls_pos The sequence is input to each layer of the Transformer encoder of the image feature extraction module VIT. During the propagation process in each layer of the Transformer encoder of the image feature extraction module VIT, each pixel of all channels is first normalized. The normalization process is shown in formula (6), and finally the feature vector y of the image is obtained: Among them, X cls_pos represents the vector with position information of the image obtained in step 2.3, E(.) represents the calculated mean, Var(.) represents the calculated variance, y represents the normalized image vector obtained in the end, and ε is a constant; Step 2.5: The features of the image normalized in step 2.4 are passed through the Transformer encoder of the image feature extraction module VIT to obtain the image feature Attention (Q I ,K I ,V I ), the Transformer encoder calculation of each layer of the image feature extraction module VIT is as follows: Q I =w q y T (6) K I =w k y T (7) V I =w v y T (8) Among them, w q ,w k ,w v is the parameter in the Transformer encoder in the image feature extraction module VIT, and d represents Q I ,K I ,V I Dimensions; Step 2.6: embed the words of the text data obtained in step 1.1 into X T1 Input to the embedding layer of the text feature extraction module BERT, and embed the input word X T1 Plus the position embedding y pos , fragment embedding y seg , mark embedding y lab , get the text embedding feature X embed ,Right now: X embed =X T1 +y pos +y seg +y lab (10) Step 2.7: embed the text feature X obtained in step 2.6 embed , input to the first 6 layers of the text feature extraction module BERT, extract text features, and obtain the text feature Attention (Q T ,K T ,V T ), the calculation process is as follows: Among them, w q ,w k ,w v is the parameter in the Transformer encoder in the text feature extraction module BERT, d k Indicates Q T ,K T ,V T Dimension.

5. The method for classifying image-text pairs in multimodal network public opinion according to claim 4 is characterized in that: The specific process of step 3 is as follows: Step 3.1, the image features obtained in step 2.5 and the text features obtained in step 2.7 are used to form image-text pairs, the image-text pairs are divided into several batches, and the parameters in the BiFusion-CLIPNet model are copied to obtain a momentum model; Step 3.2: Input each batch of data constructed in step 3.1 into the momentum model to obtain image features and text features, and extract the [CLS] token added in step 2.2 as a negative sample and add it to the negative sample queue. At the same time, input each batch of data into the BiFusion-CLIPNet model to obtain image features and text features, and extract the [CLS] token added in step 2.2 as a positive sample pair and add it to the positive sample queue. Step 3.3, use the normalized Manhattan distance to calculate the similarity S between the texts T (T1, T2), the calculation formula is shown in formula (16): Among them, T1, T2 represent two different text features extracted by the BiFusion-CLIPNet model, n is the dimension of the feature, and T j Represents the j-th dimension of text feature T; Step 3.4, use the Pearson correlation coefficient to determine the similarity S between images I (I1,I2), the calculation formula is shown in formula (17): Among them, I1, I2 represent two different image features extracted by the BiFusion-CLIPNet model, n is the dimension of the feature, and I k represents the kth dimension of image feature I, Represents the sample mean of all images in the dataset; Step 3.5, use extended cosine similarity to calculate the similarity S between the image and text pairs IT (I, T), the calculation formula is shown in formula (18): Where I and T represent the image features and text features extracted by the BiFusion-CLIPNet model, respectively; ||I||2 and ||T||2 are the L1 norms of image features and text features, respectively; W is a weighting matrix; λ is the introduced regularization parameter; Step 3.6, use the contrastive learning method to align the image features and text features in the BiFusion-CLIPNet model. The contrastive learning loss function L ITC as follows: y(I,T)=β1S T +β2S I +β3S IT (19) p(I m ,T m )=β1S T +β2S I +β3S IT (20) In formula (19), y(I,T) represents the multidimensional similarity between image features and text features in the BiFusion-CLIPNet model. In formula (20), p(I m ,T m ) represents the calculation of the multidimensional similarity between image features and text features in the momentum model, and β1, β2, and β3 are weight parameters.

6. The method for classifying image-text pairs in multimodal network public opinion according to claim 5 is characterized in that: The specific process of step 4 is as follows: Step 4.1: For the image features and text features aligned in step 3: the image features are recorded as X I , X I The global characteristics of The text feature is recorded as X T , X T The global characteristics of Step 4.2: Using global image features And the global feature of the text is Calculate the self-attention score and take the weighted sum to get the feature of the text for the image, denoted as F T2I ; Step 4.2, taking the global features of the text as Global image features Calculate the self-attention score and take the weighted sum to get the image-to-text feature, denoted as F T2I ; Step 4.3: Compare the text obtained in steps 4.1 and 4.2 to the image feature F T2I And the image features F for text I2T Normalize them separately and fuse them through the multi-layer perceptron MLP to obtain the final bidirectional fusion feature F; In step 4.4, the bidirectional fusion feature F obtained in step 4.3 is input into the last 6 layers of Transformer encoder of the text feature extraction module BERT to extract deep fusion features and obtain the final image-text fusion feature.

7. The method for classifying image-text pairs in multimodal network public opinion according to claim 6 is characterized in that: The specific process of step 5 is as follows: Step 5.1, input the bidirectional fusion features extracted in step 4.4 into the classifiers of the BiFusion-CLIPNet model and the momentum model at the same time; Step 5.2, in the classifier, first pass the image-text fusion feature obtained in step 4.4 through a linear projection layer to map the feature dimension to a higher-dimensional feature space, then pass it through a ReLU() activation function for nonlinear mapping, and finally pass it through a linear projection layer to map the feature to the category dimension, and finally pass it through the Softmax() function to output the probability distribution of the category prediction of the bidirectional fusion contrast learning model; Step 5.3, during training, the model update process is as follows: During the training process, each batch of data is input into the momentum model, and after extracting the image-text fusion features, mapping and classification are performed through steps 5.1 and 5.2 to obtain the category prediction probability distribution of the momentum model; combined with the probability distribution of the category prediction of the bidirectional fusion contrast learning model output in step 5.2, the loss function is calculated, the gradient is back-transferred, and the parameters of the BiFusion-CLIPNet model are updated. The loss function L mod The calculation formula is as follows: Among them, y label Represents the true category probability distribution, p BFC represents the category probability distribution output by the BiFusion-CLIPNet model, p mod represents the category probability distribution output by the momentum model, β represents the weight considering the BiFusion-CLIPNet model and the momentum model, and H represents the cross entropy loss.

8. The method for classifying image-text pairs in multimodal network public opinion according to claim 7 is characterized in that: The specific process of step 6 is as follows: After updating the model in step 5.3, copy the parameters in the BiFusion-CLIPNet model to the momentum model according to the copy ratio k. The formula is as follows: i m ←kθ m +(1-m)θ (17) Among them, θ m represents the parameters of the momentum model, and θ represents the parameters of the BiFusion-CLIPNet model.

Images