Image-text pair matching method based on improved image convolutional neural network

Abstract

The invention provides an image-text pair matching method based on an improved image convolutional neural network, and the method comprises the steps: introducing the image convolutional neural network to further explore the relation between features on the basis of employing the convolutional neural network to extract image data and employing TextMlp to extract text data in an original method; as one picture and characters describing the content of the picture possibly contain not only one tag, the L2-GCN graph convolutional neural network is also applied to tag data, and the fine-grained relationship between tags is explored; and the loss is calculated by combining the comparison loss function and the annular loss function, so that Circle-Soft Loss is obtained, and the image-text pair matching accuracy is further improved.

Description

Technical Field

[0001] The present invention belongs to the field of cross-modal retrieval, and in particular relates to an image-text pair matching method based on an improved graph convolutional neural network. Background Art

[0002] In the field of image-text pair matching, see Figure 1 Although the development of deep learning technology has promoted the progress of cross-modal matching, it is still challenging to achieve high-quality text-image matching due to the semantic gap between text and images. Especially when faced with large-scale unlabeled datasets, how to effectively train models to understand the correlation between different modalities and achieve accurate matching on this basis is still a key problem that needs to be solved urgently.

[0003] In solving the problem of extracting image-text pair datasets into a common space, most methods rely only on ordinary convolutional neural networks (CNN) and ordinary multi-layer perceptron (TextMlp) models for data extraction, which exposes significant limitations in cross-modal retrieval. Specifically, both technologies fail to effectively capture the deep semantic associations between modalities: although CNN performs well in visual feature extraction, it lacks semantic understanding of image content; the TextMlp model ignores the sequence relationship and contextual information between words, making the text feature representation not rich enough, thus affecting the accuracy of cross-modal matching.

[0004] For labeled datasets, since the original labels are usually mixed with noise, redundant or inconsistent information, this further hinders the establishment of semantic correspondence between modalities and weakens the performance of the retrieval system and user experience. Figure 2 As shown in Figure 1, a picture may contain multiple elements, and its corresponding text description may also include multiple tags. If the tags are not cleaned and normalized, the diversity and uncertainty of the tags will make it difficult for the system to establish accurate image-text associations, resulting in a significant decrease in the relevance of the search results.

[0005] This method combines the graph convolutional neural network in deep learning and is committed to solving the difficult problem of how to improve the matching accuracy in image-text matching. Summary of the invention

[0006] The purpose of the present invention is to provide a method for matching image and text pairs based on an improved graph convolutional neural network, which processes the image, text and label data sets through the graph convolutional neural network and the proposed L2-GCN, uses the processed labels for data matching, and uses the Soft Contrastive Loss algorithm to calculate the image and text pair data, effectively improving the accuracy of image and text pair matching and ensuring the correctness of data analysis results. The technical solutions adopted are as follows:

[0007] A method for matching image and text pairs based on an improved graph convolutional neural network comprises the following steps:

[0008] Step 1: Extract the initial features of images and texts through TextMlp and convolutional neural network;

[0009] Step 2: Create a graph of pictures and texts and put it into the graph convolutional neural network to obtain the final picture features and text features;

[0010] Step 3: Pass the provided label data set into the proposed L2-GCN for update to obtain label data with enhanced adjacency features;

[0011] Step 4: Apply Circle-soft-Loss to the final image features and text features to update the image features and text features;

[0012] Step 5: Apply label loss to the label data output in step 3 and the predicted labels of the image features and text features output in step 4 to match them and further update the features.

[0013] Preferably, the L2-GCN in step 3 is specifically:

[0014]

[0015] A other =A·Diag∈R C

[0016] F other′ =A other ·F

[0017] neighbor = Sum(A other (Dim=0))(if>0=1)

[0018] F other =(A other ·F) / neighbor

[0019]

[0020] Where A - adjacency matrix;

[0021] Diag - a square matrix with 1s on the diagonal and all other elements are 0;

[0022] R C -The dimension of the Diag matrix is ​​the same as the number of nodes;

[0023] - Updated adjacency matrix;

[0024] W-weight matrix; B-bias term; F-feature matrix of node N;

[0025] A other - An adjacency matrix with 0 on the diagonal and values ​​in other positions preserved;

[0026] F other′ - a matrix containing all features except the features of the node itself;

[0027] Dim-A other The first dimension of the matrix;

[0028] neighbor-a neighbor number of the node;

[0029] F other -The matrix after average pooling;

[0030] fc - linear layer;

[0031] F end -The final label feature matrix.

[0032] Preferably, the Circle-soft-Loss model in step 4 is:

[0033]

[0034] in, -Inter-modality loss function; -Intra-class and inter-class loss functions; - Parameter threshold.

[0035] Compared with the prior art, the advantages of the present invention are:

[0036] Compared with traditional image-text pair matching methods, graph convolutional neural networks (CNNs) can capture local information and provide translation invariance when processing image and text pair data, while learning more abstract feature representations through deep architectures, thereby showing higher accuracy and efficiency in many tasks.

[0037] L2-GCN processes label features and their first-order neighbor features separately, and then recombines them to obtain the final label information. It can capture the structured information of the data and the relationship between features, thereby achieving more accurate cross-modal information fusion and alignment.

[0038] The proposed Circle-Soft Loss algorithm uses Circle Loss to process the relationship between classes within the modality, and uses Contrastive Loss to process the relationship between modalities, and integrates the two into a unified loss function, which can overcome the problems of insufficient compactness within the class, limited ability to adjust the distance between classes, and the semantic gap between modalities. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a cross-modal retrieval application scenario;

[0040] Figure 2 It is a schematic diagram of the picture and text pair;

[0041] Figure 3 Schematic diagram of the feature matrix of node N;

[0042] Figure 4 It is a flowchart of an image-text pair matching method based on an improved graph convolutional neural network;

[0043] Figure 5 A diagram of the model on which this method is based;

[0044] Figure 6 A comparison chart of the prediction accuracy of L2-GCN. DETAILED DESCRIPTION

[0045] The following is a more detailed description of the image-text pair matching method based on the improved graph convolutional neural network of the present invention in conjunction with a schematic diagram, wherein a preferred embodiment of the present invention is shown, and it should be understood that those skilled in the art can modify the present invention described herein while still achieving the advantageous effects of the present invention. Therefore, the following description should be understood as being widely known to those skilled in the art, and not as a limitation of the present invention.

[0046] A method for matching image and text pairs based on improved graph convolutional neural network, such as Figures 3 to 5 As shown, the following steps are included:

[0047] Step 1: Extract the initial features of images and text through convolutional neural networks and TextMlp networks.

[0048] Put the image features into the convolutional neural network, and then into the multi-layer perceptron to obtain the processed image and text features:

[0049] G = MLP(CNN(G))∈R d (1)

[0050] Where G = {g 1 ,…,g k},g i to represent each image.

[0051] Put the text features into TextMLP, and then into the multi-layer perceptron to get the processed text features:

[0052] T=MLP(TextMlp(T))∈R d (2)

[0053] Where T = {t 1 ,…,t k},t i To represent each text.

[0054] Step 2: Create a graph of images and texts and put it into the graph convolutional neural network to obtain the final image and text features.

[0055] The graphical representation of the picture modality is G=(g 1 ,g 2 ……,g k ) is a set of image features, is the relationship matrix between the feature nodes of the graph and text, which is composed of the adjacency matrix express.

[0056]

[0057]

[0058] in, and W δ are two learnable parameters, g i and g j is the matrix of the i-th picture and the matrix of the j-th picture.

[0059] R(g) obtained by formula (3) (4) i ,g j ) is g i ,g j The relationship between two images.

[0060] Use formula (3) (4) to calculate the relationship between a single image and all images R (g i ,g j );

[0061] All R(g i ,g j ) After merging, we get the adjacency matrix

[0062] The obtained adjacency matrix and the image features themselves are input into GCN to obtain the final image features:

[0063]

[0064] The graphical representation of the text modality is T=(t 1 ,t 2 ……,t k ) is a set of text features, is the relationship matrix between text feature nodes, which is composed of the adjacency matrix express.

[0065]

[0066]

[0067] in, and W δ are two learnable parameters, t i and t j is the matrix of the i-th picture and the matrix of the j-th picture.

[0068] R(t i ,t j ) is t i ,t j The relationship between two texts.

[0069] Formula (6) and (7) are used to calculate the relationship between a single text and all texts, and then the adjacency matrix is ​​obtained after merging.

[0070] The obtained adjacency matrix and the text features themselves are input into GCN to obtain the final text features used:

[0071]

[0072] Step 3: Pass the provided label data set into the proposed L2-GCN to update and obtain the label data F with rich adjacency features. end .

[0073] The dataset used is composed of image-text pairs. A picture has a corresponding text to describe the picture. For a pair of images and texts, they have a common label, such as Figures 2-3 shown.

[0074] For graph convolutional neural networks, the required data are: node features and adjacency matrix. Take a picture as an example. Figure 2 A picture contains four labels: sky, blue whale, sea, and clouds. These labels are the four nodes of the picture. Each node has its own characteristics, which are aggregated together as the final label feature matrix. There is a weight relationship between each node. These weight relationships are edges, which form an adjacency matrix. The adjacency matrix is ​​a square matrix with the dimension of the number of nodes, which represents the relationship between label nodes.

[0075] Next, we process the labels. The label data is marked as T = (N, E). Similar to the above, N is the node, the feature matrix of all N is marked as F, E is the edge of the connection relationship between each node, and the adjacency matrix is ​​A. It is input into the L2-GCN model. The following is the calculation process:

[0076]

[0077]

[0078] First, update the adjacency matrix A, Diag, which is a square matrix with 1s on the diagonal and all other elements are 0.

[0079] Subtracting this matrix from the matrix of all 1s gives a matrix with 1s on the diagonal and 0s in the rest of the positions, denoted as

[0080] Using a weight matrix and bias term by Multiply to update the feature matrix and get This feature matrix retains the characteristics of all nodes themselves.

[0081] A other =ADiag∈R C (11)

[0082] F other′ =A other ·F (12)

[0083] neighbor = Sum(A other (Dim=0))(if>0=1) (13)

[0084] F other =(A other ·F) / neighbor (14)

[0085]

[0086] A other is an adjacency matrix with zeros on the diagonal and values ​​elsewhere.

[0087] For A other For example, in the first dimension, as long as the value is greater than 0, it is considered to be related to a neighbor, so the value is set to 1 and the neighbor is obtained by summing in the first dimension.

[0088] Neighbor is used to determine the number of neighbors of a node: if the first dimension (i.e. A other If the sum of the values ​​on the first dimension of is greater than 0, it is considered that there is a relationship between the two nodes and the value is set to 1. (A neighbor is a vertex associated with a vertex).

[0089] Next, A other Multiply it with F and divide it by neighbor to get the matrix after average pooling.

[0090] Finally, F other and Pass through a linear layer and add the results to obtain the final label feature matrix F end .

[0091] This allows us to explore fine-grained relationships between labels. Fine-grained relationships refer to discovering and utilizing more specific and precise types of relationships in the data. These relationships may exist between entities or between features, going beyond the coarse level provided by general classification or clustering analysis.

[0092] L2-GCN can more effectively capture the local structural information between nodes because it not only considers the properties of the nodes themselves, but also incorporates the influence of directly connected neighbors, which can more accurately reflect the actual position and role of the nodes in the network. Secondly, by integrating the information of neighbors, L2-GCN enhances the model's ability to learn fine-grained relationships between nodes, which helps to discover deeper patterns and connections in the data. In addition, this method also improves the generalization ability of the model because it can better predict node attributes or links on new or unseen data, especially when dealing with sparse or incomplete data sets.

[0093] Step 4: Apply Circle-soft-Loss to the final image and text features obtained to further update the image and text features to facilitate the final matching with the label.

[0094] Circle-soft-Loss is a re-innovation that combines the advantages of Circle Loss and Soft Contrastive Loss. These two algorithms have obvious processing effects on intra-class, inter-class and inter-modal respectively. Combining the advantages of the two algorithms and adding a threshold between the two for subtle adjustments, we get Circle-soft-Loss, which plays an important role in global similarity matching.

[0095] Circle Loss: Reweights each similarity score to emphasize those with lower optimization and provides a unified formula for category-level and pair-level labels. Its core goal is to learn a feature representation that enhances the intra-class cohesion of samples in the feature space and increases the inter-class dispersion. This property helps the model effectively distinguish different categories in subsequent tasks, especially in fine-grained recognition problems.

[0096] Normalized image universal features and text universal features, here marked as V and T respectively, and G above f and T f correspond.

[0097]

[0098]

[0099] Next, we use the label L updated by L2-GCN to identify similar samples within the class. The label L here is the same as F in the above text. end correspond.

[0100] L=L·L T (maxL=1.0)(18)

[0101] Normalize L to ensure that the sum of each row is 1 (or close to 1 to prevent division by 0):

[0102]

[0103] Here L normij Indicates whether the i-th sample and the j-th sample belong to the same category. The value is 1 for yes and 0 for no.

[0104] Each value in the intra-class similarity sample matrix is ​​at most 1, indicating that the sample belongs entirely to a single class.

[0105] Just like calculating the similarity between tags, given the normalized text feature vector T and image feature vector V, calculate the similarity matrix between these feature vectors:

[0106] V=V·V T ∈R N·N

[0107] T=T·T T ∈R N·N (20)

[0108] V,T∈R N·N Represents the similarity matrix between samples, and N represents the number of image-text pairs.

[0109] After obtaining the similarity matrix between the image and the text, use these matrices to derive the positive sample matrix and the negative sample matrix:

[0110]

[0111] For each positive sample pair And negative sample pairs Circle Loss (intra-class and inter-class loss function) is defined as follows.

[0112]

[0113] is the positive sample matrix, is the negative sample matrix, m is the threshold for better similarity separation, and γ is the scaling factor.

[0114] Soft Contrastive Loss: Soft Contrastive Loss performs well in cross-modal contrastive learning. This loss function effectively utilizes the similarity matrix between views and enhances the model's alignment ability between different modalities through normalization. Compared with traditional contrast loss functions, soft contrast loss pays more attention to the similarity between modalities, thus providing stronger adaptability and robustness in multimodal tasks.

[0115]

[0116] Formula (23) is the inter-modal loss function of the image modality.

[0117]

[0118] Formula (24) is the inter-modal loss function of the text modality.

[0119]

[0120] Formula (25) is the inter-modal loss function.

[0121]

[0122] Formula (26) is the final formula of Circle-soft-Loss, The parameter threshold for the increase was slightly adjusted and the final value was set to 0.9. This loss function can increase the similarity between classes and within classes and the similarity between modalities, thereby increasing the matching accuracy.

[0123] Step 5: Compare the label data with the predicted label using classification loss.

[0124]

[0125] y represents the predicted label, is the predicted label of the image modality, The predicted labels for the text modality are given by GCNF end Indicates the actual label after update.

[0126] For generality, use |||| 2 (Euclidean norm) is used to calculate the distance between them. This formula further updates the modal features through the difference between the predicted label and the true label, which is performed simultaneously with formula (26).

[0127] like Figure 6As shown, the horizontal axis is the prediction type and the vertical axis is the prediction accuracy.

[0128] The prediction types include: picture prediction text, text prediction picture.

[0129] "Mir" refers to the labeled dataset in step 3. The MIRFlickr dataset contains 25,000 image-text pairs collected from the Flickr website, covering a wide range of topics and scenes. Each image and its corresponding text are annotated with labels describing the image and text content. These labels can be used for image classification and retrieval tasks. The dataset includes various types of images, such as natural landscapes, urban scenes, animals, people, and events. This diversity makes the dataset extremely valuable in a variety of research tasks. We selected 2,000 image-text pairs as the test set and the remaining pairs were used for the training set.

[0130] Take the prediction type "image prediction text" as an example. Figure 6 The steps to obtain the "prediction accuracy" in are as follows: mAP is used as the measurement accuracy strategy for this experiment. The mAP strategy jointly considers ranking information and accuracy, which is a performance evaluation standard widely used in cross-modal retrieval research. First, AP (average precision) is defined as follows:

[0131]

[0132] m is the number of samples among the N results returned that have at least one sample of the same category as the query. P(i) represents the precision of the i-th retrieval instance, and R(i) represents the correlation function. If the i-th result is relevant to the query, then R(i) = 1, otherwise R(i) = 0.

[0133] The final mAP formula is expressed as:

[0134]

[0135] Where Q is the total number of queries, AP q is the average precision of the qth query.

[0136] MAP is divided into three parts: 1) The accuracy of image retrieval given a query text (T2I). 2) The accuracy of text retrieval given an image (I2T). 3) The average value of T2I and I2T (mean value). We compare the two, namely Figure 6 T2I and I2T are obtained by processing the labeled data using GCN, GAT and L2-GCN under the same loss function. Figure 6 The effect of L2-GCN is shown to be better than the previous two.

[0137] The above is only a preferred embodiment of the present invention and does not limit the present invention in any way. Any technician in the relevant technical field, without departing from the scope of the technical solution of the present invention, makes any form of equivalent replacement or modification to the technical solution and technical content disclosed in the present invention, which does not depart from the content of the technical solution of the present invention and still falls within the protection scope of the present invention.

Claims

1. A method for matching image and text pairs based on an improved graph convolutional neural network, characterized in that: The following steps are involved: Step 1: Extract the initial features of images and texts through TextMlp and convolutional neural network; Step 2: Create a graph of pictures and texts and put it into the graph convolutional neural network to obtain the final picture features and text features; Step 3: Pass the provided label data set into the proposed L2-GCN for update to obtain label data with enhanced adjacency features; Step 4: Apply Circle-soft-Loss to the final image features and text features to update the image features and text features; Step 5: Apply label loss to the label data output in step 3 and the predicted labels of the image features and text features output in step 4, and further update the features by comparing the predicted labels with the true labels.

2. The image-text pair matching method based on the improved graph convolutional neural network according to claim 1, characterized in that: The L2-GCN in step 3 is specifically: A other =ADiag∈R C F other ′=A other ·F neighbor=Sum(A other (Dim=0))(if>0=1) F other =(A other ·F) / neighbor Where A - adjacency matrix; Diag - a square matrix with 1s on the diagonal and all other elements are 0; R C -The dimension of the Diag matrix is ​​the same as the number of nodes; -Updated adjacency matrix; W-weight matrix; b-bias term; F-feature matrix of node N; A other - An adjacency matrix with 0 on the diagonal and values ​​in other positions preserved; F other ′ - a matrix containing all features except the node’s own features; Dim-A other The first dimension of the matrix; neighbor-a neighbor number of the node; F other -The matrix after average pooling; fc - linear layer; F end -The final label feature matrix.

3. The image-text pair matching method based on an improved graph convolutional neural network according to claim 1, characterized in that: The Circle-soft-Loss model in step 4 is: in, -Inter-modality loss function; - intra-class and inter-class loss function; υ - parameter threshold.

Images