Training method and device of cross-modal retrieval model

Abstract

This invention discloses a cross-modal retrieval model training method and apparatus, relating to the field of artificial intelligence technology. The training method includes: extracting local text features and local image features from data pairs in the training set; inferring the relationship between local text features and local image features based on a graph convolutional neural network to obtain joint features; decoupling the joint features; performing cross-modal consistency learning on the data pairs to obtain consistency scores for image features and text features; and training a cross-modal retrieval model based on the decoupled text features, image features, and consistency scores. This invention integrates relational structure and graph similarity features, aggregating relational structure information between local features of the same modality and heterogeneous modalities through a graph neural network, and then reconstructing local features through feature decoupling. This fully captures relational structure information within and between text and image modalities, as well as contextual semantics, improving semantic understanding and resulting in higher accuracy of the trained cross-modal retrieval model.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a training method for a cross-modal retrieval model. Background Technology

[0002] With the development of the internet, various large social networking sites generate hundreds of millions of images and texts every day. To meet the needs of people in today's world, cross-modal retrieval has become a research hotspot in recent years. Cross-modal retrieval includes text-to-image retrieval and image-to-text retrieval. The cross-modal image-text retrieval task not only requires machines to recognize image and text information, but also to learn the relationship between image and text information, and to match the semantically closest image-text pairs to find one or more optimal matching image-text pairs.

[0003] Existing cross-modal retrieval methods can be broadly categorized into two types: global feature matching and local feature matching. The former focuses on learning global semantic alignment between the two modalities, mapping the entire image and text to the same joint embedding space. The latter emphasizes learning local feature alignment to infer the similarity between the image and text, i.e., the matching between visual regions and text words. However, methods based on global feature matching neglect local details and relationships between local regions in the image and text modalities. While these methods perform well in simple cross-modal image-text retrieval tasks involving only a single object, their results are unsatisfactory for more realistic and complex natural scenes. Methods based on local feature matching pay more attention to fine-grained features, such as salient regions in images and words in text, but still neglect intramodal relational structure information and contextual semantics, leading to ambiguity during retrieval and reducing accuracy. Summary of the Invention

[0004] This invention provides a training method and apparatus for a cross-modal retrieval model to solve the problem of low accuracy in cross-modal image and text retrieval for real and complex natural scenes.

[0005] This invention is achieved through the following technical solution:

[0006] A first aspect of the present invention provides a method for training a cross-modal retrieval model, comprising:

[0007] Obtain a cross-modal data training set, which contains multiple image-text data pairs;

[0008] Extract the initial local features of the text and the initial local features of the image of the data pair;

[0009] Based on graph convolutional neural networks, the relationship between the local features of the text and the local features of the image is inferred to obtain joint features;

[0010] Decouple the joint features to obtain text decoupling features and image decoupling features that fuse the semantic structure of intra-modal and inter-modal relationships.

[0011] Based on the local features and the decoupled features, cross-modal consistency learning is performed on the data pair to obtain the consistency score between image features and text features;

[0012] A cross-modal retrieval model is trained based on the consistency score and a loss function.

[0013] This invention provides a cross-modal retrieval method that integrates relational structure and graph similarity. It aggregates relational structure information between local features of the same modality and heterogeneous modalities through graph convolutional neural networks, and then reconstructs local features of images and text through feature decoupling, so as to integrate relational structure information. This fully captures relational structure information within and between image and text modalities, as well as contextual semantics, thereby improving the degree of semantic understanding and making the accuracy of cross-modal retrieval results higher.

[0014] Furthermore, local text features are extracted, including:

[0015] Forward and backward features of each text word are extracted based on a bidirectional gated recurrent unit;

[0016] The average of the forward and backward features of a word is taken as the word feature;

[0017] The local features of the text are obtained by combining the features of each word.

[0018] Furthermore, local image features are extracted, including:

[0019] The target region features of the image are extracted using an object detection model, and then the target region features are mapped into final features using a fully connected network to obtain local image features.

[0020] Furthermore, based on graph convolutional neural networks, the relationship between the local features of the text and the local features of the image is inferred to obtain joint features, including:

[0021] By concatenating local image features with local text features along the dimensions of image regions and words, we obtain the concatenated features:

[0022]

[0023] Where V represents local image features and T represents local text features;

[0024] A fully connected graph is built using a graph convolutional neural network to infer the relationship between image feature nodes and text feature nodes:

[0025]

[0026] in, Let H be the feature vector of the i-th node in the concatenated feature H, GCN be a graph convolutional neural network, l be the number of layers in the GCN, and h be the number of layers in the concatenated feature H. j For with h i Adjacent feature nodes, W l Let A be the learning parameters of the graph convolutional neural network, and let A be the fully connected weight matrix of the graph convolutional neural network.

[0027] The GCN model is a residual connection-based GCN model, that is:

[0028] H m+1 =GCN(GCN(H m )+H)

[0029] Where m is the number of layers in the GCN model, H m+1 H represents the image-text joint features obtained after GCN inference, and H represents the splicing features.

[0030] Furthermore, by decoupling the joint features, text decoupling features and image decoupling features that fuse intra-modal and inter-modal relational structural semantics are obtained, including:

[0031] The attention weights between the inferred joint features and the original features are calculated using an attention mechanism, and the inferred joint features are weighted to obtain text decoupling features and image decoupling features.

[0032] Furthermore, the method also includes:

[0033] The image decoupling features and the text decoupling features are respectively input into a multilayer feedforward neural network based on residual connections to obtain the final image decoupling features and text structure features.

[0034] Based on local features and the decoupled features, cross-modal consistency learning is performed on the data pair to obtain a consistency score between image features and text features, including:

[0035] Based on the local features and the final image decoupling features and text decoupling features, cross-modal consistency learning is performed on the data pairs to obtain consistency scores between local features.

[0036] Furthermore, cross-modal consistency learning is performed on the data pair based on local features and the decoupled features to obtain a consistency score between image features and text features, including:

[0037] A multi-granularity consistency learning method is used to perform cross-modal consistency learning on image-text data pairs in the training set, and multi-granularity consistency scores of image features and text features are obtained. The multi-granularity consistency scores include the consistency between local features, the consistency between local features and global features, and the consistency between global features.

[0038] Furthermore, the consistency score among local features is:

[0039]

[0040] in, The consistency score between the image decoupling features and the original image features. The consistency score between the text decoupling features and the original text features;

[0041] The consistency score between the local features and the global features is:

[0042]

[0043] in, The consistency score is calculated between global image features and local image features of interest to the text. The consistency score is calculated between global text features and local text features of interest in the image.

[0044] The consistency score among the global features is shown in the following formula:

[0045] S2(I,T)=Θ(G v G t )

[0046] Among them, G v and G t These are the global features of the image and text, respectively, and Θ is the cosine similarity calculation function.

[0047] Furthermore, the loss function adopted is the hinged ternary sorting loss function, expressed as:

[0048] L(I,T)=[σ-S(I,T)+S(I,T′)] + +[σ-S(I,T)+S(I′,T′)] -

[0049] Where σ is the edge parameter, [] + To indicate the maximum value, [] - The minimum value is taken. S(I,T) is the multi-granularity consistency score of image features and text features. I′ and T′ represent the negative samples corresponding to the positive samples of the minimum batch of images and text, respectively.

[0050] A second aspect of the present invention provides a training apparatus for a cross-modal retrieval model, comprising:

[0051] The data receiving module is configured to acquire a cross-modal data training set, which contains multiple image-text data pairs;

[0052] The feature extraction module is configured to extract local text features and local image features of the data pair;

[0053] The feature extraction module is also used to infer the relationship between the local features of the text and the local features of the image based on a graph convolutional neural network to obtain joint features;

[0054] The feature extraction module is also used to decouple the joint features to obtain text decoupling features and image decoupling features that fuse the semantics of intramodal and intermodal relational structures.

[0055] The consistency score learning module is configured to perform cross-modal consistency learning on the data pair based on the local features and the decoupled features to obtain a consistency score between the image features and the text features.

[0056] The training module is configured to train a cross-modal retrieval model based on a loss function established by the consistency score.

[0057] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0058] This invention proposes a graph-based aggregation and decoupling framework. It captures the semantic structure of intra- and inter-modal relational structures through image and text feature aggregation, and reconstructs image and text features through decoupling. This aggregation-decoupling process can fully capture intra- and inter-modal relational structural information. Furthermore, a comprehensive multi-granularity semantic consistency learning method is proposed to measure the difference between original and decoupled features, as well as the similarity between local and global features across different modalities, resulting in higher accuracy for cross-modal retrieval results. Attached Figure Description

[0059] To more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of the present invention and should not be considered as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort. In the drawings:

[0060] Figure 1 This is a flowchart of a training method for a cross-modal retrieval model according to an embodiment of the present invention;

[0061] Figure 2 This is a flowchart of another training method for a cross-modal retrieval model according to an embodiment of the present invention. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the embodiments and accompanying drawings. The illustrative embodiments and descriptions of the present invention are only used to explain the present invention and are not intended to limit the present invention.

[0063] It should be noted that the terms "comprising" and "having" and any variations thereof in the specification, claims, and accompanying drawings of this invention are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to other steps or units inherent in the device.

[0064] The terminology used in the various embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the various embodiments of this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which the various embodiments of this application pertain. The terms (such as those defined in a generally used dictionary) are to be interpreted as having the same meaning as in the context of the relevant technical field and are not to be interpreted as having an idealized or overly formal meaning, unless clearly defined in the various embodiments of this application.

[0065] The embodiments of the present invention provide a training method for a cross-modal retrieval model, which is applicable to cross-modal retrieval, used for inputting images to retrieve text, or inputting semantics to retrieve images, or for detecting multimodal online rumors and fake news, etc., which helps to obtain more accurate retrieval results.

[0066] like Figure 1 As shown, Figure 1 This is a flowchart of the training method for the cross-modal retrieval model of the present invention. The model training method includes:

[0067] Step S1: Obtain a cross-modal data training set, which contains multiple image-text data pairs;

[0068] Step S2: Extract the local text features and local image features of the data pair;

[0069] Step S3: Based on the graph convolutional neural network (GCN), infer the relationship between the local features of the text and the local features of the image to obtain joint features;

[0070] Step S4: Decouple the joint features to obtain text decoupling features and image decoupling features that fuse intra-modal and inter-modal relational structural semantics;

[0071] Step S5: Based on the local features and the decoupled features, perform cross-modal consistency learning on the data pair to obtain the consistency score between image features and text features;

[0072] Step S6: Based on the consistency score, establish a loss function to train a cross-modal retrieval model.

[0073] In step S1, cross-modal data refers to image data and corresponding text data, existing in the form of data pairs. Depending on the application purpose, data from a general domain can be collected as training sample data, or data from a specific domain can be collected for training. Cross-modal data pairs can be obtained from public databases such as Flickr30K, MS-COCO, etc., or through precise data collection; the specific method is not limited.

[0074] After acquiring a predetermined amount of cross-modal data, it can be divided into a training set, a test set, and a validation set. The training set is used for model training, the validation set is used to validate the model during training to assist in model learning, and the test set is used to evaluate the model's performance. The validation and testing methods are well-known in the art and are not limited here. This invention mainly focuses on detailing the training process.

[0075] In each text-image pair, there can be one text data corresponding to one image data, one text data corresponding to multiple image data, or multiple text data corresponding to one image data.

[0076] Furthermore, in the cross-modal data training set described in S1, each image data is matched with 5 text data.

[0077] In step S2, local text features are extracted from the text data in each image-text pair, and local image features are extracted from the image data.

[0078] The steps for extracting local features from an image include:

[0079] Step S201: Extract the target region features of the image using a target detection model;

[0080] In step S202, the target region features are mapped into final features using a fully connected network to obtain local image features.

[0081] Furthermore, the object detection model employs Faster R-CNN, which extracts K target region features O = {o} from each image. j |j = 1, ..., k, oj ∈R d The value of K is determined according to actual needs.

[0082] Then, a fully connected network is used to map the initial region features of the image into the final feature representation of the image:

[0083] V = {v} j |j = 1, ..., k, v j ∈R D The mapping method is shown in the following formula:

[0084] v j =W v o j +b v (1)

[0085] Where R d and R D W represents the dimension of the image region features. v and b v These are the learning parameters of a fully connected network.

[0086] Furthermore, if there are multiple image data in the image-text pair, the local features corresponding to the multiple image data are extracted and combined to obtain the final local image features.

[0087] The steps for extracting local features of text include:

[0088] Step S211: Extract the forward and backward features of each text word based on the bidirectional gated recurrent unit (Bi-GRU);

[0089] Step S212: Take the average of the forward and backward features of the word as the word feature;

[0090] Step S213: Combine the features of each word to obtain local text features.

[0091] The bidirectional gated recirculation unit (Bi-GRU) is shown in the following formula:

[0092]

[0093] Where m is the number of words in each text. and w represents the forward and backward feature representations of the text generated by GRU. i Given the word embedding vector for each word in the sentence, the final feature representation of the text is T = {t}. i |i = 1, ..., m, t i ∈R D Let} be the mean of the forward and backward features, as shown in the following formula:

[0094]

[0095] Where t i , which represents the characteristics of each word in the text.

[0096] Furthermore, if there are multiple text data in the image-text pair, the local text features corresponding to the multiple text data are extracted and combined to obtain the final local text features.

[0097] This yields the corresponding local features of the text and images for each data pair.

[0098] In step S3, the local features of the image and the local features of the text are first concatenated in the dimensions of image region and word to obtain concatenated features. Then, the concatenated features are inferred based on the graph convolutional neural network to obtain the relationship between the local features of the text and the local feature nodes of the image, and the inferred image-text joint feature matrix is ​​obtained.

[0099] Furthermore, local image features and local text features are concatenated at the dimensions of image regions and words, specifically as follows:

[0100]

[0101] Where V represents local image features, T represents text image features, and H represents splicing features.

[0102] Furthermore, based on graph convolutional neural networks, reasoning is performed on the concatenated features to obtain joint features, including:

[0103] First, a fully connected graph is built using a graph convolutional neural network (GCN) to infer the relationship between image features and text feature nodes. After GCN inference, the expression of the feature nodes is as follows:

[0104]

[0105] in, Let h be the feature vector of the i-th node in the concatenated feature H, l be the number of GCN layers, and h be the number of nodes. j for h i Adjacent feature nodes, W l Let A be the learning parameters of the graph convolutional neural network, and let A be the fully connected weight matrix of the graph convolutional neural network. Matrix A is represented as:

[0106] A = Attention(H, H, H)

[0107]

[0108] Where Attention(Q, K, V) is the classic attention mechanism calculation method, d kIt is a scaling factor;

[0109] The convolutional neural network (GCN) model is a GCN model based on residual connections, that is:

[0110] H m+1 =GCN(GCN(H m )+H) (7)

[0111] Where m is the number of layers in the GCN model, H m+1 H represents the image-text joint features obtained after GCN inference, and H represents the splicing features.

[0112] In step S4, the image and text joint features after joint inference are separated, and the image features and text features that fuse intra-modal and inter-modal relational structure semantics are obtained respectively.

[0113] By aggregating image and text features to capture the semantic structure of intramodal and intermodal relationships, and then reconstructing the image and text features through decoupling, this aggregation and decoupling paradigm can fully capture the information of intramodal and intermodal relationships, resulting in higher accuracy of cross-modal retrieval models trained using these features.

[0114] Furthermore, the attention weights between the joint features H after GCN inference and the original features are calculated using an attention mechanism, and the inferred joint features are weighted to obtain decoupled image features and decoupled text features, as shown in the following formula:

[0115] D v =Attention(V,H,H),D t =Attention(T,H,H) (8)

[0116] Among them, D v For image decoupling features, D t This is a text decoupling feature. Cross-modal consistency learning is performed based on this image decoupling feature and the text decoupling feature.

[0117] Furthermore, the above-mentioned image decoupling feature D v Text decoupling feature D t The inputs are then fed into a multilayer feedforward neural network based on residual connections to obtain the final decoupled features, as shown in the following equation:

[0118]

[0119]

[0120] in, and σ represents the final image and text decoupling features, σ is the ReLU activation function, FFN is the feedforward neural network, and m is the number of layers in the feedforward neural network.

[0121] Furthermore, cross-modal consistency learning is performed based on the final image decoupling features and text decoupling features.

[0122] In step S5, the similarity between the local features and decoupled features of the text and the image is calculated respectively to obtain the consistency score between the local features. Specifically, the expression is as follows:

[0123]

[0124] in, The consistency score between image decoupling features and image local features. The consistency score between text decoupling features and text local features is given, where μ0 is a scaling factor and Θ is a cosine similarity calculation function. In one embodiment of the present invention, step S5 employs a multi-granularity consistency learning method to perform cross-modal consistency learning on the image-text data features in the training set, obtaining multi-granularity consistency scores for image features and text features, including consistency between local features, consistency between local features and global features, and consistency between global features.

[0125] The final consistency score is obtained by combining the three consistency scores. Specifically, the three consistency scores can be summed proportionally.

[0126] This invention employs a multi-granularity semantic consistency learning method to measure the differences between original features and decoupled features, as well as the similarity between local and global features across different modalities, resulting in more comprehensive semantic understanding and higher accuracy of cross-modal retrieval results.

[0127] Consistency score S among local features o (I, T) adopts the above formula (10).

[0128] Furthermore, the consistency score S1(I,T) between local features and global features is shown in the following equation:

[0129]

[0130] Where is the consistency score between global image features and text-focused local image features, is the consistency score between global text features and text-focused local text features, is the scaling factor, and G is the scaling factor. v G t and are the global features of the image and text, respectively, and are calculated using a multi-head self-attention mechanism, as shown in the following equation:

[0131]

[0132] Where L is the number of heads in the self-attention mechanism, and W v and W t It is a linear transformation function. and The decoupled features of the image and text, enhanced by fusing the original features, are shown in the following formula:

[0133]

[0134] Where σ is the ReLU activation function and FFN is a feedforward neural network.

[0135] Furthermore, the feature consistency score S2(I,T) between global features is shown in the following formula:

[0136] S2(I,T)=Θ(G) v G t (14)

[0137] Among them, G v and G t These are the global features of the image and the text, respectively.

[0138] Furthermore, the final consistency score S(I,T) between the image and text is shown in the following formula:

[0139] S(I,T)=γ1×S0(I,T)+γ2×(S1(I,T)+S2(I,T)) (15)

[0140] Where γ1 and γ2 are the proportionality factors of the consistency score, and their sum is 1.

[0141] In step S6, the hinged ternary ranking loss function is used to train the cross-modal data training set to obtain the trained retrieval model.

[0142] The hinge ternary sorting loss function L(I,T) is shown in the following equation:

[0143] L(I,T)=[δ-S(I,T)+S(I,T′)] + +[δ-S(I,T)+S(I′,T)] - (16)

[0144] Where δ is the edge parameter, [] + To indicate the maximum value, [] - This indicates taking the minimum value, where I′ and T′ represent the negative samples corresponding to the minimum batch of positive images and text samples, respectively.

[0145] In one embodiment of the present invention, the training method of the cross-modal retrieval model is as follows: Figure 2 As shown, it includes the following steps:

[0146] Step 1: Obtain cross-modal datasets and divide them into training, test, and validation sets.

[0147] In this embodiment, the datasets used are Flickr30K and MS-COCO. Flickr30K contains 31,783 images (29,783 for training, 1,000 for testing, and 1,000 for validation), and MS-COCO contains 123,287 images (113,287 for training, 1,000 for testing, and 1,000 for validation). Each image data is matched with 5 text data.

[0148] Step 2: Image and text feature extraction to obtain fine-grained features of images and text.

[0149] In this implementation, the Faster R-CNN object detection model is used to extract 36 target region features for each image:

[0150] O={o j |j = 1, ..., k, o j ∈R d},

[0151] Then, a fully connected network is used to map the initial region features of the image into the final feature representation of the image:

[0152] V = {v} j |j = 1, ..., k, v j ∈R D}, as shown in the following formula:

[0153] v j =W v o j +b v (1)

[0154] Where R d and R D In this embodiment, R represents the dimension of the image region features. d Take 2048 dimensions, R D Take 1024 dimensions, W v and b v These are the learning parameters of a fully connected network.

[0155] Text features are generated using a bidirectional gated recurrent unit (Bi-GRU), as shown in the following equation:

[0156]

[0157]

[0158] Where m is the number of words in each text. and The forward and backward feature tables generated by GRU for the text are word embedding vectors for each word in the sentence. In this embodiment, the Word2Vec model is used to embed words into 300-dimensional word vectors. The final feature representation of the text is T = {t}. i |i = 1, ..., m, t i ∈R D} represents the mean of the forward and backward features, as shown in the following formula:

[0159]

[0160] Where t i It represents the characteristics of each word in the text.

[0161] Step 3: Feature aggregation, connecting image and text features, and using graph convolutional neural networks to jointly infer the structural semantics within and between images and text.

[0162] By concatenating image features with text features along the dimensions of image regions and words, we obtain the joint image-text feature H, as shown in the following formula:

[0163]

[0164] A fully connected graph is constructed using GCN to infer the relationship between image and text feature nodes. After GCN inference, the expression of the feature nodes is as follows:

[0165]

[0166] in, Let h be the feature vector of the i-th node in H, l be the number of GCN layers (in this embodiment, l is taken as 2 layers), and h be the feature vector of the i-th node in H. j for h i Adjacent feature nodes, W l Let A be the learning parameters of GCN, and let A be the fully connected weight matrix of GCN. The calculation method of A is shown in the following formula:

[0167] A = Attention(H, H, H)

[0168]

[0169] Where Attention(Q, K, V) is the classic attention mechanism calculation method, d k This is the scaling factor. Unlike traditional GCNs, the GCN model described here is a residual-connected GCN model, as shown in the following equation:

[0170] Hm+1 =GCN(GCN(H m (7)

[0171] Where m is the number of layers in the GCN model, and in this embodiment, m is 2 layers, H m+1 Let H be the image-text joint feature matrix obtained after GCN inference, where H represents the concatenated features. Each residual is obtained by adding this initial concatenated feature, taking the value of H from the last layer of GCN inference. m Decouple them.

[0172] Step 4: Feature decoupling. Separate the joint image and text features after joint inference to obtain image features and text features that fuse intra-modal and inter-modal relational structure semantics.

[0173] The attention weights between the joint features H after GCN inference and the original features are calculated using an attention mechanism, and the joint features after inference are weighted as shown in the following formula:

[0174] D v =Attention(V, H, H), D t =Attention(T, H, H) (8)

[0175] Among them, D v For the image features after initial decoupling, D t These are the text features after initial decoupling.

[0176] The initially decoupled features are then input into a multilayer feedforward neural network based on residual connections to obtain the final decoupled features, as shown in the following equation:

[0177]

[0178]

[0179] in, and σ represents the final image and text decoupling features, σ is the ReLU activation function, FFN is the feedforward neural network, and m is the number of layers in the feedforward neural network, which is 3 layers in this embodiment.

[0180] Step 5: Multi-granularity consistency learning, learning feature consistency between local features, between local and global features, and between global features, and summing them proportionally to obtain the final similarity score.

[0181] Multi-granularity consistency learning methods include consistency between local features, consistency between local features and global features, and consistency between global features. The final image-text similarity score is the proportional sum of these three consistency scores.

[0182] Consistency score S between local features o (I, T) is shown in the following equation:

[0183]

[0184] in, The consistency score between the image decoupling features and the original image features. Θ represents the consistency score between the text decoupling features and the original text features, μ0 is the scaling factor (μ0 is 0.7 in this embodiment), and Θ is the cosine similarity calculation function.

[0185] The consistency score S1(I,T) between local features and global features is shown in the following formula:

[0186]

[0187] in, The consistency score is calculated between global image features and local image features of interest to the text. G represents the consistency score between global text features and local text features of interest in the image, where μ1 is a scaling factor (0.5 in this embodiment). v With G t These are the global features of the image and text, respectively, calculated using a multi-head self-attention mechanism, as shown in the following equation:

[0188]

[0189] Where L is the number of heads in the self-attention mechanism, and W v and W t It is a linear transformation function. and The decoupled features of the image and text, enhanced by fusing the original features, are shown in the following formula:

[0190]

[0191] Where σ is the ReLU activation function and FFN is a feedforward neural network.

[0192] The feature consistency score S2(I,T) between global features is shown in the following formula:

[0193] S2(I,T)=Θ(G) v G t (14)

[0194] Among them, G v and G t These are the global features of the image and the text, respectively.

[0195] The final consistency score S(I,T) between the image and text is shown in the following formula:

[0196] S(I,T)=γ1×S0(I,T)+γ2×(S1(I,T)+S2(I,T)) (15)

[0197] Wherein, γ1 and γ2 are the scaling factors of the consistency score. In this embodiment, γ1 is 0.8 and γ2 is 0.2.

[0198] Step 6: Train the set cross-modal retrieval training set using the hinged ternary ranking loss function. After training, a retrieval model is obtained. Given an image or a piece of text, the model can retrieve the most matching sentence or image.

[0199] The hinge ternary sorting loss function L(I,T) is shown in the following equation:

[0200] L(I,T)=[δ-S(I,T)+S(I,T′)] + +[δ-S(I,T)+S(T',T)] - (16)

[0201] Where δ is the edge parameter, [] + To indicate the maximum value, [] - This indicates taking the minimum value, and I′ / T′ represents the negative sample corresponding to the minimum batch of positive image and text samples. This embodiment uses a 24GB 3090 graphics card for training, employing the Adam optimizer for a total of 25 epochs. The batch size is 64, the initial learning rate is set to 0.0002, and it decays by 10% every 8 epochs.

[0202] A second aspect of the present invention provides a training apparatus for a cross-modal retrieval model, comprising:

[0203] The data receiving module is configured to acquire a cross-modal data training set, which contains multiple image-text data pairs;

[0204] The feature extraction module is configured to extract local text features and local image features of the data pair;

[0205] The feature extraction module is also used to infer the relationship between the local features of the text and the local features of the image based on a graph convolutional neural network to obtain joint features;

[0206] The feature extraction module is also used to decouple the joint features to obtain text decoupling features and image decoupling features that fuse the semantics of intramodal and intermodal relational structures.

[0207] The consistency score learning module is configured to perform cross-modal consistency learning on the data pair based on the local features and the decoupled features to obtain a consistency score between the image features and the text features.

[0208] The training module is configured to perform cross-modal consistency learning on the data pairs based on the final image decoupling features and text structure features.

[0209] A third aspect of the present invention provides a cross-modal retrieval model, which is obtained by using the training method of the cross-modal retrieval model of any embodiment of the present invention.

[0210] A fourth aspect of the present invention provides a cross-modal retrieval method, which is implemented using a retrieval model obtained by training a cross-modal retrieval model according to any embodiment of the present invention.

[0211] A fifth aspect of the present invention provides an electronic device, including a processor, a memory, an input device, an output device, and a communication device; the number of processors in the computer device may be one or more, taking one processor as an example. The processor, memory, input device, and output device in the electronic device may be connected via a bus or other means.

[0212] As a computer-readable storage medium, memory can be used to store software programs, computer-executable programs, and modules. The processor executes various functional applications and data processing of the electronic device by running the software programs, instructions, and modules stored in memory 22, thereby implementing the cross-modal retrieval model training method and cross-module retrieval method of any of the above embodiments of the present invention.

[0213] The memory may primarily include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on terminal usage. Furthermore, memory 41 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some instances, the memory may further include memory remotely located relative to the processor, which can be connected to the electronic device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0214] The input device 42 can be used to receive image or text data, and the output device 43 can be used to output the search results corresponding to the input image or the input text.

[0215] A sixth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a training method for a cross-modal retrieval model and a cross-module retrieval method according to any embodiment of the present invention. The storage medium may be ROM / RAM, a magnetic disk, an optical disk, etc.

[0216] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A cross-modal retrieval model training method, characterized in that, The method includes: Obtain a cross-modal data training set, which contains multiple image-text data pairs; Extract the local text features and local image features of the data pairs; Based on graph convolutional neural networks, the relationship between the local features of the text and the local features of the image is inferred to obtain joint features; Decouple the joint features to obtain text decoupling features and image decoupling features that fuse the semantic structure of intra-modal and inter-modal relationships. Based on the local features and the decoupled features, cross-modal consistency learning is performed on the data pair to obtain the consistency score between image features and text features; A cross-modal retrieval model is trained based on the consistency score and a loss function. The method of inferring the relationship between the local features of the text and the local features of the image based on a graph convolutional neural network to obtain joint features includes: By concatenating local image features with local text features along the dimensions of image regions and words, we obtain the concatenated features: in, It is a local feature of the image. It is a local feature of the text; A fully connected graph is constructed using a graph convolutional neural network to infer the relationship between local features of an image and local feature nodes of text. in, For splicing features The first in The feature vectors of each node, where GCN is a graph convolutional neural network and l is the number of GCN layers. To and Adjacent feature nodes, These are the learning parameters for a graph convolutional neural network. It is the fully connected weight matrix of a graph convolutional neural network; The GCN model is a residual connection-based GCN model, that is: Where m is the number of layers in the GCN model, H represents the image-text joint features obtained after GCN inference, and H represents the splicing features. The step of performing cross-modal consistency learning on the data pair based on the local features and the decoupled features to obtain a consistency score between image features and text features includes: A multi-granularity consistency learning method is used to perform cross-modal consistency learning on image-text data pairs in the training set to obtain multi-granularity consistency scores for image features and text features. The multi-granularity consistency scores include consistency between local features, consistency between local features and global features, and consistency between global features. The consistency score among the local features is: in, The consistency score between image decoupling features and image local features. The consistency score between text decoupling features and text local features; The consistency score between the local features and the global features is: in, The consistency score is calculated between global image features and local image features of interest to the text. The consistency score is calculated between global text features and local text features of interest in the image. The consistency score among the global features is: in, and These are the global features of the image and the text, respectively. This is the function for calculating cosine similarity.

2. The method according to claim 1, characterized in that, The extraction of local text features includes: Forward and backward features of each text word are extracted based on a bidirectional gated recurrent unit; The average of the forward and backward features of a word is taken as the word feature; The local features of the text are obtained by combining the features of each word.

3. The method according to claim 1, characterized in that, The extraction of local image features includes: The target region features of the image are extracted using an object detection model, and then the target region features are mapped into final features using a fully connected network to obtain local image features.

4. The method according to claim 1, characterized in that, The decoupling of the joint features yields text decoupling features and image decoupling features that fuse intra-modal and inter-modal relational structural semantics, including: The attention weights between the inferred joint features and the original features are calculated using an attention mechanism, and the inferred joint features are weighted to obtain text decoupling features and image decoupling features.

5. The method according to claim 4, characterized in that, The method further includes: The image decoupling features and the text decoupling features are respectively input into a multilayer feedforward neural network based on residual connections to obtain the final image decoupling features and text decoupling features; Based on the local features and the decoupled features, cross-modal consistency learning is performed on the data pair to obtain a consistency score between image features and text features, including: Based on the local features and the final image decoupling features and text decoupling features, cross-modal consistency learning is performed on the data pairs to obtain consistency scores between local features.

6. The method according to claim 1, characterized in that, The loss function adopted is the hinged ternary sorting loss function, expressed as follows: in, For edge parameters, This indicates taking the maximum value. This indicates taking the minimum value. It is a multi-granularity consistency score of image features and text features. , These represent the negative samples corresponding to the minimum batch of positive images and text samples, respectively.

7. A training device for a cross-modal retrieval model, characterized in that, For performing the cross-modal retrieval model training method according to any one of claims 1-6, the training apparatus comprises: The data receiving module is configured to acquire a cross-modal data training set, which contains multiple image-text data pairs; The feature extraction module is configured to extract local text features and local image features of the data pair; The feature extraction module is also used to infer the relationship between the local features of the text and the local features of the image based on a graph convolutional neural network to obtain joint features; The feature extraction module is also used to decouple the joint features to obtain text decoupling features and image decoupling features that fuse the semantics of intramodal and intermodal relational structures. The consistency score learning module is configured to perform cross-modal consistency learning on the data pair based on the local features and the decoupled features to obtain a consistency score between the image features and the text features. The training module is configured to train a cross-modal retrieval model based on a loss function established by the consistency score.

Images