Image-text retrieval method based on multi-scale feature set extraction and alignment

Abstract

The invention discloses an image-text retrieval method based on multi-scale feature set extraction and alignment. Text and image data are preliminarily coded by using a pre-training model. Multi-scale feature set extraction is completed through multiple times of down-sampling operation based on a self-attention mechanism, and the granularity level of the feature set is enriched; multi-scale features from low to high from a down-sampling path are screened out through a feature selection module based on a slot attention mechanism, so that a more significant fine-grained and multi-level feature set is obtained to represent image and text data; gaussian distribution is used for modeling cosine similarities of the image-text features, a Gaussian distribution model is iteratively updated, different weights are set for the cosine similarities with different confidence degrees, and the similarity between a single feature and another modal feature set is obtained; and through a self-attention mechanism, obtaining scores of different features in a modal, and taking the scores as weights to participate in calculation of the similarity of the image text pairs. According to the method, the image-text retrieval problem in a real scene can be solved.

Description

Technical Field

[0001] The present invention relates to a feature set extraction and alignment method in the field of image and text retrieval. Background Art

[0002] In the past two decades, different types of media data such as text, images, and videos have grown rapidly. Usually, we use different types of data to more accurately describe a topic or event. For example, web pages usually contain not only text descriptions, but also matching images or videos. These different types of data are called multimodal data, which exhibits heterogeneous characteristics between modalities and has a wide range of applications.

[0003] The rapid growth of multimodal data makes it difficult for users to effectively search for information of interest, which has led to the emergence of various retrieval technologies. However, most of these search technologies are aimed at retrieval within a single modality, such as keyword-based retrieval and content-based retrieval, and they only perform similarity searches of the same media type, such as text retrieval, image retrieval, audio retrieval, and video retrieval. In recent years, the popularity of mobile devices and emerging social networking sites (such as Facebook, Flickr, YouTube, and Twitter) has led to higher and higher requirements for cross-modal data retrieval by users. For example, a user is visiting the Great Wall and hopes to retrieve relevant text, audio, and other materials as a guide by taking photos. Generally, the purpose of cross-modal retrieval is to use data in one modality as a query to retrieve related data in other modalities.

[0004] As one of the core topics of cross-modal computing research, image-text cross-modal retrieval has always been highly valued by academia and industry. Image-text retrieval is divided into two tasks: searching for text with images and searching for images with text. Since images and text are two modalities, an inevitable problem in image-text retrieval is to align the information of the two modalities in the same feature space. The current image-text retrieval models are mainly divided into two types: dual-stream structure and single-stream structure.

[0005] (1) Two-stream structure: Since images and text are two very different modalities, it is natural to encode the image and text separately first, and then map their features into a common semantic space to calculate the similarity between the image and text. Early image-text retrieval models were basically two-stream structure models, and their main improvements focused on how to map the features of the two modalities into the same semantic space and align them.

[0006] (2) Single-stream structure: With the widespread application of Transformer in the field of images and the rise of large pre-trained models, some people have begun to propose whether it is possible to directly use the same encoder to encode the two modalities and directly learn to align them into a semantic space without having to encode the image and text separately and then align them. The core of this type of model is: how to learn a better visual-text joint feature. Summary of the invention

[0007] In order to solve the problem that the dual-stream model of image-text retrieval cannot effectively utilize fine-grained information, the present invention proposes an image-text retrieval method based on multi-scale feature set extraction and alignment. The proposed multi-scale feature extraction and alignment method can better utilize fine-grained information to realize the similarity calculation of images and texts. Figure 1 As shown, the technical solution for implementing the present invention includes processes such as multi-scale feature set extraction, feature screening and set alignment.

[0008] The technical solution is as follows:

[0009] Step 1: Encode the image-text pair (i.e., Image and Text) for which similarity calculation is required through a pre-trained encoder to generate the corresponding image local feature set and text local feature set (i.e., V and T).

[0010] Step 2: Input the image feature set and the text feature set into the multi-scale feature set extraction module (i.e., SetPre[·]) to generate the corresponding multi-scale feature set (i.e., V mul and T mul ).

[0011] Step 3: Set the multi-scale feature set V mul and T mul Iterative input is sent to the feature selection module (i.e., FeatureSelect[·]) to screen out a representative multi-scale feature set (i.e., V final and T final ).

[0012] Step 4: Input the multi-scale feature set into the set alignment module and calculate the similarity Sim (V final ,t j ).

[0013] Step 5: Use the self-attention mechanism to calculate the self-attention weights Weight(t i ), and use it as a weight with Sim(V final ,t i )The weighted sum is used to obtain the similarity between image and text pairs.

[0014] Step 6: Use hard triplet loss to train the multi-scale feature set extraction module, set alignment module, and weight prediction module, and use the Adam optimizer to update the network weight gradient.

[0015] Furthermore, the encoding process of the encoder in step 1 is expressed as:

[0016]

[0017] Among them, ImageEncoder is a pre-trained image encoder, TextEncoder is a pre-trained text encoder, and D is the feature encoding dimension.

[0018] Furthermore, the process of extracting the multi-scale feature set in step 2 is expressed as:

[0019] S t = Downsampling(S t-1 )

[0020] S mul =Concat(S 0 ,...,S T ,S global )

[0021] Where S t Represents a set of local features of different scales, S global Represents global features, Concat represents concatenation, Downsampling represents downsampling, and the downsampling process uses a multi-head self-attention mechanism. The specific expression is:

[0022] a. First, average pooling is used to obtain the global feature vector. The calculation process is: S global =Avg(S 0 )

[0023] b. Then, the multi-scale local feature set is calculated, and the output of each position is obtained using the multi-head self-attention mechanism. The input feature set is recorded as S t-1 , the calculation process is:

[0024] 1) Pass the original input through the query matrix W q , the bond matrix W k , value matrix W v Mapping to query space Q, key space K and value space V, the formula is as follows:

[0025] Q t =S t-1 W q , K t =S t-1 W k , V t=S t-1 W v

[0026] 2) Calculate the attention distribution of each position and weight the corresponding results:

[0027]

[0028] Where s(q i ,k i ) is the fractional value after dot product and scaling.

[0029] 3) Then use downsampling. The formula is as follows:

[0030] S t =MLP(context)

[0031] Where MLP is a multi-layer perceptron that maps the input shape from n*D to

[0032] c. Then, S t As the input of the next layer, repeat step b T times to obtain local feature sets of different scales, and concatenate them to obtain the final multi-scale feature set, which can be expressed as follows:

[0033] S mul ={S 0 ,...,S T ,S global}.

[0034] Furthermore, the specific process of feature selection in step three is as follows:

[0035] a. Divide by scale and sequentially divide the local feature set S t Input the feature selection module based on the slot attention mechanism to obtain the latest feature set E t , the calculation process is:

[0036] 1) Set the local feature set S t Through the key matrix W k , value matrix W v Mapped to the key space K and the value space V, the learnable element slot matrix E t By querying the matrix W q Mapped to the query space Q, the formula is as follows:

[0037] K t =S t W k , V t =S t W v , Q t =Et W q

[0038] in Num is a hyperparameter used to specify the number of output features of the feature selection module.

[0039] 2) Get S t and E t The attention graph A between them, the process is expressed as:

[0040] First, calculate the score matrix M, which is expressed as follows:

[0041]

[0042] Then normalize to get the attention map, which is expressed as follows:

[0043]

[0044] 3) Update the element slot matrix E using the obtained attention map A t , which can be expressed as follows:

[0045]

[0046] Where W o is a learnable linear projection,

[0047] b. Then, the global features are normalized and superimposed with the element slot matrix to obtain a multi-scale feature set after feature screening, which is expressed as follows:

[0048] S final =LN(E T )+[LN(S global )]

[0049] Where LN represents the normalization operation, and finally

[0050] Furthermore, the specific process of aligning the feature set in step 4 is as follows:

[0051] a. Express the filtered image features as The filtered text features are expressed as The cosine similarity matrix S between a single image feature and a single text feature is calculated and expressed as follows:

[0052]

[0053] Where i,j∈{1,…,Num}.

[0054] b. Calculate the similarity Sim(V) between a single text feature and a set of image features final ,t i ), the calculation process is:

[0055] 1) Establish a Gaussian distribution model for the similarity between text features and image features, expressed as follows:

[0056]

[0057] Where x is the value of the cosine similarity matrix S, μ is the mean of x, and σ is the standard deviation of x.

[0058] 2) Calculate the mean μ′ and variance σ′ of the cosine similarity matrix S in each batch:

[0059]

[0060] 3) Update the global mean μ and variance σ, expressed as follows:

[0061] μ=(1-λ)μ+λμ′

[0062] σ=(1-λ)σ+λσ′

[0063] Where λ is a hyperparameter that specifies the update rate of the mean and variance.

[0064] 4) Use the variance and standard deviation to get the latest Gaussian distribution model, and calculate the similarity thresholds with confidence levels of 90%, 80%, and 70%, respectively. The formula is as follows:

[0065] threshold1=μ+z 0.05 *σ

[0066] threshold2=μ+z 0.1 *σ

[0067] threshold3=μ+z 0.15 *σ

[0068] where z p is the value of z that makes the normal cumulative distribution function (CDF) reach p.

[0069] 5) According to the relationship between cosine similarity and confidence, the weight matrix is ​​set, which is expressed as follows:

[0070]

[0071] Among them, β1, β2, and β3 are hyperparameters used to determine the weights of feature vectors of different importance.

[0072] 6) The similarity between a single text feature and an image feature set is obtained by weighted summation and normalization using the weight matrix, which can be expressed as follows:

[0073]

[0074] Furthermore, the specific process of aligning the feature set in step 5 is as follows:

[0075] a. Calculate the weight of each text feature vector in the whole i ), the specific process is:

[0076] 1) Pass the original input through the query matrix W q , the bond matrix W k Mapped to query space Q, key space K, the formula is as follows:

[0077] Q=T final W q , K = T final W k

[0078] in

[0079] 2) After dot product Q and K, sum and normalize them, we get the attention weight, which can be expressed as follows:

[0080]

[0081] 3) The final similarity of the image and text is obtained by weighted summation, which is expressed as follows:

[0082]

[0083] Furthermore, the construction of the hard triplet loss in step 6 is composed of the non-matching text data with the highest similarity to the image and the non-matching image data with the highest similarity to the text in the batch as negative samples, which is expressed by the following formula:

[0084]

[0085] in α is a hyperparameter. When the similarity difference between the matching image-text pair and the non-matching image-text pair is greater than α, the loss will drop to 0.

[0086] The beneficial effect of this method is that the image-text retrieval method based on multi-scale feature set extraction and alignment combines image or text features of different scales, enriches the granularity of features, and thus obtains a more representative image-text multi-scale feature set. At the same time, in the feature set matching stage, a Gaussian distribution model is established to make the feature weight distribution more fine-grained and able to focus on more information. Therefore, the proposed image-text retrieval method based on multi-scale feature set extraction and alignment can effectively obtain a more representative feature set and obtain more accurate results in the set alignment stage. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] Figure 1 It is the overall architecture diagram of the present invention.

[0088] Figure 2 It is the multi-scale feature extraction module of the present invention.

[0089] Figure 3 Blocks are selected for the features of the present invention.

[0090] Figure 4 It is the collection alignment module of the present invention. DETAILED DESCRIPTION

[0091] A typical embodiment of a method for image-text retrieval based on multi-scale feature set extraction and alignment of the present invention is described in detail below, and the method is further described in detail. It is necessary to point out here that the following embodiments are only used to further illustrate the method and cannot be understood as limiting the scope of protection of the method. Personnel skilled in the art in the field may make some non-essential improvements and adjustments to the method based on the above content of the method, which still fall within the scope of protection of the present invention.

[0092] The present invention proposes a method for image-text retrieval based on multi-scale feature set extraction and alignment, such as Figure 1 As shown, the technical solution for implementing the present invention includes processes such as multi-scale feature set extraction, feature screening and set alignment.

[0093] Step 1: Encode the image-text pair (i.e., Image and Text) for which similarity calculation is required through a pre-trained encoder to generate the corresponding image local feature set and text local feature set (i.e., V and T).

[0094] Step 2: Input the image feature set and the text feature set into the multi-scale feature set extraction module (i.e., SetPre[·]) to generate the corresponding multi-scale feature set (i.e., V mul and T mul ).

[0095] Step 3: Set the multi-scale feature set V mul and T mulIterative input is sent to the feature selection module (i.e., FeatureSelect[·]) to screen out a representative multi-scale feature set (i.e., V final and T final ).

[0096] Step 4: Input the multi-scale feature set into the set alignment module and calculate the similarity Sim (V final ,t j ).

[0097] Step 5: Use the self-attention mechanism to calculate the self-attention weights Weight(t i ), and use it as a weight with Sim(V final ,t i )The weighted sum is used to obtain the similarity between image and text pairs.

[0098] Step 6: Use hard triplet loss to train the multi-scale feature set extraction module, set alignment module, and weight prediction module, and use the Adam optimizer to update the network weight gradient.

[0099] Furthermore, the encoding process of the encoder in step 1 is expressed as:

[0100]

[0101] Among them, ImageEncoder is a pre-trained image encoder, TextEncoder is a pre-trained text encoder, and D is the feature encoding dimension.

[0102] Furthermore, the process of extracting the multi-scale feature set in step 2 is expressed as:

[0103] S t = Downsampling(S t-1 )

[0104] S mul =Concat(S 0 ,...,S T ,S global )

[0105] Where S t Represents a set of local features of different scales, S global Represents global features, Concat represents concatenation, Downsampling represents downsampling, and the downsampling process uses a multi-head self-attention mechanism. The specific expression is:

[0106] a. First, average pooling is used to obtain the global feature vector. The calculation process is: S global =Avg(S0 )

[0107] b. Then, the multi-scale local feature set is calculated, and the output of each position is obtained using the multi-head self-attention mechanism. The input feature set is recorded as S t-1 , the calculation process is:

[0108] 1) Pass the original input through the query matrix W q , the bond matrix W k , value matrix W v Mapping to query space Q, key space K and value space V, the formula is as follows:

[0109] Q t =S t-1 W q , K t =S t-1 W k , V t =S t-1 W v

[0110] 2) Calculate the attention distribution of each position and weight the corresponding results:

[0111]

[0112] Where s(q i ,k i ) is the fractional value after dot product and scaling.

[0113] 3) Then use downsampling. The formula is as follows:

[0114] S t =MLP(context)

[0115] Where MLP is a multi-layer perceptron that maps the input shape from n*D to

[0116] c. Then, S t As the input of the next layer, repeat step b T times to obtain local feature sets of different scales, and concatenate them to obtain the final multi-scale feature set, which can be expressed as follows:

[0117] S mul ={S 0 ,...,S T ,S global}.

[0118] Furthermore, the specific process of feature selection in step three is as follows:

[0119] a. Divide by scale and sequentially divide the local feature set St Input the feature selection module based on the slot attention mechanism to obtain the latest feature set E t , the calculation process is:

[0120] 1) Set the local feature set S t Through the key matrix W k , value matrix W v Mapped to the key space K and the value space V, the learnable element slot matrix E t By querying the matrix W q Mapped to the query space Q, the formula is as follows:

[0121] K t =S t W k , V t =S t W v , Q t =E t W q

[0122] in Num is a hyperparameter used to specify the number of output features of the feature selection module.

[0123] 2) Get S t and E t The attention graph A between them, the process is expressed as:

[0124] First, calculate the score matrix M, which is expressed as follows:

[0125]

[0126] Then normalize to get the attention map, which is expressed as follows:

[0127]

[0128] 3) Update the element slot matrix E using the obtained attention map A t , which can be expressed as follows:

[0129]

[0130] Where W o is a learnable linear projection,

[0131] b. Then, the global features are normalized and superimposed with the element slot matrix to obtain a multi-scale feature set after feature screening, which is expressed as follows:

[0132] S final =LN(ET )+[LN(S global )]

[0133] Where LN represents the normalization operation, and finally

[0134] Furthermore, the specific process of aligning the feature set in step 4 is as follows:

[0135] a. Express the filtered image features as The filtered text features are expressed as The cosine similarity matrix S between a single image feature and a single text feature is calculated and expressed as follows:

[0136]

[0137] Where i,j∈{1,…,Num}.

[0138] b. Calculate the similarity Sim(V) between a single text feature and a set of image features final ,t i ), the calculation process is:

[0139] 1) Establish a Gaussian distribution model for the similarity between text features and image features, expressed as follows:

[0140]

[0141] Where x is the value of the cosine similarity matrix S, μ is the mean of x, and σ is the standard deviation of x.

[0142] 2) Calculate the mean μ′ and variance σ′ of the cosine similarity matrix S in each batch:

[0143]

[0144] 3) Update the global mean μ and variance σ, expressed as follows:

[0145] μ=(1-λ)μ+λμ′

[0146] σ=(1-λ)σ+λσ′

[0147] Where λ is a hyperparameter that specifies the update rate of the mean and variance.

[0148] 4) Use the variance and standard deviation to get the latest Gaussian distribution model, and calculate the similarity thresholds with confidence levels of 90%, 80%, and 70%, respectively. The formula is as follows:

[0149] threshold1=μ+z 0.05 *σ

[0150] threshold2=μ+z 0.1 *σ

[0151] threshold3=μ+z 0.15 *σ

[0152] where z p is the value of z that makes the normal cumulative distribution function (CDF) reach p.

[0153] 5) According to the relationship between cosine similarity and confidence, the weight matrix is ​​set, which is expressed as follows:

[0154]

[0155] Among them, β1, β2, and β3 are hyperparameters used to determine the weights of feature vectors of different importance.

[0156] 6) The similarity between a single text feature and an image feature set is obtained by weighted summation and normalization using the weight matrix, which can be expressed as follows:

[0157]

[0158] Furthermore, the specific process of aligning the feature set in step 5 is as follows:

[0159] a. Calculate the weight of each text feature vector in the whole i ), the specific process is:

[0160] 1) Pass the original input through the query matrix W q , the bond matrix W k Mapped to query space Q, key space K, the formula is as follows:

[0161] Q=T final W q , K = T final W k

[0162] in

[0163] 2) After dot product Q and K, sum and normalize them, we get the attention weight, which can be expressed as follows:

[0164]

[0165] 3) The final similarity of the image and text is obtained by weighted summation, which is expressed as follows:

[0166]

[0167] Furthermore, the construction of the hard triplet loss in step 6 is composed of the non-matching text data with the highest similarity to the image and the non-matching image data with the highest similarity to the text in the batch as negative samples, which is expressed by the following formula:

[0168]

[0169] in α is a hyperparameter. When the similarity difference between the matching image-text pair and the non-matching image-text pair is greater than α, the loss will drop to 0.

[0170] In the example of the present invention, the image-text pairs used in network training are from the MSCOCO dataset and the Flickr30k dataset. By rotating, reversing, scaling and other operations on the image (the text data corresponding to the changed image remains unchanged), the dataset is expanded to 5 times. During network training, the image pre-training encoder uses FasterRCNN, and the text pre-training encoder uses Bert. In the encoding stage, the image and text data are encoded into a 1024-dimensional feature vector set. The number of iterations T described in the multi-scale feature set extraction is 2. The number of element slots Num described in the feature selection module is 4. The parameter learning rate λ described in the feature set alignment module is 0.01, and the cosine similarity weights β1, β2 and β3 of different thresholds are respectively 0.5, 0.3, and 0.2. The α in the triple loss function is 0.2. The corresponding weight gradient is updated using the Adam optimizer with an initial learning rate of 0.0001, and is reduced to 0.00001 after 15 rounds of training, and a total of 30 rounds of training are trained. The training environment of the proposed method is the Pytorch framework with Nvidia RTX 3090GPU and Intel Core(TM)i7-7820X CPU@3.60Hz.

Claims

1. A graphic and text retrieval method based on multi-scale feature set extraction and alignment, characterized in that: The steps include: Step 1: Encode the image and text pairs (i.e., Image and Text) for which similarity calculation is required through a pre-trained encoder to generate corresponding image local feature sets and text local feature sets (i.e., V and T); Step 2: Input the image feature set and the text feature set into the multi-scale feature set extraction module (i.e., SetPre[·]) to generate the corresponding multi-scale feature set (i.e., V mul and T mul ); Step 3: Set the multi-scale feature set V mul and T mul Iterative input is sent to the feature selection module (i.e., FeatureSelect[·]) to screen out a representative multi-scale feature set (i.e., V final and T final ); Step 4: Input the multi-scale feature set into the set alignment module and calculate the similarity Sim (V final ,t j ); Step 5: Use the self-attention mechanism to calculate the self-attention weights Weight(t i ), and use it as a weight with Sim(V final ,t i ) The weighted sum is used to obtain the similarity between the image and text pairs; Step 6: Use hard triplet loss to train the multi-scale feature set extraction module, set alignment module, and weight prediction module, and use the Adam optimizer to update the network weight gradient.

2. The image-text retrieval method based on multi-scale feature set extraction and alignment according to claim 1, characterized in that: The specific process of step 2 is as follows: S t =Downsampling(S t-1 ) S mul =Concat(S 0 ,...,S T ,S global ) Where S t Represents a set of local features of different scales, S global represents the global feature, Concat represents the concatenation operation, Downsampling represents downsampling, and the downsampling process uses the multi-head self-attention mechanism, which is specifically expressed as: a. First, average pooling is used to obtain the global feature vector. The calculation process is: S global =Avg(S 0 ); b. Then, the multi-scale local feature set is calculated, and the output of each position is obtained using the multi-head self-attention mechanism. The input feature set is recorded as S t-1 , the calculation process is: 1) Pass the original input through the query matrix W q , the bond matrix W k , value matrix W v Mapping to query space Q, key space K and value space V, the formula is as follows: Q t =S t-1 W q ,K t =S t-1 W k ,V t =S t-1 W v 2) Calculate the attention distribution of each position and weight the corresponding results: Where s(q i ,k i ) is the fractional value after dot product and scaling; 3) Then use downsampling, which is expressed as follows: S t =MLP(context) Where MLP is a multi-layer perceptron that maps the input shape from n*D to c. Then, S t As the input of the next layer, repeat step b T times to obtain local feature sets of different scales, and concatenate them to obtain the final multi-scale feature set, which can be expressed as follows: S mul ={S 0 ,...,S T ,S global } Where S mul This is the final multi-scale feature set.

3. The image-text retrieval method based on multi-scale feature set extraction and alignment according to claim 1, characterized in that: The specific process of step 3 is as follows: a. Divide by scale and sequentially divide the local feature set S t Input the feature selection module based on the slot attention mechanism to obtain the latest feature set E t , the calculation process is: 1) Set the local feature set S t Through the key matrix W k , value matrix W v Mapped to the key space K and the value space V, the learnable element slot matrix E t By querying the matrix W q Mapped to the query space Q, the formula is as follows: K t =S t W k ,V t =S t W v ,Q t =E t W q in Num is a hyperparameter used to specify the number of output features of the feature selection module. 2) Get S t and E t The attention graph A between them, the process is expressed as: First, calculate the score matrix M, which is expressed as follows: Then normalize to get the attention map, which is expressed as follows: 3) Update the element slot matrix E using the obtained attention map A t , which can be expressed as follows: Where W o is a learnable linear projection, b. Then, the global features are normalized and superimposed with the element slot matrix to obtain a multi-scale feature set after feature screening, which is expressed as follows: S final =LN(E T )+[LN(S global )] Where LN represents the normalization operation, and finally 4. The image-text retrieval method based on multi-scale feature set extraction and alignment according to claim 1, characterized in that: The specific process of step 4 is as follows: a. Express the filtered image features as The filtered text features are expressed as The cosine similarity matrix S between a single image feature and a single text feature is calculated and expressed as follows: Where i,j∈{1,…,Num}; b. Calculate the similarity Sim(V) between a single text feature and a set of image features final ,t i ), the calculation process is: 1) Establish a Gaussian distribution model for the similarity between text features and image features, expressed as follows: Where x is the value of the cosine similarity matrix S, μ is the mean of x, and σ is the standard deviation of x; 2) Calculate the mean μ′ and variance σ′ of the cosine similarity matrix S in each batch: 3) Update the global mean μ and variance σ, expressed as follows: μ=(1-λ)μ+λμ′ σ=(1-λ)σ+λσ′ Where λ is a hyperparameter used to specify the update rate of the mean and variance; 4) Using the variance and standard deviation to obtain the latest Gaussian distribution model, we calculated the similarity thresholds with confidence levels of 90%, 80%, and 70%, respectively, which are expressed as follows: threshold1=μ+z 0.05 *s threshold2=μ+z 0.1 *σ threshold3=μ+z 0.15 *s where z p is the z value that makes the normal distribution cumulative distribution function (CDF) reach p; 5) According to the relationship between cosine similarity and confidence, the weight matrix is ​​set, which is expressed as follows: Among them, β1, β2, and β3 are hyperparameters used to determine the weights of feature vectors of different importance; 6) The similarity between a single text feature and an image feature set is obtained by weighted summation and normalization using the weight matrix, which can be expressed as follows: Among them, Sim(V final ,t j ) is the final similarity of the cross-modal feature to the set.

5. The image-text retrieval method based on multi-scale feature set extraction and alignment according to claim 1, characterized in that: The specific process of step 4 is as follows: a. Calculate the weight of each text feature vector in the whole i ), the specific process is: 1) Pass the original input through the query matrix W q , the bond matrix W k Mapped to query space Q, key space K, the formula is as follows: Q=T final W q ,K=T final W k in 2) After dot product Q and K, sum and normalize them, we get the attention weight, which can be expressed as follows: 3) The final similarity of the image and text is obtained by weighted summation, which is expressed as follows: Among them, Sim(Image, Text) is the final cross-modal similarity.

Images