Image-text retrieval method based on multi-scale feature set extraction and alignment

Abstract

The application discloses a kind of based on multi-scale feature set extraction and alignment graphic-text retrieval method, using pre-training model to text and image data preliminary coding.Multiple times based on the downsampling operation of self-attention mechanism is used to complete multi-scale feature set extraction, and the granularity level of feature set is enriched;By the feature selection module based on slot attention mechanism, the multi-scale features from low to high from the downsampling path are filtered out, so that more significant fine-grained, multi-level feature set is obtained to represent image and text data;The cosine similarity of graphic-text feature is modeled using Gaussian distribution, and the Gaussian distribution model is iteratively updated, different weights are set for cosine similarity of different confidence, and the similarity between a single feature and another modal feature set is obtained;Through self-attention mechanism, the scores of different features in the mode are obtained, which are used as weights to participate in the calculation of the similarity of image-text pair.The application can solve the problem of graphic-text retrieval in real scene.

Description

Technical Field

[0001] This invention relates to a method for feature set extraction and alignment in the field of image and text retrieval. Background Technology

[0002] Over the past two decades, different types of media data, such as text, images, and videos, have grown rapidly. We often use different types of data to more accurately describe a topic or event. For example, web pages typically contain not only text descriptions but also matching images or videos. These different types of data are called multimodal data, exhibiting heterogeneous characteristics between modalities and having a wide range of applications.

[0003] The rapid growth of multimodal data has made it difficult for users to effectively search for information of interest, leading to the development of various retrieval technologies. However, most of these technologies are designed for intramodal retrieval, such as keyword-based and content-based searches. They only perform similarity searches within the same media type, such as text, image, audio, and video retrieval. In recent years, the widespread adoption of mobile devices and emerging social networking sites (such as Facebook, Flickr, YouTube, and Twitter) has increased users' demands for cross-modal data retrieval. For example, a user visiting the Great Wall might want to retrieve related text, audio, or other materials as a guide by taking photos. Typically, the purpose of cross-modal retrieval is to use data from one modality as a query to retrieve related data from other modalities.

[0004] Image-text cross-modal retrieval, as one of the core topics in cross-modal computing research, has always received high attention from academia and industry. Image-text retrieval is divided into two tasks: image-to-text retrieval and text-to-image retrieval. Since images and text are two different modalities, an unavoidable problem in image-text retrieval is aligning the information from both modalities within the same feature space. Current image-text retrieval models are mainly divided into two-stream structures and single-stream structures.

[0005] (1) Two-stream structure: Since images and text are two very different modalities, it is natural to encode images and text separately first, and then map their features to a common semantic space to calculate the similarity between images and text. Early image and text retrieval models were basically two-stream structure models. Their main improvements focused on how to map the features of the two modalities to the same semantic space and align them.

[0006] (2) Single-stream structure: With the widespread application of Transformer in the image field and the rise of pre-trained large models, some people have begun to propose whether it is possible to directly use the same encoder to encode two modalities and directly learn to align them into a semantic space, without having to encode the image and text separately and then perform alignment. The core of this type of model is: how to learn a better visual-text joint feature. Summary of the Invention

[0007] To address the issue that dual-stream image-text retrieval models cannot effectively utilize fine-grained information, this invention proposes an image-text retrieval method based on multi-scale feature set extraction and alignment. The proposed multi-scale feature extraction and alignment method can better utilize fine-grained information to calculate the similarity between images and text. Figure 1 As shown, the technical solution for implementing the present invention includes processes such as multi-scale feature set extraction, feature filtering, and set alignment.

[0008] The technical solution is as follows:

[0009] Step 1: Encode the image-text pairs (i.e., Image and Text) to be used for similarity calculation using a pre-trained encoder to generate the corresponding image local feature set and text local feature set (i.e., V and T).

[0010] Step 2: Input the image feature set and text feature set into the multi-scale feature set extraction module (i.e., SetPre[·]) to generate the corresponding multi-scale feature set (i.e., V). mul and T mul ).

[0011] Step 3: Combine the multi-scale feature set V mul and T mul The iterative input is fed into the feature selection module (i.e., FeatureSelect[·]) to filter out a representative multi-scale feature set (i.e., V). final and T final ).

[0012] Step 4: Input the multi-scale feature set into the set alignment module to calculate the similarity Sim(V) between a single text feature and the image feature set. final ,t j ).

[0013] Step 5: Calculate the self-attention weights (t) for different target data features using the self-attention mechanism. i ), and use it as a weight with Sim(V) final ,t i The similarity between image and text pairs is obtained by weighted summation.

[0014] Step 6: Train the multi-scale feature set extraction module, set alignment module, and weight prediction module using hard triplet loss, and update the network weight gradient using the Adam optimizer.

[0015] Furthermore, the encoder encoding process in step one is described as follows:

[0016]

[0017] ImageEncoder is a pre-trained image encoder, TextEncoder is a pre-trained text encoder, and D is the feature encoding dimension.

[0018] Furthermore, the process of extracting the multi-scale feature set in step two is described as follows:

[0019] S t =Downsampling(S t-1 )

[0020] S mul =Concat(S 0 ,...,S T ,S global )

[0021] Where S t S represents a set of local features at different scales. global This represents global features, Concat represents the concatenation operation, and Downsampling represents downsampling. The downsampling process utilizes a multi-head self-attention mechanism. Specifically, it is described as follows:

[0022] a. First, average pooling is used to obtain the global feature vector. The calculation process is as follows: S global =Avg(S 0 )

[0023] b. Subsequently, the multi-scale local feature set is calculated, and the output at each location is obtained using a multi-head self-attention mechanism. The input feature set is denoted as S. t-1 The calculation process is as follows:

[0024] 1) The original input is processed through the query matrix W q Key matrix W k Value matrix W v The mapping to the query space Q, key space K, and value space V is expressed by the following formula:

[0025] Q t =S t-1 W q K t =S t-1 W k V t=S t-1 W v

[0026] 2) Calculate the attention distribution at each location and sum the corresponding results using a weighted method:

[0027]

[0028] Where s(q) i ,k i ) is the fraction value after dot product and scaling.

[0029] 3) Next, downsampling is performed. The formula is as follows:

[0030] S t =MLP(context)

[0031] MLP stands for Multilayer Perceptron, which maps the input shape from n*D to...

[0032] c. Then, S t As input to the next layer, step bT is repeated to obtain local feature sets at different scales. These sets are then concatenated to obtain the final multi-scale feature set, which is expressed by the following formula:

[0033] S mul ={S 0 ,...,S T ,S global}

[0034] Furthermore, the specific process of feature selection in step three is as follows:

[0035] a. Divide according to scale, and sequentially divide the local feature set S t Input the feature selection module based on slot attention mechanism to obtain the latest feature set E t The calculation process is as follows:

[0036] 1) Set the local feature set S t Through the bond matrix W k Value matrix W v Mapping to the key space K and value space V, the learnable element slot matrix E t By querying matrix W q Mapping to the query space Q can be expressed by the following formula:

[0037] K t =S t W k V t =S t W v Q t =Et W q

[0038] in Num is a hyperparameter used to specify the number of features output by the feature selection module.

[0039] 2) Obtain S t and E t The attention graph A between them is represented as follows:

[0040] First, calculate the score matrix M, which can be expressed by the following formula:

[0041]

[0042] The attention map is then obtained through normalization, and is expressed by the following formula:

[0043]

[0044] 3) Update the element slot matrix E using the obtained attention map A. t The formula is as follows:

[0045]

[0046] Among them W o It is a learnable linear projection.

[0047] b. Subsequently, the global features are normalized and superimposed with the element slot matrix to obtain a multi-scale feature set after feature selection, expressed by the following formula:

[0048] S final =LN(E T )+[LN(S global )]

[0049] Where LN represents the normalization operation, and finally...

[0050] Furthermore, the specific process of feature set alignment in step four is as follows:

[0051] a. Represent the filtered image features as follows: The filtered text features are represented as follows: The cosine similarity matrix S between a single image feature and a single text feature is calculated and expressed by the following formula:

[0052]

[0053] Where i,j∈{1,…,Num}.

[0054] b. Calculate the similarity Sim(V) between a single text feature and the image feature set. final ,t i The calculation process is as follows:

[0055] 1) Establish a Gaussian distribution model for the similarity between text features and image features, expressed by the following formula:

[0056]

[0057] Where x is the value of the cosine similarity matrix S, μ is the mean of x, and σ is the standard deviation of x.

[0058] 2) Calculate the mean μ′ and variance σ′ of the cosine similarity matrix S in each batch:

[0059]

[0060] 3) Update the global mean μ and variance σ, expressed by the following formula:

[0061] μ=(1-λ)μ+λμ′

[0062] σ=(1-λ)σ+λσ′

[0063] Where λ is a hyperparameter used to specify the update rate of the mean and variance.

[0064] 4) Using the variance and standard deviation, obtain the latest Gaussian distribution model, and calculate the similarity thresholds with confidence levels of 90%, 80%, and 70%, respectively. The formulas are as follows:

[0065] threshold1 = μ + z 0.05 *σ

[0066] threshold2=μ+z 0.1 *σ

[0067] threshold3=μ+z 0.15 *σ

[0068] Where z p This is the z-value that makes the cumulative distribution function (CDF) of the normal distribution reach p.

[0069] 5) Based on the relationship between cosine similarity and confidence, set up a weight matrix, expressed by the following formula:

[0070]

[0071] β1, β2, and β3 are hyperparameters used to determine the weights of feature vectors of different importance.

[0072] 6) After weighted summation and normalization using the weight matrix, the similarity between a single text feature and the set of image features is obtained, expressed by the following formula:

[0073]

[0074] Furthermore, the specific process of feature set alignment in step five is as follows:

[0075] a. Calculate the weight (t) of each text feature vector in the whole. i The specific process is as follows:

[0076] 1) The original input is processed through the query matrix W q Key matrix W k Mapping to the query space Q and the key space K, expressed by the following formula:

[0077] Q = T final W q K = T final W k

[0078] in

[0079] 2) The attention weights are obtained by dot product of Q and K, summation, and normalization, and are expressed by the following formula:

[0080]

[0081] 3) The final image-text pair similarity is obtained by weighted summation, expressed by the following formula:

[0082]

[0083] Furthermore, the construction of the hard triplet loss in step six, which uses the non-matching text data with the highest image similarity and the non-matching image data with the highest text similarity in the batch as negative samples, is expressed by the following formula:

[0084]

[0085] in α is a hyperparameter. When the similarity difference between the matched and unmatched image-text pairs is greater than α, the loss will decrease to 0.

[0086] The advantages of this method are as follows: the image-text retrieval method based on multi-scale feature set extraction and alignment combines image or text features at different scales, enriching the granularity of the features and thus obtaining a more representative multi-scale feature set of images and text. Simultaneously, in the feature set matching stage, a Gaussian distribution model is established, making the feature weight allocation more granular and able to focus on more information. Therefore, the proposed image-text retrieval method based on multi-scale feature set extraction and alignment can effectively obtain a more representative feature set and achieve more accurate results in the set alignment stage. Attached image description:

[0087] Figure 1 This is a diagram of the overall architecture of the present invention.

[0088] Figure 2 This is the multi-scale feature extraction module of the present invention.

[0089] Figure 3 This is the feature selection block of the present invention.

[0090] Figure 4 This is the set alignment module of the present invention. Detailed Implementation

[0091] The following details a typical embodiment of the image and text retrieval method based on multi-scale feature set extraction and alignment according to the present invention, providing a further specific description of the method. It is important to note that the following embodiments are only used to further illustrate the method and should not be construed as limiting the scope of protection of the method. Any non-essential improvements and adjustments made to the method by those skilled in the art based on the above description still fall within the scope of protection of this invention.

[0092] This invention proposes a text and image retrieval method based on multi-scale feature set extraction and alignment, such as... Figure 1 As shown, the technical solution for implementing the present invention includes processes such as multi-scale feature set extraction, feature filtering, and set alignment.

[0093] Step 1: Encode the image-text pairs (i.e., Image and Text) to be used for similarity calculation using a pre-trained encoder to generate the corresponding image local feature set and text local feature set (i.e., V and T).

[0094] Step 2: Input the image feature set and text feature set into the multi-scale feature set extraction module (i.e., SetPre[·]) to generate the corresponding multi-scale feature set (i.e., V). mul and T mul ).

[0095] Step 3: Combine the multi-scale feature set V mul and T mulThe iterative input is fed into the feature selection module (i.e., FeatureSelect[·]) to filter out a representative multi-scale feature set (i.e., V). final and T final ).

[0096] Step 4: Input the multi-scale feature set into the set alignment module to calculate the similarity Sim(V) between a single text feature and the image feature set. final ,t j ).

[0097] Step 5: Calculate the self-attention weights (t) for different target data features using the self-attention mechanism. i ), and use it as a weight with Sim(V) final ,t i The similarity between image and text pairs is obtained by weighted summation.

[0098] Step 6: Train the multi-scale feature set extraction module, set alignment module, and weight prediction module using hard triplet loss, and update the network weight gradient using the Adam optimizer.

[0099] Furthermore, the encoder encoding process in step one is described as follows:

[0100]

[0101] ImageEncoder is a pre-trained image encoder, TextEncoder is a pre-trained text encoder, and D is the feature encoding dimension.

[0102] Furthermore, the process of extracting the multi-scale feature set in step two is described as follows:

[0103] S t =Downsampling(S t-1 )

[0104] S mul =Concat(S 0 ,...,S T ,S global )

[0105] Where S t S represents a set of local features at different scales. global This represents global features, Concat represents the concatenation operation, and Downsampling represents downsampling. The downsampling process utilizes a multi-head self-attention mechanism. Specifically, it is described as follows:

[0106] a. First, average pooling is used to obtain the global feature vector. The calculation process is as follows: S global =Avg(S0 )

[0107] b. Subsequently, the multi-scale local feature set is calculated, and the output at each location is obtained using a multi-head self-attention mechanism. The input feature set is denoted as S. t-1 The calculation process is as follows:

[0108] 1) The original input is processed through the query matrix W q Key matrix W k Value matrix W v The mapping to the query space Q, key space K, and value space V is expressed by the following formula:

[0109] Q t =S t-1 W q K t =S t-1 W k V t =S t-1 W v

[0110] 2) Calculate the attention distribution at each location and sum the corresponding results using a weighted method:

[0111]

[0112] Where s(q) i ,k i ) is the fraction value after dot product and scaling.

[0113] 3) Next, downsampling is performed. The formula is as follows:

[0114] S t =MLP(context)

[0115] MLP stands for Multilayer Perceptron, which maps the input shape from n*D to...

[0116] c. Then, S t As input to the next layer, step bT is repeated to obtain local feature sets at different scales. These sets are then concatenated to obtain the final multi-scale feature set, which is expressed by the following formula:

[0117] S mul ={S 0 ,...,S T ,S global}

[0118] Furthermore, the specific process of feature selection in step three is as follows:

[0119] a. Divide according to scale, and sequentially divide the local feature set St Input the feature selection module based on slot attention mechanism to obtain the latest feature set E t The calculation process is as follows:

[0120] 1) Set the local feature set S t Through the bond matrix W k Value matrix W v Mapping to the key space K and value space V, the learnable element slot matrix E t By querying matrix W q Mapping to the query space Q can be expressed by the following formula:

[0121] K t =S t W k V t =S t W v Q t =E t W q

[0122] in Num is a hyperparameter used to specify the number of features output by the feature selection module.

[0123] 2) Obtain S t and E t The attention graph A between them is represented as follows:

[0124] First, calculate the score matrix M, which can be expressed by the following formula:

[0125]

[0126] The attention map is then obtained through normalization, and is expressed by the following formula:

[0127]

[0128] 3) Update the element slot matrix E using the obtained attention map A. t The formula is as follows:

[0129]

[0130] Among them W o It is a learnable linear projection.

[0131] b. Subsequently, the global features are normalized and superimposed with the element slot matrix to obtain a multi-scale feature set after feature selection, expressed by the following formula:

[0132] S final =LN(ET )+[LN(S global )]

[0133] Where LN represents the normalization operation, and finally...

[0134] Furthermore, the specific process of feature set alignment in step four is as follows:

[0135] a. Represent the filtered image features as follows: The filtered text features are represented as follows: The cosine similarity matrix S between a single image feature and a single text feature is calculated and expressed by the following formula:

[0136]

[0137] Where i,j∈{1,…,Num}.

[0138] b. Calculate the similarity Sim(V) between a single text feature and the image feature set. final ,t i The calculation process is as follows:

[0139] 1) Establish a Gaussian distribution model for the similarity between text features and image features, expressed by the following formula:

[0140]

[0141] Where x is the value of the cosine similarity matrix S, μ is the mean of x, and σ is the standard deviation of x.

[0142] 2) Calculate the mean μ′ and variance σ′ of the cosine similarity matrix S in each batch:

[0143]

[0144] 3) Update the global mean μ and variance σ, expressed by the following formula:

[0145] μ=(1-λ)μ+λμ′

[0146] σ=(1-λ)σ+λσ′

[0147] Where λ is a hyperparameter used to specify the update rate of the mean and variance.

[0148] 4) Using the variance and standard deviation, obtain the latest Gaussian distribution model, and calculate the similarity thresholds with confidence levels of 90%, 80%, and 70%, respectively. The formulas are as follows:

[0149] threshold1 = μ + z 0.05 *σ

[0150] threshold2=μ+z 0.1 *σ

[0151] threshold3=μ+z 0.15 *σ

[0152] Where z p This is the z-value that makes the cumulative distribution function (CDF) of the normal distribution reach p.

[0153] 5) Based on the relationship between cosine similarity and confidence, set up a weight matrix, expressed by the following formula:

[0154]

[0155] β1, β2, and β3 are hyperparameters used to determine the weights of feature vectors of different importance.

[0156] 6) After weighted summation and normalization using the weight matrix, the similarity between a single text feature and the set of image features is obtained, expressed by the following formula:

[0157]

[0158] Furthermore, the specific process of feature set alignment in step five is as follows:

[0159] a. Calculate the weight (t) of each text feature vector in the whole. i The specific process is as follows:

[0160] 1) The original input is processed through the query matrix W q Key matrix W k Mapping to the query space Q and the key space K, expressed by the following formula:

[0161] Q = T final W q K = T final W k

[0162] in

[0163] 2) The attention weights are obtained by dot product of Q and K, summation, and normalization, and are expressed by the following formula:

[0164]

[0165] 3) The final image-text pair similarity is obtained by weighted summation, expressed by the following formula:

[0166]

[0167] Furthermore, the construction of the hard triplet loss in step six, which uses the non-matching text data with the highest image similarity and the non-matching image data with the highest text similarity in the batch as negative samples, is expressed by the following formula:

[0168]

[0169] in α is a hyperparameter. When the similarity difference between the matched and unmatched image-text pairs is greater than α, the loss will decrease to 0.

[0170] In this invention, the image-text pairs used during network training are from the MSCOCO dataset and the Flickr30k dataset. The datasets are expanded by a factor of 5 by performing operations such as image rotation, inversion, and scaling (while keeping the corresponding text data unchanged). During network training, the image pre-trained encoder uses Faster R-CNN, and the text pre-trained encoder uses BERT. In the encoding stage, the image and text data are encoded into a 1024-dimensional feature vector set. The iteration count T in the multi-scale feature set extraction is set to 2. The number of element slots Num in the feature selection module is set to 4. The learning rate λ in the feature set alignment module is set to 0.01, and the cosine similarity weights β1, β2, and β3 for different thresholds are set to 0.5, 0.3, and 0.2, respectively. The α value in the triplet loss function is set to 0.2. The corresponding weight gradients are updated using the Adam optimizer with an initial learning rate of 0.0001, and reduced to 0.00001 after 15 training epochs, for a total of 30 training epochs. The training environment for the proposed method is a PyTorch framework with an Nvidia RTX 3090 GPU and an Intel Core™ i7-7820X CPU at 3.60 Hz.

Claims

1. A method for retrieving images based on multi-scale feature set extraction and alignment, characterized in that, The method comprises the following steps: Step one, encode the image-text pair Image and Text to be calculated by the pre-trained encoder to generate the corresponding image local feature set V and text local feature set T; Step two, input the image feature set and the text feature set into the multi-scale feature set extraction module SetPre[·] respectively to generate corresponding multi-scale feature sets V mul and T mul ; Step three, collect multi-scale features set V mul and T mul Iterative input to the feature selection module FeatureSelect[·], filter out representative multi-scale feature set V final and T final ; Step four, input the multi-scale feature set into the set alignment module, and calculate the similarity Sim(V final ,t j ) of a single text feature to the image feature set. Step five, calculate the self-attention weight Weight(t i ) of different target data features by using the self-attention mechanism, and take it as a weight to weight-sum Sim(V final ,t i ) to obtain the similarity of the image-text pair; Step six, use hard triplet loss to train the multi-scale feature set extraction module, set alignment module, weight prediction module, and use Adam optimizer to update the network weight gradient; Step four is specifically as follows: a、the filtered image features are represented as the filtered text features are represented as a cosine similarity matrix S between the individual image features and the individual text features is calculated and is represented by the following formula: Wherein i, j ∈ {1, …, Num}; b. Calculate the similarity Sim(V final ,t i ) of the single text feature to the image feature set, the calculation process is: 1) Establish a Gaussian distribution model for the similarity between text features and image features, which is represented by the following formula: Where x is the value of the cosine similarity matrix S, μ is the mean of x, and σ is the standard deviation of x; 2) Calculate the mean μ' and variance σ' of the cosine similarity matrix S in each batch: 3) Update the global mean μ and variance σ, which are represented by the following formulas: μ = (1 - λ)μ + λμ' σ = (1 - λ)σ + λσ' Where λ is a hyperparameter used to specify the update rate of the mean and variance; 4) Use the variance and standard deviation to obtain the latest Gaussian distribution model, and calculate the similarity threshold with a confidence of 90%, 80%, and 70%, respectively, which is represented by the following formula: threshold1 = μ + z 0.05 *σ threshold2 = μ + z 0.1 *σ threshold3 = μ + z 0.15 *σ where z p is the value that makes the normal distribution cumulative distribution function (CDF) reach p; 5) According to the relationship between the cosine similarity and the confidence, set the weight matrix, which is represented by the following formula: Where β1, β2, and β3 are hyperparameters used to determine the weights of different important feature vectors; 6) Use the weight matrix to perform weighted summation and normalization to obtain the similarity between a single text feature and an image feature set, which is represented by the following formula: where Sim(V final ,t j ) is the similarity of the final cross-modal feature to the set. 2.The method of claim 1, wherein, The specific process of step two is as follows: S t = Downsample(S t-1 ) S mul = Concat(S 0 ,...,S T ,S global ) where S t denotes a set of local features of different scales, S global denotes global features, Concat denotes a concatenation operation, and Downsampling denotes a downsampling process that utilizes a multi-head self-attention mechanism, which is specifically expressed as: a、First, the global feature vector is obtained by using average pooling, and the calculation process is: S global = Avg(S 0 ); b、Subsequently, a multi-scale local feature set is calculated, a multi-head self-attention mechanism is used to obtain the output of each position, and the input feature set is recorded as S t-1 The calculation process is as follows: 1) The original input is mapped to the query space Q, the key space K and the value space V by the query matrix W q , the key matrix W k and the value matrix W v , which can be expressed as follows: Q t = S t-1 W q , K t = S t-1 W k , V t = S t-1 W v 2) Calculate the attention distribution at each position and perform weighted summation on the corresponding results: Where q i For Q t The i-th column, k i For K t The i-th column, v j For V t The j-th column, s(q i ,k i () is the fraction value after dot product and scaling; 3) Then use to perform downsampling, which is represented by the following formula: S t = MLP(context) MLP = MLP(input_shape = n * D, output_shape = 1) where MLP is a multi-layer perceptron mapping the input shape from n * D to c. Then, S t As the input of the next layer, repeat step b T times to obtain a set of local features at different scales, and concatenate them to obtain the final multi-scale feature set, which can be expressed as follows: S mul = {S 0 ,...,S T , S global} where S mul i.e. the final multi-scale feature set. 3.The method of claim 1, wherein, The specific process of step three is as follows: a、Divide the local feature set S by scale in turn t Input the feature selection module based on slot attention mechanism to obtain the latest feature set E t The calculation process is: 1) A local feature set S t By key matrix W k , value matrix W v Map to the key space K and the value space V, the learnable element slot matrix E t By query matrix W q Map to the query space Q, expressed as follows: K t = S t W k , V t = S t W v , Q t = E t W q wherein Num is a hyper-parameter for specifying the number of features output by the feature selection module, 2), obtaining S t and E t between the attention map A, the process is represented as: First, calculate the score matrix M, which is represented by the following formula: Then normalize to obtain the attention map, which is represented by the following formula: where M n,k is the value of the score matrix in the n-th row and k-th column, A n,k is the value of the attention map A in the n-th row and k-th column; 3) updating the element slot matrix E using the obtained attention map A t In formula, where W o is a learnable linear projection, b. Then, normalize the global features and add them to the element slot matrix to obtain the multi-scale feature set after feature screening, which is represented by the following formula: S final = LN(E T ) + [LN(S global )] where LN denotes a normalization operation, S global denotes a global feature vector, and finally 4.The method of claim 1, wherein, The specific process of step five is as follows: a. Calculate the weight (t) of each text feature vector in the whole. i The specific process is as follows: 1) The original input is mapped to the query space Q by the query matrix W q , the key matrix W k is mapped to the key space K, which can be expressed as follows: Q = T final W q , K = T final W k wherein 2) Perform dot product, summation, and normalization on Q and K to obtain the attention weight, which is represented by the following formula: where D h is the dimension of the space Q and K, M i,j is the value of the i-th row and j-th column of the attention weight matrix M; 3) Perform weighted summation to obtain the final image-text pair similarity, which is represented by the following formula: where V final is the multi-scale feature set corresponding to the image, and Sim(Image, Text) is the final cross-modal similarity.

Images