Image-text retrieval method and device based on multi-scale hybrid model, medium and product

Abstract

The invention discloses an image-text retrieval method and device based on a multi-scale hybrid model, a medium and a product, and relates to the technical field of image-text retrieval, and the method comprises the following steps: obtaining a reference object and a plurality of to-be-matched objects; respectively putting the reference object and the to-be-matched objects into a feature extraction model to obtain features of the reference object and features of the to-be-matched objects, and determining global similarity, multi-scale similarity and mixed similarity between the to-be-matched objects and the reference object; sorting the to-be-matched objects according to a sequence of the global similarities from large to small, and determining the first 2K to-be-matched objects as first candidate objects; sorting the first candidate objects according to a sequence of the multi-scale similarities from large to small, and determining the first K objects to be matched as second candidate objects; and sorting the second candidate objects according to a sequence of the mixing similarities from large to small to obtain an image-text retrieval result of the reference object. According to the method, the image-text retrieval accuracy and efficiency are improved.

Description

Technical Field

[0001] The present application relates to the field of image and text retrieval technology, and in particular to an image and text retrieval method, device, medium and product based on a multi-scale hybrid model. Background Art

[0002] Image-text retrieval aims to match images and texts based on the semantic similarity of content. Most existing methods extract visual representations based on Convolutional Neural Networks (CNNs) and text representations based on Recurrent Neural Networks (RNNs), and use triplet loss functions for optimization. Although these methods directly map global information to high-dimensional space, they ignore local information that reflects the relationship between objects and the distinction between important objects and redundant information due to the complexity of images. Using only global features, the model has a better understanding of the overall image, but the model lacks fine-grained perception of objects.

[0003] With the development of language representation models such as BERT, fine-grained features of text have become easy to obtain. At the same time, in the image modality, object detection methods such as Faster-RCNN have also shown excellent performance in extracting fine-grained features. This has made fine-grained retrieval increasingly popular in recent years. Fine-grained retrieval improves retrieval performance by exploring local correspondences between vision and language. Attention mechanisms are usually introduced to align elements of the two modalities.

[0004] In addition, with the rapid development of remote sensing image acquisition technology, the amount of remote sensing data has expanded rapidly in recent years, and many studies on image and text retrieval are specifically conducted on remote sensing images. Unlike natural images, remote sensing images usually have multi-scale and multi-target characteristics, and there is a lot of background noise. To address these problems, multi-scale image feature fusion technology is often used in remote sensing image processing. The input image is fed into a convolutional neural network, and convolution image features are extracted from different levels (receptive fields). These features are then fused to obtain multi-scale fusion features. After obtaining the fused features, downstream image and text retrieval tasks can be performed. This fusion process can integrate information at different scales and enhance the model's understanding of image details and context. Simple multi-scale image feature fusion is equivalent to the representation of medium-granularity features of the image between global and local features.

[0005] Most methods either focus on coarse-grained image and text representations or fine-grained local cross-modal correspondences between image regions and text words, resulting in low accuracy and efficiency in image-text retrieval. Summary of the invention

[0006] The purpose of this application is to provide a method, device, medium and product for image and text retrieval based on a multi-scale hybrid model to solve the problem of low accuracy and efficiency of image and text retrieval.

[0007] To achieve the above objectives, this application provides the following solutions:

[0008] In a first aspect, the present application provides a method for image-text retrieval based on a multi-scale hybrid model, comprising:

[0009] Acquire a reference object and multiple objects to be matched; the reference object is text or image; when the reference object is text, the object to be matched is an image; when the reference object is an image, the object to be matched is text;

[0010] The reference object and each of the objects to be matched are respectively put into a feature extraction model to obtain features of the reference object and features of each of the objects to be matched; the feature extraction model is obtained by training a multi-scale hybrid model, and the multi-scale hybrid model is constructed based on Faster-RCNN, Transformer, a multi-scale fusion module, a downsampling module and a text encoder;

[0011] Determining the global similarity, multi-scale similarity and mixed similarity between each of the objects to be matched and the reference object based on the features of each of the objects to be matched and the features of the reference object respectively;

[0012] Sort the objects to be matched according to the global similarity from large to small, and determine the first 2K objects to be matched as the first candidate objects; K>1;

[0013] Sort the first candidate objects in descending order of multi-scale similarity, and determine the first K objects to be matched as second candidate objects;

[0014] The second candidate objects are sorted in descending order according to the mixed similarity to obtain the image-text retrieval result of the reference object.

[0015] Optionally, the features of the image include: global features, local features and first multi-scale features; the features of the text include: sentence-level features, word-level features and second multi-scale features.

[0016] Optionally, the process of determining the feature extraction model includes:

[0017] Acquire a training set; the training set includes a plurality of image-text pairs, and the image-text pairs include sample images and corresponding sample texts;

[0018] Based on Faster-RCNN, Transformer, multi-scale fusion module, downsampling module and text encoder, construct the multi-scale hybrid model;

[0019] The multi-scale hybrid model is trained based on the training set using a triplet loss function to obtain the feature extraction model.

[0020] Optionally, the multi-scale hybrid model includes: an image branch and a text branch;

[0021] The image branch includes: a first sub-branch and a second sub-branch;

[0022] The first sub-branch is used to generate a first multi-scale feature; the second sub-branch is used to generate a global feature and a local feature;

[0023] The first sub-branch includes: a multi-scale fusion module, a Transformer module and a downsampling module connected in sequence;

[0024] The multi-scale fusion module is used to sample the image, generate multiple blocks, and project and splice the blocks to obtain multi-scale fusion image features;

[0025] The Transformer module is used to enhance the multi-scale fusion image features by using a self-attention mechanism to obtain enhanced multi-scale fusion image features;

[0026] The downsampling module is used to downsample the enhanced multi-scale fusion image features to obtain a first multi-scale feature;

[0027] The second sub-branch includes: a Faster-RCNN module and a first Transformer encoding module connected in sequence;

[0028] The Faster-RCNN module is used to extract regional features and regional borders of the image;

[0029] The first Transformer encoding module is used to determine global features and local features based on regional features and regional borders of the image;

[0030] The text branch is used to generate text features;

[0031] The text branch includes: a text encoder, a second Transformer encoding module and a splicing module connected in sequence;

[0032] The text encoder is used to encode the text to obtain a text encoding output;

[0033] The second Transformer encoding module is used to project the text encoding output to obtain sentence-level features and word-level features;

[0034] The concatenation module is used to concatenate sentence-level features and word-level features to obtain second multi-scale features.

[0035] Optionally, the text encoder is a BERT model.

[0036] Optionally, when the reference object is text and the object to be matched is an image, determining the global similarity, multi-scale similarity and mixed similarity between each object to be matched and the reference object based on the features of each object to be matched and the features of the reference object, respectively, includes:

[0037] Determine any object to be matched as the current image;

[0038] Determine the global similarity between the text and the current image based on the sentence-level features of the text and the global features of the current image;

[0039] Determine the local similarity between the text and the current image based on the word-level features of the text and the local features of the current image;

[0040] Determining a multiscale similarity between the text and the current image based on the second multiscale feature of the text and the first multiscale feature of the current image;

[0041] According to the global similarity, local similarity and multi-scale similarity between the text and the current image, the mixed similarity between the text and the current image is determined.

[0042] Optionally, when the reference object is an image and the object to be matched is text, determining the global similarity, multi-scale similarity and mixed similarity between each object to be matched and the reference object based on the features of each object to be matched and the features of the reference object, respectively, includes:

[0043] Determine any object to be matched as the current text;

[0044] Determine the global similarity between the image and the current text based on the global features of the image and the sentence-level features of the current text;

[0045] Determine the local similarity between the image and the current text based on the local features of the image and the word-level features of the current text;

[0046] Determining a multiscale similarity between the image and the current text based on a first multiscale feature of the image and a second multiscale feature of the current text;

[0047] According to the global similarity, local similarity and multi-scale similarity between the image and the current text, the mixed similarity between the image and the current text is determined.

[0048] In a second aspect, the present application provides a computer device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any of the above-mentioned image and text retrieval methods based on the multi-scale hybrid model.

[0049] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-mentioned image-text retrieval methods based on a multi-scale hybrid model.

[0050] In a fourth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned image-text retrieval methods based on a multi-scale hybrid model.

[0051] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0052] The present application discloses a method, device, medium and product for image-text retrieval based on a multi-scale hybrid model. First, a reference object and multiple objects to be matched are obtained; the reference object is text or image; when the reference object is text, the object to be matched is an image; when the reference object is an image, the object to be matched is text; then, the reference object and each object to be matched are respectively put into a feature extraction model to obtain the features of the reference object and the features of each object to be matched; the feature extraction model is obtained by training the multi-scale hybrid model, and the multi-scale hybrid model is based on Faster-RCNN, Transformer, multi-scale fusion module, downsampling module The method is constructed by a block and text encoder; secondly, based on the features of each object to be matched and the features of the reference object, the global similarity, multi-scale similarity and mixed similarity between each object to be matched and the reference object are determined; then, the objects to be matched are sorted in order from large to small according to the global similarity, and the first 2K objects to be matched are determined as the first candidate objects; again, the first candidate objects are sorted in order from large to small according to the multi-scale similarity, and the first K objects to be matched are determined as the second candidate objects; finally, the second candidate objects are sorted in order from large to small according to the mixed similarity, and the image-text retrieval results of the reference object are obtained. This application extracts the features of text and images through a feature extraction model obtained by training a multi-scale hybrid model, and uses the global similarity, multi-scale similarity and mixed similarity determined based on the features for sorting and screening, thereby improving the accuracy and efficiency of image-text retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0054] Figure 1 A schematic diagram of a flow chart of a method for image-text retrieval based on a multi-scale hybrid model provided in an embodiment of the present application;

[0055] Figure 2 This is a schematic diagram of the image-text retrieval architecture based on the multi-scale hybrid model;

[0056] Figure 3 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0057] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0058] The purpose of this application is to provide a method, device, medium and product for image and text retrieval based on a multi-scale hybrid model, aiming to improve the accuracy and efficiency of image and text retrieval.

[0059] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0060] In an exemplary embodiment, Figure 1 and Figure 2 As shown, a method for image-text retrieval based on a multi-scale hybrid model is provided, including:

[0061] Step 1: Obtain a reference object and multiple objects to be matched.

[0062] The reference object is text or image; when the reference object is text, the object to be matched is an image; when the reference object is an image, the object to be matched is text.

[0063] Specifically, images include natural images and remote sensing images.

[0064] Step 2: Add the reference object and each object to be matched to the feature extraction model respectively to obtain the features of the reference object and the features of each object to be matched.

[0065] Among them, the feature extraction model is obtained by training a multi-scale hybrid model, and the multi-scale hybrid model is built based on Faster-RCNN, Transformer, multi-scale fusion module, downsampling module and text encoder.

[0066] As an optional implementation, the features of the image include: global features, local features and first multi-scale features; the features of the text include: sentence-level features, word-level features and second multi-scale features.

[0067] As an optional implementation, the process of determining the feature extraction model includes:

[0068] A training set is obtained; the training set includes multiple image-text pairs, and the image-text pairs include sample images and corresponding sample texts.

[0069] A multi-scale hybrid model is constructed based on Faster-RCNN, Transformer, multi-scale fusion module, downsampling module and text encoder.

[0070] The multi-scale mixture model is trained based on the training set using the triplet loss function to obtain the feature extraction model.

[0071] Specifically, the triplet loss function is a widely used loss function in the field of image text retrieval. Calculate, then, use triplet loss to optimize the entire model. The triplet loss function is expressed as:

[0072]

[0073] Among them, Loss is the triple loss function value; γ is the margin, which is set to 0.2; S(I1,T1) is the mixed similarity between the anchor sample image I1 and the anchor sample text T1; is a negative sample image Mixed similarity with the anchor sample text T1; [κ] + =max(κ,0), that is, take the larger value between κ and 0; is the anchor sample image I1 and the negative sample text During training, the triplet loss makes the distance between anchor samples and negative samples as far as possible and as close as possible to positive samples. For performance reasons, negative samples are extracted from mini-batches instead of globally.

[0074] As an optional implementation, the multi-scale hybrid model includes: an image branch and a text branch.

[0075] The image branch includes: a first sub-branch and a second sub-branch.

[0076] The first sub-branch is used to generate the first multi-scale feature; the second sub-branch is used to generate the global feature and the local feature.

[0077] The first sub-branch includes: a multi-scale fusion module, a Transformer module and a downsampling module connected in sequence.

[0078] The multi-scale fusion module is used to sample the image, generate multiple blocks, and project and splice the blocks to obtain multi-scale fusion image features.

[0079] Specifically, compared with natural images, remote sensing images often contain a large number of targets, and the targets in the same scene may have considerable size differences. In addition, remote sensing images usually have extensive backgrounds and contexts that are irrelevant to the intended focus, and these background contents hinder the effective extraction of relevant information from remote sensing images. The image features obtained using a multi-scale method can increase the model's attention to salient objects while suppressing useless feature expressions. Therefore, the present application adds the multi-scale fusion image features extracted by the multi-scale fusion module to the process of image and text retrieval to improve the accuracy of image and text retrieval. The multi-scale fusion module samples the image through kernels of different sizes (i.e., 4×4, 8×8, 16×16, and 32×32), and generates 4 blocks with the same center but different scales. All blocks are projected and spliced ​​together to form multi-scale fusion image features.

[0080] The Transformer module is used to enhance the multi-scale fusion image features using the self-attention mechanism to obtain enhanced multi-scale fusion image features.

[0081] Specifically, the multi-scale fusion image features are enhanced through the self-attention mechanism of the Transformer module to make the features more expressive.

[0082] The downsampling module is used to downsample the enhanced multi-scale fusion image features to obtain the first multi-scale feature V M .

[0083] Specifically, the downsampling module performs a hierarchically designed downsampling operation on the enhanced multi-scale fusion image features to obtain the first multi-scale feature V M , V M ={i'0,i1',i'2,…,i' M-1 ,i' M}. Among them, i'0 is the first multi-scale feature V M The global feature element in; i1' is the first multi-scale feature V M The first local feature element in; i'2 is the first multi-scale feature V M The second local characteristic element in; i'M-1 is the first multi-scale feature V M The M-1th local characteristic element in; i' M is the first multi-scale feature V M The Mth local feature element in .

[0084] The second sub-branch includes: a Faster-RCNN module and a first Transformer encoding module connected in sequence.

[0085] The Faster-RCNN module is used to extract regional features and region borders of images.

[0086] Specifically, this application uses Faster-RCNN as an image encoder to extract the regional features and regional borders of the image. For a given image, Faster-RCNN generates each detected regional border b of the image j and regional characteristics f j The image representation is defined as V = {v0,v1,v2,…,v k-1 ,v k}, where v0 is the global image representation of the image; v j is the jth local image representation of the image, v j =[f j ,b j ], j = 1, 2,…, k, k is the number of image regions.

[0087] The first Transformer encoding module is used to determine the global feature V based on the regional features and regional borders of the image. G and local features V L .

[0088] Specifically, in order to improve the model's understanding of global and local information, this application connects the regional features and regional borders output by Faster-RCNN to the first Transformer encoding module, which consists of four identical standard Transformer encoder layers, each of which consists of a multi-head self-attention mechanism and a fully connected feedforward operator. The expression of the first Transformer encoding module is:

[0089] {V G ,V L}={i0,[i1,i2,…,i k-1 ,i k ]}=ITE(V).

[0090] Among them, V G is the global feature of the image, V G=i0; V L is a local feature, V L =[i1,i2,…,i k-1 ,i k ],i j is the jth element of the local feature; ITE(·) is the feature extraction operation of the first Transformer encoding module.

[0091] The text branch is used to generate text features.

[0092] The text branch includes: a text encoder, a second Transformer encoding module and a concatenation module connected in sequence.

[0093] The text encoder is used to encode text to obtain text encoding output.

[0094] Specifically, when the text is a sentence containing l words, the text encoder generates l+1 tags for the entire sentence and words to obtain the text encoding output; the text encoding output B is expressed as B = {b0, b1, b2, ..., b l-1 ,b l}, where b0 is the global representation of the sentence; b q is the local representation of q words, q = 1, 2, …, l.

[0095] As an optional implementation, the text encoder is a BERT model.

[0096] Specifically, traditional text feature representation methods use RNN-based models, such as Long Short-Term Memory (LSTM) or Gate Recurrent Unit (GRU). The pre-trained model BERT uses a self-attention-based Transformer structure to learn semantic relationships. This model has powerful feature extraction capabilities and can generate deep bidirectional language representations for word tokens.

[0097] The second Transformer encoding module is used to project the text encoding output to obtain the sentence-level feature T G and word-level features T L .

[0098] Specifically, the second Transformer encoding module includes four identical Transformer encoder layers, through which the text encoding output is projected into a D-dimensional embedding, and the text features are output. The text features include sentence-level features T G and word-level features T L .

[0099] Specifically, the expression of the second Transformer encoding module is:

[0100] {T G ,T L}={t0,[t1,t2,…,t l-1 ,t l ]}=TTE(B).

[0101] Among them, T G is the sentence-level feature, T G =t0;T L is the word-level feature, T L =[t1,t2,…,t l-1 ,t l ], t q is the qth element of the word-level feature; TTE(·) is the feature extraction operation of the second Transformer encoding module.

[0102] The concatenation module is used to concatenate sentence-level features and word-level features to obtain the second multi-scale feature T M .

[0103] Step 3: Based on the features of each to-be-matched object and the features of the reference object, determine the global similarity, multi-scale similarity and mixed similarity between each to-be-matched object and the reference object.

[0104] As an optional implementation, when the reference object is text and the object to be matched is an image, step 3 includes:

[0105] Step 311: Determine any object to be matched as the current image.

[0106] Step 312: Determine the global similarity between the text and the current image based on the sentence-level features of the text and the global features of the current image.

[0107] Specifically, the global similarity S1(V G ,T G ) is calculated as:

[0108]

[0109] Among them, ||·|| is the modulus of the matrix.

[0110] Step 313: Determine the local similarity between the text and the current image based on the word-level features of the text and the local features of the current image.

[0111] Specifically, the local similarity S2(V M ,T M ) is calculated as:

[0112]

[0113] Step 314: Determine the multiscale similarity between the text and the current image based on the second multiscale feature of the text and the first multiscale feature of the current image.

[0114] Specifically, the multi-scale similarity S3(V L ,T L ) is calculated as:

[0115]

[0116] Step 315: Determine the mixed similarity between the text and the current image based on the global similarity, local similarity and multi-scale similarity between the text and the current image.

[0117] Specifically, the calculation formula of the mixed similarity S(I,T) is:

[0118] S(I,T)=(1-α-β)S1(V G ,T G )+αS2(V M ,T M )+βS3(V L ,T L ).

[0119] Among them, α and β are hyperparameters used to adjust the ratio of the three similarities.

[0120] As an optional implementation, when the reference object is an image and the object to be matched is text, step 3 includes:

[0121] Step 321: Determine any to-be-matched object as the current text.

[0122] Step 322: Determine the global similarity between the image and the current text based on the global features of the image and the sentence-level features of the current text.

[0123] Step 323: Determine the local similarity between the image and the current text based on the local features of the image and the word-level features of the current text.

[0124] Step 324: Determine the multiscale similarity between the image and the current text based on the first multiscale feature of the image and the second multiscale feature of the current text.

[0125] Step 325: Determine the mixed similarity between the image and the current text based on the global similarity, local similarity and multi-scale similarity between the image and the current text.

[0126] Step 4: Sort the objects to be matched according to the global similarity from large to small, and determine the first 2K objects to be matched as the first candidate objects; K>1.

[0127] Step 5: Sort the first candidate objects in descending order of multi-scale similarity, and determine the first K objects to be matched as the second candidate objects.

[0128] Step 6: Sort the second candidate objects in descending order of mixed similarity to obtain the image-text retrieval result of the reference object.

[0129] Furthermore, the method of the present application is experimentally evaluated, which specifically includes the following contents.

[0130] The method of this application was trained and evaluated on four datasets: MS-COCO, Flickr30K, RSICD, and RSITMD. MS-COCO and Flickr30K are the most commonly used datasets for the study of general cross-modal image-text retrieval methods, and RSICD and RSITMD are the most commonly used datasets for cross-modal image-text retrieval of remote sensing images. Each image in these four datasets corresponds to five different descriptive sentences. MS-COCO has 123,287 images, which are divided into 113,287 images for training, 5,000 images for verification, and 5,000 images for testing. The method of this application is evaluated on 1K and 5K test images, respectively. MS-COCO1K uses 1K test images for five independent tests and takes the average to report the results, while MS-COCO5K refers to testing directly on the full 5K test images. The Flickr30K dataset contains 31,783 images, which divides the dataset into 29,783 images for training, 1,000 images for verification, and 1,000 images for testing. The RSICD dataset contains 10,921 remote sensing images covering 31 scenes, and the image size is fixed to 224 × 224. The RSITMD dataset contains 4,743 images, and the image size is fixed to 256 × 256. In addition, RSITMD provides 1 to 5 fine-grained keywords for each sample image. For the RSICD and RSITMD datasets, 80%, 10%, and 10% of the datasets are used as training sets, validation sets, and test sets, respectively.

[0131] Two evaluation indicators were selected: K recall rate (R@K) and mR, which are the most commonly used indicators in cross-modal image-text retrieval tasks. R@K (K=1, 5, and 10) represents the proportion of correct images or texts contained in the top K retrieval results. The higher the R@K value, the better the performance. This application performed image-to-text and text-to-image retrieval modes respectively, and obtained three R@K values ​​for each mode. The average of all six R@K values ​​of the two retrieval modes is expressed as mR.

[0132] Implementation details: The experiments were conducted on a workstation equipped with a GeForce RTX4090 GPU, an Intel Xeon Gold 6248R CPU, and 256 GB of memory. For different image sizes in different datasets, they were uniformly scaled to 256 × 256 pixels and input into the network. The embedding space dimensions of both images and texts were set to 1024. During training, Adam was used as the optimizer. The learning rate was set to 2e -4 , decaying by 0.7 every 20 training epochs. The epoch and batch sizes are 70 and 100, respectively. At inference time, the hyperparameters α and β in the hybrid similarity are both set to 0.3, and the number of samples K selected in the three-stage inference is 100. Subsequent experiments will analyze the sensitivity of these hyperparameters in detail. In order to make the experimental results more convincing, for each set of experiments, five identical training and validation runs are performed to obtain the average results.

[0133] Performance comparison: Several of the most typical and advanced image-text retrieval methods in recent years were selected and compared with the method of this application on the above four datasets. Among the methods used for comparison, VSE++, SCAN, CAMP, and CLIP reported experimental results on all four datasets. VSRN, TGDT, and CUSA only reported results on the MS-COCO and Flickr30K datasets. MTFN, AMFMN, LW-MCR, TGKT, and MSITA are image-text retrieval methods proposed for remote sensing images, and only reported results on the RSICD and RSITMD datasets. Table 1 is a brief introduction to these methods, N means for natural images, and R means for remote sensing images.

[0134] Table 1 Introduction to the current best cross-modal image-text retrieval methods for comparison

[0135]

[0136] Tables 2 to 5 show the experimental results on four datasets, MS-COCO (1K and 5K), Flickr30K, RSICD, and RSITMD. For the sake of fairness, this application directly quotes the experimental results of each model in its corresponding original paper. Overall, the model of this application achieves the highest average retrieval accuracy on the RSICD and RSITMD datasets, and the second highest average retrieval accuracy on the Flickr30K and MS-COCO (5K) datasets.

[0137] As can be seen from Tables 2 and 3, the method of this application has an mR that is about 20 higher than the coarse-grained method VSE++, and its performance also exceeds other fine-grained cross-modal retrieval methods SCAN, CAMP, VSRN, TGDT, and even exceeds the pre-trained model-based method CLIP on MS-COCO5K. The main reason is that other methods only use coarse-grained global image features and fine-grained local image features, while this application also considers medium-grained multi-scale fusion image features. On MS-COCO (1K and 5K) and Flickr30K, the method of this application is slightly inferior to CUSA, because CUSA uses the CLIP pre-trained model and complex semantic alignment technology, while the method of this application is not optimized in terms of fine-grained semantic alignment.

[0138] Table 2 Comparison of image and text retrieval performance on the MS-COCO dataset

[0139]

[0140] Table 3 Comparison of image and text retrieval performance on the Flickr30K dataset

[0141]

[0142] Tables 4 and 5 show the performance of each model on the two remote sensing datasets RSICD and RSITMD. Due to the complexity of remote sensing images, the retrieval accuracy of various methods is significantly lower than the results on natural image datasets. The mR of this application reached 24.1 and 37.9 on RSICD and RSITMD, respectively, exceeding all other methods. VSE++, SCAN, CAMP, which are not specifically proposed for remote sensing images, generally perform lower than other remote sensing image retrieval methods on these two datasets. Although AMFMN and MSITA also consider the multi-scale nature of remote sensing images, like other methods, they only consider coarse and fine granularity, without intermediate granularity, and their performance is lower than the method of the embodiment of this application. In short, this application adds multi-scale information on the basis of global information and local information, and improves cross-modal retrieval performance on both natural images and remote sensing images.

[0143] Table 4 Comparison of image and text retrieval performance on the RSICD dataset

[0144]

[0145]

[0146] Table 5 Comparison of image-text retrieval performance on RSITMD dataset

[0147]

[0148] According to the complexity of each modality and the significant semantic gap between different modalities, the method of this application can extract global features, local features and multi-scale features of images and texts, and improve the performance of image-text retrieval by exploring multi-granular feature similarities in shared space. Extensive experiments conducted on four common datasets of natural images and remote sensing images show that the method of this application outperforms other models for image-text retrieval tasks.

[0149] In an exemplary embodiment, a computer device is provided, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement a graphic and text retrieval method based on a multi-scale hybrid model.

[0150] In an exemplary embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, a method for image-text retrieval based on a multi-scale hybrid model is implemented.

[0151] In an exemplary embodiment, a computer program product is provided, including a computer program, which implements a multi-scale hybrid model-based image and text retrieval method when executed by a processor.

[0152] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 3 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for image and text retrieval based on a multi-scale hybrid model is implemented.

[0153] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0154] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0155] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0156] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0157] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0158] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A method for image-text retrieval based on a multi-scale hybrid model, characterized in that: The image-text retrieval method based on the multi-scale hybrid model includes: Acquire a reference object and multiple objects to be matched; the reference object is text or image; when the reference object is text, the object to be matched is an image; when the reference object is an image, the object to be matched is text; The reference object and each of the objects to be matched are respectively put into a feature extraction model to obtain features of the reference object and features of each of the objects to be matched; the feature extraction model is obtained by training a multi-scale hybrid model, and the multi-scale hybrid model is constructed based on Faster-RCNN, Transformer, a multi-scale fusion module, a downsampling module and a text encoder; Determining the global similarity, multi-scale similarity and mixed similarity between each of the objects to be matched and the reference object based on the features of each of the objects to be matched and the features of the reference object respectively; Sort the objects to be matched according to the global similarity from large to small, and determine the first 2K objects to be matched as the first candidate objects; K>1; Sort the first candidate objects in descending order of multi-scale similarity, and determine the first K objects to be matched as second candidate objects; The second candidate objects are sorted in descending order according to the mixed similarity to obtain the image-text retrieval result of the reference object.

2. The image-text retrieval method based on a multi-scale hybrid model according to claim 1, characterized in that: The features of the image include: global features, local features and first multi-scale features; the features of the text include: sentence-level features, word-level features and second multi-scale features.

3. The image-text retrieval method based on a multi-scale hybrid model according to claim 2, characterized in that: The process of determining the feature extraction model includes: Acquire a training set; the training set includes a plurality of image-text pairs, and the image-text pairs include sample images and corresponding sample texts; Based on Faster-RCNN, Transformer, multi-scale fusion module, downsampling module and text encoder, construct the multi-scale hybrid model; The multi-scale hybrid model is trained based on the training set using a triplet loss function to obtain the feature extraction model.

4. The image-text retrieval method based on a multi-scale hybrid model according to claim 3, characterized in that: The multi-scale hybrid model includes: an image branch and a text branch; The image branch includes: a first sub-branch and a second sub-branch; The first sub-branch is used to generate a first multi-scale feature; the second sub-branch is used to generate a global feature and a local feature; The first sub-branch includes: a multi-scale fusion module, a Transformer module and a downsampling module connected in sequence; The multi-scale fusion module is used to sample the image, generate multiple blocks, and project and splice the blocks to obtain multi-scale fusion image features; The Transformer module is used to enhance the multi-scale fusion image features by using a self-attention mechanism to obtain enhanced multi-scale fusion image features; The downsampling module is used to downsample the enhanced multi-scale fusion image features to obtain a first multi-scale feature; The second sub-branch includes: a Faster-RCNN module and a first Transformer encoding module connected in sequence; The Faster-RCNN module is used to extract regional features and regional borders of the image; The first Transformer encoding module is used to determine global features and local features based on regional features and regional borders of the image; The text branch is used to generate text features; The text branch includes: a text encoder, a second Transformer encoding module and a splicing module connected in sequence; The text encoder is used to encode the text to obtain a text encoding output; The second Transformer encoding module is used to project the text encoding output to obtain sentence-level features and word-level features; The concatenation module is used to concatenate sentence-level features and word-level features to obtain second multi-scale features.

5. The image-text retrieval method based on a multi-scale hybrid model according to claim 4, characterized in that: The text encoder is a BERT model.

6. The image-text retrieval method based on a multi-scale hybrid model according to claim 2, characterized in that: When the reference object is a text and the object to be matched is an image, determining the global similarity, multi-scale similarity and mixed similarity between each of the objects to be matched and the reference object based on the features of each of the objects to be matched and the features of the reference object, including: Determine any object to be matched as the current image; Determine the global similarity between the text and the current image based on the sentence-level features of the text and the global features of the current image; Determine the local similarity between the text and the current image based on the word-level features of the text and the local features of the current image; Determining a multiscale similarity between the text and the current image based on the second multiscale feature of the text and the first multiscale feature of the current image; According to the global similarity, local similarity and multi-scale similarity between the text and the current image, the mixed similarity between the text and the current image is determined.

7. The image-text retrieval method based on a multi-scale hybrid model according to claim 2, characterized in that: When the reference object is an image and the object to be matched is a text, determining the global similarity, multi-scale similarity and mixed similarity between each of the objects to be matched and the reference object based on the features of each of the objects to be matched and the features of the reference object, including: Determine any object to be matched as the current text; Determine the global similarity between the image and the current text based on the global features of the image and the sentence-level features of the current text; Determine the local similarity between the image and the current text based on the local features of the image and the word-level features of the current text; Determine a multiscale similarity between the image and the current text based on the first multiscale feature of the image and the second multiscale feature of the current text; According to the global similarity, local similarity and multi-scale similarity between the image and the current text, the mixed similarity between the image and the current text is determined.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the image-text retrieval method based on the multi-scale hybrid model as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image-text retrieval method based on the multi-scale hybrid model described in any one of claims 1 to 7 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the image-text retrieval method based on the multi-scale hybrid model described in any one of claims 1 to 7 is implemented.

Images