Cross-modal image-text retrieval method and device

Abstract

The invention is suitable for the technical field of image-text retrieval, and provides a cross-modal image-text retrieval method and device, and the method comprises the steps: obtaining a text retrieval image sample; based on the text retrieval image sample, training a preset cross-modal image-text retrieval model to obtain a trained cross-modal image-text retrieval model; the preset cross-modal image-text retrieval model adopts an image and text double-flow network parallel processing structure; the preset cross-modal image-text retrieval model comprises an image-text feature extraction module, a cross attention module, a feature pooling module and a feature fusion module; the cross attention module is used for capturing correlation among different modes of a text retrieval image sample; the feature pooling module is used for performing feature dimension reduction along the regional dimension in a learnable manner; and obtaining a to-be-retrieved image-text, and inputting the to-be-retrieved image-text into the trained cross-modal image-text retrieval model to obtain a retrieval result. The image-text retrieval method and device can improve the accuracy of image-text retrieval.

Description

Technical Field

[0001] The present application belongs to the field of image and text retrieval technology, and in particular, relates to a cross-modal image and text retrieval method and device. Background Art

[0002] Cross-modal image-text retrieval is an important part of the information retrieval field. Its goal is to establish associations between data in different modalities, such as querying corresponding descriptive text through images. One of the challenges facing cross-modal retrieval is to solve the problem of insufficient representation of feature information between modalities. Images and texts are closely related in high-level semantics, but there is still room for exploration of the underlying feature expression forms. Traditional cross-modal retrieval methods often rely on artificial prior knowledge, while modern deep learning methods automatically learn modal features through neural networks to measure similarity more accurately.

[0003] Traditional cross-modal retrieval methods manually extract features to characterize data of different modalities, and then construct a common representation space for similarity measurement. For example, the image part is represented by scale-invariant feature transform (SIFT). Cross-modal retrieval technology based on deep learning includes unsupervised, supervised and self-supervised methods. Unsupervised methods such as deep canonical correlation analysis (DCCA) map features through neural networks to enhance information expression. In supervised learning, such as the joint feature method, triple loss and semantic label classification are integrated into the loss function to improve model performance. Self-supervised learning uses large-scale pre-trained models such as VlLBERT, CLIP and BLIP series. The attention mechanism plays an important role in the field of cross-modal retrieval. It can focus on the key information between modalities and select significant features according to the context, thereby effectively realizing information interaction between modalities. For example, the SCAN model uses cross attention to reassign the attention weights to the paired text and image, and then calculates the overall similarity. The attention mechanism is not used alone. It is often widely used in feature extraction and image-text alignment in deep learning retrieval technology. For example, the image and text parts are encoded using self-attention-based visual Transformer and BERT respectively, or local attention is expanded into a new similarity vector representation for alignment.

[0004] However, due to the richness of semantic information in images and texts, as well as the huge heterogeneity of information between different modalities, the accuracy of image and text retrieval is not high. The performance of existing cross-modal retrieval models based on the cross-attention mechanism needs to be further improved. Summary of the invention

[0005] The embodiments of the present application provide a cross-modal image and text retrieval method and device, which can improve the accuracy of image and text retrieval.

[0006] This application is implemented through the following technical solutions:

[0007] In a first aspect, an embodiment of the present application provides a cross-modal image and text retrieval method, including:

[0008] Get text retrieval image samples;

[0009] Based on the text retrieval image samples, the preset cross-modal image-text retrieval model is trained to obtain a trained cross-modal image-text retrieval model; the preset cross-modal image-text retrieval model adopts an image and text dual-stream network parallel processing structure; the preset cross-modal image-text retrieval model includes an image-text feature extraction module, a cross-attention module, a feature pooling module and a feature fusion module; the cross-attention module is used to capture the correlation between different modes of the text retrieval image samples; the feature pooling module is used to perform feature dimensionality reduction along the regional dimension in a learnable manner;

[0010] Obtain the image and text to be retrieved, and input the image and text to be retrieved into the trained cross-modal image and text retrieval model to obtain the retrieval result.

[0011] In a second aspect, an embodiment of the present application provides a cross-modal image and text retrieval device, which executes the cross-modal image and text retrieval method of the first aspect, including:

[0012] A sample acquisition module is used to acquire text retrieval image samples;

[0013] The model training module is used to train the preset cross-modal image-text retrieval model based on the text retrieval image samples to obtain the trained cross-modal image-text retrieval model; the preset cross-modal image-text retrieval model adopts the image and text dual-stream network parallel processing structure; the preset cross-modal image-text retrieval model includes an image-text feature extraction module, a cross-attention module, a feature pooling module and a feature fusion module; the cross-attention module is used to capture the correlation between different modes of text retrieval image samples; the feature pooling module is used to perform feature dimensionality reduction along the regional dimension in a learnable manner;

[0014] The retrieval result output module is used to obtain the image and text to be retrieved, and input the image and text to be retrieved into the trained cross-modal image and text retrieval model to obtain the retrieval result.

[0015] Compared with the related art, the embodiments of the present application have the following beneficial effects:

[0016] The cross-modal image and text retrieval method and device of the embodiment of the present application adopts a cross-modal image and text retrieval model with cross-attention and feature aggregation for parallel processing of image and text dual-stream networks. The feature extraction module captures local feature representations rich in semantic information, and then the cross-attention module is used to capture the correlation between different modalities. It can effectively capture the local fine-grained semantic associations between images and texts, and the feature fusion module optimizes the fusion of image area and text sequence features, thereby better representing the deep semantic information of images and texts. The pooling strategy not only enhances the model's ability to capture cross-modal information, but also reduces the computational cost, making the model more efficient.

[0017] The beneficial effects of the above-mentioned second aspect embodiment refer to the beneficial effects of the first aspect embodiment, which will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0019] Figure 1 This is an application scenario diagram of a cross-modal image and text retrieval method and device provided in an embodiment of the present application;

[0020] Figure 2 It is the overall framework of the cross-modal image-text retrieval model provided in one embodiment of the present application;

[0021] Figure 3 It is a flowchart of a cross-modal image and text retrieval method provided by an embodiment of the present application;

[0022] Figure 4 This is an example of an image retrieval text application provided by an embodiment of the present application;

[0023] Figure 5 This is an example of a result of a text query image provided by an embodiment of the present application;

[0024] Figure 6 It is a schematic diagram of a process for training a preset cross-modal image-text retrieval model provided by an embodiment of the present application;

[0025] Figure 7 This is a schematic diagram of dynamically generated weights provided by an embodiment of the present application;

[0026] Figure 8 is a schematic diagram of a similarity matrix provided in an embodiment of the present application;

[0027] Fig. 9It is another example of image retrieval text provided by an embodiment of the present application;

[0028] Fig.10 This is another example of the result of a text query image provided by an embodiment of the present application;

[0029] Fig.11 It is a structural diagram of a cross-modal image and text retrieval device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0030] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0031] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0032] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0033] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0034] References to "an embodiment", "one embodiment" or "some embodiments" described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0035] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0036] The current cross-modal image-text retrieval model faces some technical shortcomings. On the one hand, although traditional methods such as SIFT, LDA and CCA can initially realize the association of features of different modalities, they rely on artificially designed features and mathematical statistical analysis, which makes them impractical in processing semantic associations in complex scenarios, especially in mining deep semantic relationships between images and texts.

[0037] On the other hand, although some methods based on deep learning have solved the above problems to a certain extent, they also have shortcomings. Although unsupervised learning models such as DCCA can map features of different modalities to the same space, they lack fine control over local features. Supervised learning methods such as the joint feature method, although they improve model performance through triple loss and semantic label classification loss, are still limited by the equal weight fusion of local features and fail to fully remove redundant features. Self-supervised learning methods such as VlLBERT, CLIP, and BLIP, a series of large visual-language pre-trained models, have shown excellent performance in pre-training on large-scale datasets, but the huge number of parameters and computing requirements make them difficult to deploy and apply in resource-limited environments.

[0038] Although the attention mechanism has achieved remarkable results in the field of cross-modal retrieval, it only relies on the attention mechanism for feature fusion, fails to effectively remove unnecessary information, and fails to fully characterize the local features of images and texts, resulting in limited retrieval performance and efficiency of the model. For example, although the SCAN model uses the cross-attention mechanism, it does not take into account the redundancy of image and text semantics, but aligns and calculates all regional information of the modalities equally. Some improved multi-head attention and local cross-attention methods have improved the model's feature selection ability, but have also failed to selectively use the most effective regional information based on the characteristics of the image and text itself.

[0039] Faster RCNN is a target detection model that can be used for image feature extraction. Its backbone network can use VGG, ResNet, etc. for feature map extraction. After extracting the feature map, the selected region and candidate box are generated. The redundant candidate boxes are eliminated by the non-maximum suppression method to obtain the final RoI (Region of Interest), and then through linear mapping, the image region features with good representation ability are obtained; BERT is a pre-trained language representation model based on Transformer. Its core is the encoder part of the Transformer architecture. It introduces the MLM (masked language model) mechanism to generate deep bidirectional language features and is widely used for sequence feature extraction; Currently, a variety of pooling strategies can achieve good regional dimensionality reduction effects, such as average pooling, maximum pooling, random pooling, etc. These operations are essentially the process of aggregating features in the local area dimension direction of the image or text, so that the multi-feature regions of the image and text are aggregated into a single feature region. In addition, the adaptive feature pooling method has also been applied in the field of cross-modal retrieval. For example, the GPO (Generalized Pooling Operator) model uses the automatic learning of the best pooling strategy to aggregate features on data of different modalities.

[0040] As mentioned in the background technology, the performance of the existing cross-modal retrieval model based on the cross-attention mechanism needs to be further improved. To this end, the present invention designs a cross-modal image-text retrieval model based on cross-attention and feature aggregation based on Faster RCNN and BERT to improve the accuracy of image-text retrieval.

[0041] The cross-modal image-text retrieval model of the present invention realizes bidirectional cross-modal retrieval of images and texts. The proposed model can play a role in general scenarios that require deep extraction of visual and text semantic features, such as the media field, security monitoring, and e-commerce. The model parameters of the present invention are trained using image and text datasets in general scenarios, and can also be adapted to vertical industry fields such as electricity.

[0042] Reference Figure 1 , which is an application scenario diagram of the cross-modal image-text retrieval method and device of the present invention. The prediction method and device run in the server 102. The server 102 obtains power data and related data from the power grid database 101.

[0043] The image is captured by the end-side acquisition device 101 and uploaded to the cloud 102. After retrieval, the text content such as the description of regulations and safety hazards related to the image is quickly obtained to simplify the process. For example, the inspection personnel input a picture at the terminal 103, which describes the construction situation of two workers. The model of the cloud 102 can extract the fine-grained information in the image and convert it into a vector retrieval problem, that is, the user-input picture and the text description information stored in the database are interactively modeled to generate a vector representation of all texts with rich semantic information and the input image, and then the most matching description is found through the similarity score, and the warning information text such as "not wearing a safety helmet" and "not wearing insulating gloves" in the database is returned to the terminal 103 to prevent safety accidents.

[0044] In order to improve the performance of the attention mechanism in cross-modal retrieval, the model of the cloud 102 is a cross-modal image-text retrieval model based on cross-attention and feature aggregation proposed by the present invention. The overall framework of the model is as follows: Figure 2 shown. Figure 3 is a flow chart of a cross-modal image and text retrieval method provided by an embodiment of the present application, referring to Figure 2 and Figure 3 , the cross-modal image-text retrieval method comprises:

[0045] Step 201, obtaining text retrieval image samples.

[0046] For example, examples of text retrieval image samples are as follows: Figure 4 and Figure 5 As shown, Figure 4 This is an application example of image retrieval text. The similarity value of the text is given in brackets, where the correctly retrieved text is indicated by "√", the wrong result is marked with "×", and the true label represents the matching text. Figure 5 This is an example of the result of a text query image. Each text has a unique image corresponding to it. The similarity value is also given in brackets, and the correct matching image is marked with a check mark.

[0047] If the cross-modal image-text retrieval model is applied to aspects such as safety management at power operation sites, a large number of safety hazard images and corresponding regulations and clause descriptions can be obtained to form text retrieval image samples.

[0048] Step 202: Based on the text retrieval image samples, a preset cross-modal image-text retrieval model is trained to obtain a trained cross-modal image-text retrieval model.

[0049] Among them, the preset cross-modal image and text retrieval model adopts a dual-stream network parallel processing structure of image and text; the preset cross-modal image and text retrieval model includes an image and text feature extraction module, a cross-attention module, a feature pooling module and a feature fusion module; the cross-attention module is used to capture the correlation between different modalities of text retrieval image samples; the feature pooling module is used to perform feature dimensionality reduction along the regional dimension in a learnable way.

[0050] For example, the core of the cross-modal image-text retrieval model is a combination of the cross-attention mechanism and the feature aggregation strategy. The cross-attention mechanism can effectively capture the local fine-grained semantic associations between images and texts, while the feature aggregation strategy optimizes the fusion of image region and text sequence features through learnable weight parameters, thereby better representing the deep semantic information of images and texts.

[0051] The feature pooling module is a learnable pooling module that can process the features of variable-length sequences, learn weight parameters through neural networks, effectively aggregate the features of images and texts, and further improve the retrieval performance of the model.

[0052] Step 203, obtaining the image and text to be searched, and inputting the image and text to be searched into the trained cross-modal image and text search model to obtain the search result.

[0053] This embodiment adopts a cross-modal image-text retrieval model with cross-attention and feature aggregation for parallel processing of image and text dual-stream networks. The feature extraction module captures local feature representations rich in semantic information, and then the cross-attention module is used to capture the correlation between different modalities. It can effectively capture the local fine-grained semantic associations between images and texts, and the feature fusion module optimizes the fusion of image region and text sequence features, so as to better represent the deep semantic information of images and texts. The pooling strategy not only enhances the model's ability to capture cross-modal information, but also reduces the computational cost, making the model more efficient.

[0054] In one embodiment, the process of training the preset cross-modal image-text retrieval model is as follows: Figure 6 As shown, step 202 includes:

[0055] Step 2021, input the text retrieval image sample into the image-text feature extraction module to obtain the image-text features.

[0056] Exemplarily, the text retrieval image sample includes an image modality and a text modality.

[0057] Step 2021 includes:

[0058] The image modality is input into the image-text feature extraction module, and an image feature set is obtained through linear mapping. The text modality is input into the image-text feature extraction module, and a text feature set is obtained through linear mapping. The image feature set and the text feature set have the same preset dimension. The image-text features include the image feature set and the text feature set.

[0059] Specifically, for image representation: for each input image, use the Faster R-CNN network pre-trained on Visual Genomes data to extract features to obtain the visual representation of the first k regions of interest (ROI). Use I0 to represent the image modality. After feature extraction, the image region set v is obtained, expressed as v = (v1, v2, ..., v m ), m is the number of ROIs. Each visual area v m There are 2048 dimensions, namely v m ∈R 2048 In batch training, the feature dimensions of each region of the image remain unchanged and do not need to be adjusted.

[0060] For text representation: The pre-trained BERT model is used to extract features for the text part. Specifically, the bag-of-words method (Bow) can be used to extract features. T0 represents the text modality. After word segmentation, a sentence consisting of n words is input into the BERT network to obtain e=(e1, e2,…, e n ), where each region has 768 fixed dimensions, i.e., e n ∈R 768 In the batch training process, text is usually input in batch size units, and the length of different texts in each batch is different, so the maximum length of text in each batch is set to n max , when the number of words in a text is n <n max When padding is used to ensure the consistency of the number of words, the input text mode is converted into a set of sequences containing text and padding bits.

[0061] For example, the cross-attention mechanism of the subsequent cross-attention module requires that the image and text features have the same dimension. Therefore, it is necessary to perform linear mapping on the image region set and the text region set. Set the common dimension to d, and establish a fully connected mapping layer on the image side and the text side respectively. Let v and e be the feature inputs of a text retrieval image sample to the image and text respectively, and the calculation formula of the linear mapping is as follows:

[0062] I=W img v+b img (1)

[0063] T=W txt e+b txt(2)

[0064] Where I represents the image feature set of m regions extracted; T represents the text feature set of n regions extracted, i.e., I = (i1, i2, ..., im), T = (t1, t2, ..., tn). img , b img are the image weight and image bias of the model, W txt , b txt are the text weight and text bias of the model respectively; the dimension of each i and each t is d.

[0065] Step 2022, input the image and text features into the cross attention module to obtain the image and text area features.

[0066] Exemplarily, step 2022 includes:

[0067] The image feature set and the text feature set are input into the cross attention module to calculate the regional similarity between each text region in the text feature set and each image region in the image feature set.

[0068] Based on the regional similarity, the text regions corresponding to each image region are aggregated to obtain the text region features corresponding to each image region. This is equivalent to inputting the image feature set and the text feature set into the cross attention module, calculating the regional similarity between the text region in the text feature set and each image region in the image feature set, and calculating the regional similarity between the image region in the image feature set and each text region in the text feature set.

[0069] Based on the region similarity, the image regions corresponding to each text region are aggregated to obtain the image region features corresponding to each text region; the image and text region features include image region features and text region features.

[0070] Specifically, given m image feature sets and n text feature sets, each feature set is a d-dimensional vector. In order for the neural network to learn the optimal parameters, in the forward reasoning process, the similarity is calculated to guide the back propagation training of the neural network to achieve the optimal performance. The similarity score consists of three parts: S1, S2 and S3, where S1 refers to the similarity between the pooled image features and the aggregated text features generated by the guidance, S2 refers to the similarity between the pooled text features and the aggregated image features generated by the guidance, and S3 refers to the similarity between the aggregated image and the aggregated text.

[0071] First, for m image regions, calculate the similarity score of n text regions to each region i in m, and then aggregate region t m times according to the similarity to obtain m text region features corresponding to the image region. The m text region features T′ are expressed as:

[0072] T′=attn(i a ,T), a=1,2,…,m (3)

[0073] Due to the difference in the length of image and text sequences, bidirectional attention is usually considered to obtain better retrieval results. Therefore, for n text regions, the same operation is performed, and region i is aggregated n times according to similarity to obtain n image region features corresponding to the text region. The n image region features I′ are expressed as:

[0074] I′=attn(t b ,I), b=1,2,…,n (4)

[0075] Among them, attn(·) represents the computational attention mechanism, i a or b represents the features of each image region or text region after linear mapping, T and I represent the corresponding text feature set and image feature set respectively, T′ and I′ represent the text region features or image region features obtained by cross attention aggregation respectively, that is, T′=(t1′,t2′,…,t m '), I'=(i1',i2',…,i n ').

[0076] In models such as SCAN, for a specific direction, the similarity of the region pairs is directly calculated and aggregated using methods such as LogSumExp to obtain a similarity matrix. In the training process of this model, the data of a certain modality is first aggregated and reduced in dimension, and then the similarity is calculated with another modality. This process of aggregation and dimensionality reduction is feature pooling.

[0077] Step 2023, input the image and text area features into the feature pooling module to obtain the image local features and the text global features.

[0078] This module designs two pooling modules for image features I and text features T, aiming to perform efficient feature aggregation for different data modalities. The pooling operation originally originated from convolutional neural networks, also known as downsampling. It reduces the dimension of data by imitating the human visual system, which can reduce information redundancy and effectively reduce network parameters while preventing overfitting.

[0079] In addition to LogSumExp pooling aggregation, a variety of pooling strategies can also achieve good regional dimensionality reduction effects, such as average pooling, maximum pooling, random pooling, etc. These operations are essentially the process of aggregating features in the local area dimension of the image or text, so that the n feature areas of the text or the m feature areas of the image are summarized into a unified feature representation. After pooling, the image features still retain the regional characteristics, while the text features are based on the overall paragraph representation extracted by the BERT model. The pooled text features can also be regarded as the global features of the entire sentence.

[0080] Exemplarily, step 2023 includes:

[0081] Firstly, the image region features and text region features are input into the feature pooling module, and the image region features and text region features are sorted along the region dimension to obtain image region sequence features and text region sequence features respectively.

[0082] Directly performing weighted aggregation processing on regional data will ignore some key information of the original local region, thereby reducing the learning effect. In order to solve this problem, this model sorts the input data (i or t) along the regional dimension before pooling to obtain image region sequence features and text region sequence features. For example, for an image feature matrix with a dimension of d = 2048 for k = 36 regions, 2048 arrays of length 36 can be obtained. This process is the process of sorting each array by absolute value, so that the feature area with a larger absolute value is located at the front end of the array as an obvious feature, and the area with a smaller absolute value and approaching zero is located at the back end as an unobvious feature. Combined with the pooling parameters learned by the neural network for each region, it is possible to retain important information to the maximum extent while filtering out unimportant information.

[0083] Then, the image region sequence features are weighted aggregated to obtain the image local features.

[0084] Exemplarily, weighted aggregation processing is performed on image region sequence features to obtain image local features, including: forming the image region sequence features into a feature matrix, inputting the feature matrix into a neural network, and obtaining dynamic weight parameters of the image region sequence features; and based on the dynamic weight parameters of the image region sequence features, weighted aggregation processing is performed on the image region sequence features to obtain image local features.

[0085] Similarly, the text region sequence features are weighted and aggregated to obtain the text global features.

[0086] Specifically, the pooling process can be specified in the form of parameters during model training. Different rules (such as average, maximum, random, etc.) often have different effects. Compared with specifying the aggregation method, letting the model learn a pooling strategy based on the characteristics of the data itself to achieve self-adjustment of the aggregation degree of each region, that is, using a set of learnable parameters to weight the importance of each region will achieve better retrieval results. Based on this, the present invention designs a set of weight parameters: α = (α1, α2, ..., α N ), represents the weight of the image and text area.

[0087]

[0088] Among them, j represents the jth image or text area, N represents the number of image or text areas, and normalization can be completed through the Softmax operation, which sums the weighted original areas to obtain the final pooled area:

[0089]

[0090] Where x is the image region i or text region t, X P is the final region, i.e. image I P or text T P , j represents the current j-th region, corresponding to the image, N=m or, corresponding to the text, N=n.

[0091] Sharing the pooling parameters of text and images will lead to a new problem: since the number of text and image regions is usually inconsistent, it is difficult to determine a number of regions that can take both into account at the same time. In each batch of training and testing, the number of image regions is a fixed value pre-extracted by Faster R-CNN, while the number of text regions varies with the length of the longest text in each batch. To address this problem, the present invention proposes a strategy for dynamically updating parameters:

[0092]

[0093] Where, s = 1, 2, ..., N, α s represents a series of weight parameters, f(·) represents the neural network model, W is the parameter set of the network, and variables p and q are dynamic input values. They are dynamically constructed according to the length of the image m and text n in the current batch. As the length of the text and image changes, the sum of p and q will also be dynamically adjusted. These dynamic values ​​can form a feature matrix as the input of the neural network. The process of generating a dynamically changing α is the process of keeping N unchanged and changing the value of p or q, such as Figure 7 shown.

[0094] For a certain image sequence or text sequence, the input of the module is only related to the length of the sequence. In order to ensure that the sum of the obtained weights α is 1, Softmax normalization is applied to the last layer of the neural network. When the input sequence length is N, the dimension of the feature matrix is ​​(N,2), which is then passed to a linear layer with a dimension of (2,1), and then passed through the normalization function to generate a weight vector α of length N, in which the sum of all elements is 1. This design can flexibly handle the situation where the length of the input sequence changes, so that the network model does not need to predetermine a fixed sequence length. In order to fully select appropriate information for different modalities, the weights of the linear layers at both ends are independent of each other. The model can adaptively adjust the weights according to the input data, aiming to generalize various pooling strategies, thereby ensuring that the algorithm can find the most suitable pooling strategy for different inputs. After this layer, the local feature I of the image is obtained. P With the global feature T P , used for the next step of feature fusion.

[0095] Step 2024, input the image and text area features into the feature fusion module to obtain the image and text aggregation features.

[0096] Exemplarily, the cross-attention calculation in step 2022 obtains the image region features and text region features after the attention operation. Considering that the text fully contains the context information of the sentence after the BERT feature extraction, the pooled text features in step 2023 can be considered as global features containing the overall information of the text.

[0097] Step 2024 includes:

[0098] The image region features are input into the feature fusion module, and the image region features are subjected to attention fusion operation with the text features of the single region to obtain the image aggregation features; the text region features are input into the feature fusion module, and the text region features are subjected to attention fusion operation with the image features of the single region to obtain the text aggregation features; the image-text aggregation features include image aggregation features and text aggregation features.

[0099] Specifically, the feature information obtained by pooling is used as a reference to compare the text features of the m regions in the text region feature T′ with the image features of the single region I P Perform attention fusion operation to obtain text aggregation features. The text aggregation feature T" is expressed as:

[0100] T″=attn(t a ',I P ), a=1,2,…,m (8)

[0101] Similarly, the image features of n regions in the image region feature I′ are combined with the text features of a single region T PPerform attention fusion operation to obtain the aggregated image features, that is, the image aggregate features. The image aggregate feature I" is expressed as:

[0102] I″=attn(i b ',T P ), b=1,2,…,n (9)

[0103] Step 2025, based on the image-text aggregation feature, the image local feature and the text global feature, obtain the first feature similarity, the second feature similarity and the third feature similarity.

[0104] Exemplarily, the overall image-text similarity is set to be represented by S. The present invention uses three different similarities: the first feature similarity is the similarity between the text aggregation feature and the image local feature, S1 = s(T″, I P ); The second feature similarity is the similarity between the image aggregation feature and the text global feature, S2 = s(I″, T P ); The third feature similarity is the similarity between the local features of the image and the global features of the text, S3 = s(I P ,T P ). Where s(·) represents the cosine similarity. In order to ensure that the retrieval model fully learns all potential similarity associations, a direct calculation branch S3 for the feature similarity after bimodal pooling is set.

[0105] Step 2026, based on the first feature similarity, the second feature similarity and the third feature similarity, obtain the loss function of the preset cross-modal image-text retrieval model.

[0106] S is the overall similarity of an input image and text pair, which can be obtained by S=S1+S2+S3, and the loss function is calculated accordingly.

[0107] During training, the number of image-text pairs in each batch is fixed, so a similarity matrix can be constructed, such as Figure 8 As shown, the visualization shows the similarity relationship between each batch of image and text pairs.

[0108] The elements on the diagonal of the matrix represent matching pairs with the highest similarity; the elements on the off-diagonal represent non-matching pairs. When training each batch, the similarity between all matching pairs (positive samples) is maximized, while the similarity between non-matching pairs (negative samples) is minimized. We use a hinge-based bidirectional triplet loss function and a difficult negative sample mining method to improve computational efficiency. The cross-modal alignment loss function, that is, the loss function of the preset cross-modal image-text retrieval model, is hard (I,T) is expressed as:

[0109]

[0110] in, is a small batch text retrieval image sample, B is the size of a small batch text retrieval image sample, I i is the i-th image modality, T i is the i-th text mode; [·] + is the ReLU function, α is the boundary value; S(I i ,T j ) is the overall similarity between the i-th image modality and the j-th text modality; Represents the current mini-batch text retrieval image sample and the i-th image modality I i The negative sample with the highest similarity, Represents the number of images in the current mini-batch text retrieval image sample and the i-th text modality T i The negative sample with the highest similarity; S() is the similarity.

[0111] Step 2027, using the loss function, maximize the similarity between all positive samples and minimize the similarity between negative samples to obtain a trained cross-modal image-text retrieval model.

[0112] Exemplarily, the constraint st of the loss function is used to maximize the similarity between all positive samples and minimize the similarity between negative samples.

[0113] The core of the cross-modal image-text retrieval model of this embodiment is a combination of a cross-attention mechanism and a feature aggregation strategy. The cross-attention mechanism can effectively capture the local fine-grained semantic associations between images and texts, while the feature aggregation strategy optimizes the fusion of image region and text sequence features through learnable weight parameters, thereby better representing the deep semantic information of images and texts. In order to further improve the retrieval performance of the model, a learnable pooling module is designed, which can process the features of variable-length sequences, learn weight parameters through neural networks, and effectively aggregate the features of images and texts. This pooling strategy not only enhances the model's ability to capture cross-modal information, but also reduces the computational cost, making the model more efficient.

[0114] In one embodiment, in order to verify the effectiveness of the cross-modal image-text retrieval method, a model experimental analysis is carried out, and the specific implementation method of the present invention is further analyzed through the image-text retrieval experimental results.

[0115] The Flickr 30k dataset and the MS COCO dataset are used. The Flickr 30k dataset contains 31,783 images, each with 5 manually annotated texts. The training set contains 28,000 images, the validation set contains 1,000 images, and the test set contains 1,000 images; the MS COCO dataset contains 123,287 images, each with 5 text annotations. The training set contains 113,287 images, the validation set contains 5,000 images, and the test set contains 5,000 images. The Recall at K (R@k) indicator is used on the two datasets to measure the performance of the image-text and text-image bidirectional retrieval tasks. The R@k indicator calculates the recall rate of the retrieved text (or the retrieved image) under a given image query in the top k results. The higher the R@k, the better the retrieval effect. This paper adopts three indicators: R@1, R@5 and R@10. In order to more comprehensively evaluate the effectiveness of the proposed method, the R@sum indicator is introduced, which is the sum of all evaluation indicators in the bidirectional retrieval task.

[0116] The Adam optimizer was used, with a batch size of 64 and an epoch of 20. The initial global learning rate was set to 0.0005, and the learning rate of the BERT model was set to 0.0002. A dynamic learning rate adjustment strategy was adopted, and all learning rates were decayed to 10% of the original value at the 10th and 15th epochs. The boundary value α in the loss function was set to 0.3.

[0117] In order to improve the training efficiency, the weight parameters of the feature extractor are not fine-tuned for the image part, and the feature vector of d = 2048 dimensions of the extracted K = 36 regional features is obtained; the pre-trained BERT basic model with 12 attention heads is used for text. BERT inputs masked tokenized text and outputs a fixed-length d = 768-dimensional feature vector. During the training process, BERT is fine-tuned to better adapt to the retrieval task. After each epoch, the recall rate is verified on the validation dataset, and the model weight information of the epoch with the best retrieval effect is saved for evaluation.

[0118] The proposed model can effectively make up for the insufficient interaction between features in each modality and within the modality, and achieve the best retrieval effect. Experiments were carried out on MS COCO and Flickr 30k datasets in the two directions of image retrieval and text retrieval to verify the proposed model.

[0119] Tables 1 and 2 respectively compare the R@k retrieval results of the proposed model and the current mainstream retrieval model on the Flickr 30k and MS COCO datasets.

[0120] Table 1 Performance results of R@k (k = 1, 5, 10) on the Flickr 30k dataset (%)

[0121]

[0122] Table 2 Performance results of R@k (k = 1, 5, 10) on the MS COCO dataset (5K Test Images) (%)

[0123]

[0124] As can be seen from Table 1 and Table 2, the results of the proposed method are better than those of the comparison method. The comprehensive performance of image-text and text-image bidirectional retrieval is the best, and the R@sum values ​​on Flickr 30k and MS COCO are 499.8% and 527.1% respectively; compared with the baseline model (SCAN), they increased by 34.8% and 19.2% respectively; compared with the HAS model, the R@sum value on the Flickr 30k dataset is 41.3% higher; compared with the CKE model, the R@sum value on the MS COCO dataset increased by 3.7%. Although the comparison method greatly improves the retrieval accuracy, the matching relationship between the two modalities is still not fully mined. HAS does not interact with cross-modal features, resulting in low performance; CKE's consensus knowledge embedding does not take into account the causal intervention between text modalities. In comparison, the method of the present invention uses image-text cross attention to effectively interact with input features, and the aggregation module can also obtain stronger image-text representation information, thus having a better retrieval effect.

[0125] Ablation experiments are conducted on the Flickr 30k dataset to comprehensively evaluate the impact of each module on the experimental results. The objective function consists of three parts: the similarity S1 between the attention of the text to the image and the pooled image information, the similarity S2 between the attention of the image to the text and the pooled text information, and the similarity S3 between the image and text information after pooling aggregation. Relevant experiments are conducted to exclude the pooling aggregation module, exclude the image attention module, exclude the text attention module, and exclude all attention modules, and compared with the proposed model (S1+S2+S3) to verify its effectiveness. The experimental results are shown in Table 3.

[0126] Table 3 Ablation experiment (Flickr 30k dataset, %)

[0127]

[0128] According to the results in Table 3, the use of attention mechanism for local feature fusion and the use of feature pooling module for dimensionality reduction and screening of local information have significantly improved the retrieval effect. The use of text attention + feature pooling (S1 + S3) alone and the use of image attention + feature pooling (S2 + S3) alone have roughly the same effect on the retrieval effect. Compared with the baseline model (S1 + S2), the R@sum index has increased by about 20%, and the R@sum value of the baseline model with only feature pooling (S3) has increased by about 13%. It can be seen that the use of attention and feature pooling can effectively improve the retrieval effect. This is mainly due to the fact that the model with attention can effectively fuse image (or text) information with the most representative text (or image) information. In addition, the matching of image and text information after the two feature pools also enhances the model's ability to learn cross-modal alignment information to a certain extent. The proposed model effectively combines the advantages of BERT and Faster R-CNN feature extraction, and makes full use of the characteristics of the attention mechanism, thereby effectively improving the accuracy of retrieval.

[0129] It should be noted that Figure 4 , Fig. 9 This example shows an example of text retrieval from an image using this model, listing the top 5 texts with the highest similarity to a given image. The similarity values ​​of the texts are given in brackets, where the correctly retrieved text is indicated by “√”, the wrong result is indicated by “×”, and the true label indicates the matching text. Figure 4 In the retrieval of , although the subject of the image is similar to the background color, the model successfully extracts key information, such as entities such as "dog", "purple object" and "Frisbee" through attention and self-learning pooling mechanism, showing good recognition ability. Although the baseline model can recognize the subject in the image, there is still room for improvement in processing details such as "white paws", "Frisbee" and "panting". Fig. 9 In the retrieval, the model proposed in this invention retrieves mostly the concept of "car repairman", while the comparative method does not extract the correspondence between people and objects very well, and obviously has limitations in processing fine-grained and abstract information, which proves the good effect of the proposed method.

[0130] The model of the present invention can also be applied to fields such as power transmission line image defect analysis. By building a large-scale power equipment image library, including visible light, infrared, deep vision and other aspects, covering various scenes of the equipment and combining the corresponding text description to fine-tune and optimize the model, a defect matching analysis model can be obtained. Compared with traditional defect analysis, this model combined with BERT can deeply understand text semantic information, including the logic of semantics. Through matching modeling with detailed prompt text and equipment image information in the database, the target image with the highest similarity is obtained. For example, the input text is "a photo describing a pin on a pole tower, connected to a hanging ring, which needs to bear weight and has no pin". The model can analyze the semantic restriction conditions and execute logic such as "identifying pins", "whether load-bearing" and "whether pins are missing" to find the corresponding image and reduce the probability of false alarms.

[0131] Fig.10 Results of text query images are shown, with each text having a unique image corresponding to it. The top 5 images with the highest similarity to the given text are listed. The similarity values ​​are also given in brackets, and the correctly matched images are marked with a check mark. For the keywords such as "young woman", "black shirt and jeans" and "sweeping the floor" in the retrieval, the proposed model can not only retrieve the correct results, but also rank them first, indicating that the introduction of feature pooling aggregation can better represent the semantic information of images and texts. When processing the same query, although the baseline model can identify "broom" and rod-shaped objects directly related to the "sweeping the floor" action, its retrieval results are relatively limited and fail to fully capture the rich semantic information in the query text. This comparison highlights the significant advantages of the proposed method in improving retrieval results and verifies the effectiveness of the learnable aggregation strategy in enhancing the ability of text-image semantic matching.

[0132] See also Fig.11 An embodiment of the present application provides a cross-modal image and text retrieval device that executes the cross-modal image and text retrieval method of the above embodiment, including a sample acquisition module 301, a model training module 302 and a retrieval result output module 303.

[0133] The sample acquisition module 301 is used to acquire text retrieval image samples;

[0134] The model training module 302 is used to train a preset cross-modal image-text retrieval model based on text retrieval image samples to obtain a trained cross-modal image-text retrieval model.

[0135] Among them, the preset cross-modal image-text retrieval model adopts a parallel processing structure of image and text dual-stream networks; the preset cross-modal image-text retrieval model includes an image-text feature extraction module, a cross-attention module, a feature pooling module and a feature fusion module; the cross-attention module is used to capture the correlation between different modalities of text retrieval image samples; the feature pooling module is used to perform feature dimensionality reduction along the regional dimension in a learnable way;

[0136] The retrieval result output module 303 is used to obtain the image and text to be retrieved, and input the image and text to be retrieved into the trained cross-modal image and text retrieval model to obtain the retrieval result.

[0137] Exemplarily, the model training module 302 is specifically used for:

[0138] Input the text retrieval image sample into the image-text feature extraction module to obtain the image-text features;

[0139] Input the image and text features into the cross attention module to obtain the image and text region features;

[0140] Input the image and text area features into the feature pooling module to obtain the local features of the image and the global features of the text;

[0141] Input the image and text area features into the feature fusion module to obtain the image and text aggregation features;

[0142] Based on the image-text aggregation feature, the image local feature and the text global feature, the first feature similarity, the second feature similarity and the third feature similarity are obtained;

[0143] Based on the first feature similarity, the second feature similarity and the third feature similarity, a loss function of a preset cross-modal image-text retrieval model is obtained;

[0144] The loss function is used to maximize the similarity between all positive samples and minimize the similarity between negative samples to obtain a trained cross-modal image-text retrieval model.

[0145] Exemplarily, in the model training module 302, the text retrieval image sample is input into the image-text feature extraction module to obtain the image-text features, including:

[0146] The image modality is input into the image-text feature extraction module, and the image feature set is obtained through linear mapping;

[0147] The text modality is input into the image-text feature extraction module, and a text feature set is obtained through linear mapping; the image feature set and the text feature set have the same preset dimension; the image-text feature includes the image feature set and the text feature set

[0148] Exemplarily, in the model training module 302, the image-text features include an image feature set and a text feature set;

[0149] Input the image and text features into the cross attention module to obtain the image and text region features, including:

[0150] The image feature set and the text feature set are input into the cross attention module, and the regional similarity between each text region in the text feature set and each image region in the image feature set is calculated;

[0151] Based on the region similarity, the text regions corresponding to each image region are aggregated to obtain the text region features corresponding to each image region;

[0152] Based on the region similarity, the image regions corresponding to each text region are aggregated to obtain the image region features corresponding to each text region; the image and text region features include image region features and text region features.

[0153] Exemplarily, in the model training module 302, the image-text region features include image region features and text region features;

[0154] Input the image and text area features into the feature pooling module to obtain the local features of the image and the global features of the text, including:

[0155] The image region features and the text region features are input into the feature pooling module, and the image region features and the text region features are sorted along the region dimension to obtain the image region sequence features and the text region sequence features respectively;

[0156] Perform weighted aggregation processing on the image region sequence features to obtain the local features of the image;

[0157] The text region sequence features are weighted and aggregated to obtain the global features of the text.

[0158] Exemplarily, in the model training module 302, weighted aggregation processing is performed on the image region sequence features to obtain the image local features, including:

[0159] The image region sequence features are combined into a feature matrix and input into a neural network to obtain dynamic weight parameters of the image region sequence features;

[0160] Based on the dynamic weight parameters of the image region sequence features, the image region sequence features are weighted aggregated to obtain the image local features.

[0161] Exemplarily, in the model training module 302, the image-text region features include image region features and text region features;

[0162] The image and text region features are input into the feature fusion module to obtain the image and text aggregation features, including:

[0163] The image region features are input into the feature fusion module, and the image region features are fused with the text features of the single region to obtain the image aggregation features;

[0164] The text region features are input into the feature fusion module, and the text region features are fused with the image features of the single region to obtain the text aggregation features; the image-text aggregation features include image aggregation features and text aggregation features.

[0165] Exemplarily, in the model training module 302, the image-text aggregation feature includes an image aggregation feature and a text aggregation feature;

[0166] The first feature similarity is the similarity between the text aggregate feature and the image local feature; the second feature similarity is the similarity between the image aggregate feature and the text global feature; the third feature similarity is the similarity between the image local feature and the text global feature.

[0167] Exemplarily, in the model training module 302, the loss function l of the cross-modal image-text retrieval model is preset hard (I,T) is expressed as:

[0168]

[0169] in, is a small batch text retrieval image sample, B is the size of a small batch text retrieval image sample, I i is the i-th image modality, T i is the i-th text mode; [·] + is the ReLU function, α is the boundary value; S(I i ,T j ) is the overall similarity between the i-th image modality and the j-th text modality; Represents the current mini-batch text retrieval image sample and the i-th image modality I i The negative sample with the highest similarity, Represents the number of images in the current mini-batch text retrieval image sample and the i-th text modality T i The negative sample with the highest similarity; S() is the similarity.

[0170] It should be noted that although several units / modules or sub-units / modules of the cross-modal image-text retrieval device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units / modules described above can be concretized in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided into multiple units / modules for concretization.

[0171] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0172] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the cross-modal image and text retrieval method provided in the above embodiment of the present application is implemented.

[0173] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0174] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A cross-modal image-text retrieval method, characterized in that: include: Get text retrieval image samples; Based on the text retrieval image sample, the preset cross-modal image-text retrieval model is trained to obtain a trained cross-modal image-text retrieval model; the preset cross-modal image-text retrieval model adopts an image and text dual-stream network parallel processing structure; the preset cross-modal image-text retrieval model includes an image-text feature extraction module, a cross-attention module, a feature pooling module and a feature fusion module; the cross-attention module is used to capture the correlation between different modes of the text retrieval image sample; the feature pooling module is used to perform feature dimensionality reduction along the regional dimension in a learnable manner; Obtain the image and text to be retrieved, and input the image and text to be retrieved into the trained cross-modal image and text retrieval model to obtain a retrieval result.

2. The cross-modal image-text retrieval method according to claim 1, characterized in that: The method of training a preset cross-modal image-text retrieval model based on the text retrieval image sample to obtain a trained cross-modal image-text retrieval model includes: Inputting the text retrieval image sample into the image-text feature extraction module to obtain image-text features; Inputting the image-text features into the cross attention module to obtain image-text region features; Inputting the image and text area features into the feature pooling module to obtain image local features and text global features; Inputting the image-text region features into the feature fusion module to obtain image-text aggregation features; Based on the image-text aggregation feature, the image local feature and the text global feature, a first feature similarity, a second feature similarity and a third feature similarity are obtained; Based on the first feature similarity, the second feature similarity and the third feature similarity, obtaining a loss function of the preset cross-modal image-text retrieval model; By using the loss function, the similarity between all positive samples is maximized and the similarity between negative samples is minimized to obtain a trained cross-modal image-text retrieval model.

3. The cross-modal image-text retrieval method according to claim 2, characterized in that: The text retrieval image sample includes image modality and text modality; The step of inputting the text retrieval image sample into the image-text feature extraction module to obtain image-text features includes: Inputting the image modality into the image-text feature extraction module, and obtaining an image feature set through linear mapping; The text modality is input into the image-text feature extraction module, and a text feature set is obtained through linear mapping; the image feature set and the text feature set have the same preset dimension; the image-text feature includes the image feature set and the text feature set.

4. The cross-modal image-text retrieval method according to claim 2, characterized in that: The image and text features include an image feature set and a text feature set; The step of inputting the image-text features into the cross attention module to obtain image-text region features includes: Inputting the image feature set and the text feature set into the cross attention module, and calculating the region similarity between each text region in the text feature set and each image region in the image feature set; Based on the region similarity, the text regions corresponding to each image region are aggregated to obtain text region features corresponding to each image region; Based on the region similarity, the image regions corresponding to each text region are aggregated to obtain image region features corresponding to each text region; the image-text region features include the image region features and the text region features.

5. The cross-modal image-text retrieval method according to claim 2, characterized in that: The image-text area features include image area features and text area features; The step of inputting the image and text area features into the feature pooling module to obtain image local features and text global features includes: Inputting the image region features and the text region features into the feature pooling module, sorting the image region features and the text region features along the region dimension, respectively, to obtain image region sequence features and text region sequence features, respectively; Performing weighted aggregation processing on the image region sequence features to obtain the image local features; The text region sequence features are weighted and aggregated to obtain the text global features.

6. The cross-modal image-text retrieval method according to claim 5, characterized in that: The step of performing weighted aggregation processing on the image region sequence features to obtain the image local features includes: Combining the image region sequence features into a feature matrix, inputting the matrix into a neural network, and obtaining dynamic weight parameters of the image region sequence features; Based on the dynamic weight parameters of the image region sequence features, weighted aggregation processing is performed on the image region sequence features to obtain the image local features.

7. The cross-modal image-text retrieval method according to claim 2, characterized in that: The image-text area features include image area features and text area features; The step of inputting the image-text region feature into the feature fusion module to obtain the image-text aggregation feature includes: Inputting the image region features into the feature fusion module, performing attention fusion operation on the image region features and the text features of the single region to obtain image aggregation features; The text region feature is input into the feature fusion module, and the text region feature is subjected to attention fusion operation with the image feature of a single region to obtain a text aggregation feature; the image-text aggregation feature includes the image aggregation feature and the text aggregation feature.

8. The cross-modal image-text retrieval method according to claim 2, characterized in that: The image-text aggregation feature includes an image aggregation feature and a text aggregation feature; The first feature similarity is the similarity between the text aggregate feature and the image local feature; the second feature similarity is the similarity between the image aggregate feature and the text global feature; the third feature similarity is the similarity between the image local feature and the text global feature.

9. The cross-modal image-text retrieval method according to claim 1, characterized in that: The loss function l of the preset cross-modal image-text retrieval model hard (I,T) is expressed as: in, is a small batch text retrieval image sample, B is the size of a small batch text retrieval image sample, I i is the i-th image modality, T i is the i-th text mode; [·] + is the ReLU function, α is the boundary value; S(I i ,T j ) is the overall similarity between the i-th image modality and the j-th text modality; Represents the current mini-batch text retrieval image sample and the i-th image modality I i The negative sample with the highest similarity, Represents the number of images in the current mini-batch text retrieval image sample and the i-th text modality T i The negative sample with the highest similarity; S() is the similarity.

10. A cross-modal image and text retrieval device, characterized in that: Applying the cross-modal image-text retrieval method according to any one of claims 1 to 9 comprises: A sample acquisition module is used to acquire text retrieval image samples; A model training module is used to train a preset cross-modal image-text retrieval model based on the text retrieval image sample to obtain a trained cross-modal image-text retrieval model; the preset cross-modal image-text retrieval model adopts an image and text dual-stream network parallel processing structure; the preset cross-modal image-text retrieval model includes an image-text feature extraction module, a cross-attention module, a feature pooling module and a feature fusion module; the cross-attention module is used to capture the correlation between different modes of the text retrieval image sample; the feature pooling module is used to perform feature dimensionality reduction along the regional dimension in a learnable manner; The retrieval result output module is used to obtain the image and text to be retrieved, and input the image and text to be retrieved into the trained cross-modal image and text retrieval model to obtain the retrieval result.

Images