Image retrieval model training method and device, image retrieval method and equipment

Abstract

Embodiments of the present application provide a kind of image retrieval model training method, device and equipment and image retrieval method, involve image processing technical field.The present application includes: the sample image is divided to obtain multiple sub-sample images, and the sample text is divided to obtain multiple sub-sample texts;Determine the sub-sample image feature corresponding to sub-sample image and the sub-sample text feature corresponding to sub-sample text;In all sub-sample text features, determine the positive sample and negative sample corresponding to each sub-sample image feature, determine the sub-image semantic contrast loss corresponding to sub-sample image feature;In all sub-sample image features, determine the positive sample and negative sample corresponding to each sub-sample text feature, determine the sub-text semantic contrast loss corresponding to sub-sample text feature;Determine sub-semantic contrast loss based on sub-image semantic contrast loss and sub-text semantic contrast loss;And update image retrieval model based on sub-semantic contrast loss, until image retrieval model meets convergence condition and obtains target image retrieval model.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a training method, apparatus, device, and image retrieval method for an image retrieval model. Background Technology

[0002] The internet generates massive amounts of data anytime, anywhere, including multimedia information such as text, images, videos, and audio. This rich and diverse content makes extracting the truly needed information from this vast amount of data a pressing issue. When users search, they need to describe the features of the image they require using text. By associating and matching these two different modalities of information—image and text—and aligning their features, the corresponding image is obtained.

[0003] If feature alignment is achieved by training the model by labeling the target image with positive and negative samples, this method can only achieve global semantic alignment that is either "positive" or "negative". When there are multiple semantics in the text or image, there may be a phenomenon where some semantics in the image and the negative sample text do not match, but other parts of the semantics match. Using negative samples with local semantic matching to optimize the model will cause the model to learn incorrect information, making the model's ability to mine fine-grained local semantics weak. It cannot mine the local semantics in the text, which may cause the model to mistakenly identify positive samples as negative samples or vice versa when applied, making it impossible for the model to accurately associate and match images and text. Summary of the Invention

[0004] This application provides a training method, apparatus, and device for an image retrieval model. When training the image retrieval model, sample images and sample text are divided into sub-sample images and sub-sample text, respectively. The loss corresponding to each sub-sample image and the loss corresponding to each sub-sample text are determined. Finally, the image retrieval model is updated based on the losses corresponding to all sub-sample images and sub-sample text until the model converges. This enables the trained image retrieval model to better recognize multiple semantics in images and text, thereby accurately associating and matching images and text.

[0005] The first aspect of this application provides a training method for an image retrieval model, comprising: acquiring multiple sample pairs, wherein each sample pair includes a sample image and a sample text; for each sample pair, dividing the sample image into multiple sub-sample images, and dividing the sample text into multiple sub-sample texts; inputting the sub-sample images into an image encoder in an image retrieval model to obtain corresponding sub-sample image features, and inputting the sub-sample text into a text encoder in the image retrieval model to obtain corresponding sub-sample text features; for each sub-sample image feature, determining positive and negative samples corresponding to the sub-sample image feature from all the sub-sample text features based on the correlation between the sub-sample image feature and each sub-sample text feature; and determining a sub-image semantic contrast loss based on the sub-sample image feature and the positive and negative samples corresponding to the sub-sample image feature; wherein the correlation between the sub-sample image feature and the positive sample corresponding to the sub-sample image feature is higher than the correlation between the sub-sample image feature and the negative sample corresponding to the sub-sample image feature; the sub-image semantic contrast loss is used to represent... The image retrieval model is characterized by its ability to distinguish between positive and negative samples corresponding to the subsample image features. For each subsample text feature, based on the correlation between the subsample text feature and each subsample image feature, positive and negative samples corresponding to the subsample text feature are determined from all subsample image features. A subtext semantic contrast loss is determined based on the subsample text feature and the corresponding positive and negative samples. The correlation between the subsample text feature and the corresponding positive sample is higher than the correlation between the subsample text feature and the corresponding negative sample. The subtext semantic contrast loss characterizes the image retrieval model's ability to distinguish between positive and negative samples corresponding to the subsample text feature. Based on the subimage semantic contrast loss corresponding to all subsample text features and the subtext semantic contrast loss corresponding to all subsample image features, a sub-semantic contrast loss corresponding to the sample pair is determined. The image retrieval model is updated based on the sub-semantic contrast losses corresponding to multiple sample pairs until the image retrieval model meets the convergence condition, thus obtaining the target image retrieval model.

[0006] The image retrieval model training method provided in this application first divides the sample image and sample text into sub-sample images and sub-sample text, respectively, and then determines the positive and negative samples corresponding to each sub-sample image and each sub-sample text. Thus, when training the image retrieval model, the sub-image semantic contrast loss corresponding to each sub-sample image and the sub-text semantic contrast loss corresponding to each sub-sample text are determined. Then, based on all sub-image semantic contrast losses and all sub-text semantic contrast losses, the sub-semantic contrast loss corresponding to each sample pair is determined. The image retrieval model is updated based on the sub-semantic contrast losses corresponding to multiple sample pairs until the image retrieval model meets the convergence condition, thus obtaining the target image retrieval model.

[0007] Understandably, sub-image semantic contrast loss can characterize the ability of an image retrieval model to identify positive and negative samples corresponding to the features of a sub-image, and sub-text semantic contrast loss can characterize the ability of an image retrieval model to identify positive and negative samples corresponding to the features of a sub-text. Therefore, by using sub-image semantic contrast loss and sub-text semantic contrast loss to update the image retrieval model, the updated image retrieval model can better acquire the positive and negative samples of each sub-image in the image and the positive and negative samples of each sub-text in the text. This enables semantic alignment of text and image at a fine-grained level, allowing the trained target image retrieval model to accurately associate and match images and text.

[0008] Combining the first implementation of the first aspect, based on the correlation between subsample image features and each subsample text feature, positive and negative samples corresponding to subsample image features are determined from all subsample text features. This includes: determining the attention value vectors of subsample image features and each subsample text feature to obtain a set of text attention value vectors corresponding to the subsample image features; the value of each attention value vector in the text attention value vector set is positively correlated with the correlation between the subsample image features and the subsample text features corresponding to the attention value vectors; determining the positive and negative samples corresponding to the subsample image features based on the set of text attention value vectors corresponding to the subsample image features; and determining the positive and negative samples corresponding to the subsample text features from all subsample image features based on the correlation between subsample text features and each subsample image feature. This includes: determining the attention value vectors of subsample text features and each subsample image feature to obtain a set of text attention value vectors corresponding to the subsample text features; the value of each attention value vector in the text attention value vector set is positively correlated with the correlation between the subsample text features and the subsample image features corresponding to the attention value vectors; and determining the positive and negative samples corresponding to the subsample text features based on the set of text attention value vectors corresponding to the subsample text features.

[0009] Combining the second implementation method of the first aspect, based on the set of text attention value vectors corresponding to the subsample text features, the positive and negative samples corresponding to the subsample text features are determined, including: inputting the set of text attention value vectors corresponding to all subsample text features and the set of image attention value vectors corresponding to all subsample image features into the attention filtering unit; suppressing non-maximum values ​​in the set of text attention value vectors to obtain the filtered set of text attention value vectors corresponding to the subsample text features, and suppressing non-maximum values ​​in the set of image attention value vectors to obtain the filtered set of image attention value vectors corresponding to the subsample image features; and determining the positive and negative samples corresponding to the subsample text features based on the filtered set of text attention value vectors corresponding to the subsample text features and the filtered set of image attention value vectors corresponding to the subsample image features.

[0010] Combining the third implementation method of the first aspect, based on the filtered text attention value vector set corresponding to the subsample text features and the filtered image attention value vector set corresponding to the subsample image features, the positive and negative samples corresponding to the subsample text features are determined, including: determining the maximum attention value in the filtered text attention value vector set corresponding to each subsample text feature among all subsample text features, obtaining the text maximum attention value set; each text maximum attention value in the text maximum attention value set corresponds to a subsample text feature; determining the filtered image attention value vector set corresponding to each subsample image feature among all subsample image features. The maximum attention value in the image is used to obtain the image maximum attention value set; each image maximum attention value in the image maximum attention value set corresponds to a subsample image feature; for subsample text features, the text maximum attention value corresponding to the subsample text feature is multiplied by the maximum attention value of each image in the image maximum attention value set to obtain the target set corresponding to the subsample text feature; a first vector and a second vector are selected from the target set, where the extreme value of the first vector is greater than the extreme value of the second vector; the subsample image feature corresponding to the first vector is taken as the positive sample corresponding to the subsample text feature; the subsample image feature corresponding to the second vector is taken as the negative sample corresponding to the subsample text feature.

[0011] Combining the fourth implementation method of the first aspect, the attention filtering unit includes multiple multi-head attention modules; the set of text attention value vectors corresponding to all sub-sample text features and the set of image attention value vectors corresponding to all sub-sample image features are input into the attention filtering unit. The attention filtering unit suppresses non-maximum values ​​in the set of image attention value vectors to obtain the filtered set of text attention value vectors corresponding to the sub-sample text features and the filtered set of image attention value vectors corresponding to the sub-sample image features. This includes: inputting the set of text attention value vectors corresponding to all sub-sample text features and the set of image attention value vectors corresponding to all sub-sample image features into the first multi-head attention module of the attention filtering unit, and the input of other multi-head attention modules being the input of the previous multi-head attention module. The process involves each multi-head attention module performing the following steps: determining the maximum attention value in the set of text attention value vectors corresponding to each subsample text feature, thus obtaining a first maximum attention value set; determining the maximum attention value in the set of image attention value vectors corresponding to each subsample image feature, thus obtaining a second maximum attention value set; determining a filtering threshold based on the first and second maximum attention value sets; setting attention value vectors less than or equal to the filtering threshold to 0 in all sets of text attention value vectors corresponding to all subsample text features and all sets of image attention value vectors corresponding to all subsample image features; and outputting the updated set of text attention value vectors corresponding to all subsample text features and the updated set of image attention value vectors corresponding to all subsample image features.

[0012] Combining the fifth implementation method of the first aspect, the filtering threshold is determined based on the first maximum attention value set and the second maximum attention value set, including: inputting the first maximum attention value set and the second maximum attention value set into a weighted neural network, determining the correlation strength between the first maximum attention value set and the second maximum attention value set through the weighted neural network, and obtaining attention filtering weights; multiplying the elements in the first maximum attention value set and the elements in the second maximum attention value set to obtain multiple element dot products; calculating the average of all element dot products to obtain the dot product mean; and determining the filtering threshold based on the dot product mean and the attention filtering weights.

[0013] Combining the sixth implementation method of the first aspect, the sample image is divided into multiple sub-sample images, including: inputting the sample text and preset prompt information into the target large language model, so that the target large language model splits the sample text into multiple sub-sample texts according to the preset prompt information; dividing the sample text into multiple sub-sample texts includes: determining the key points in the sample image; inputting the sample image and sample text into the image encoder and text encoder respectively to obtain the corresponding sample image features and sample text features; performing gradient updates based on the sample image features and sample text features to obtain the heatmap corresponding to the sample image; taking the local region with the strongest response in the heatmap as the image sub-region that matches the sample text; determining the key points and key point combinations within the image sub-region, and setting a cropping box based on the key points and key point combinations within the image sub-region to crop the sample image to obtain multiple sub-sample images.

[0014] In conjunction with the seventh implementation method of the first aspect, the method further includes: obtaining the synonyms, near-synonyms, and antonyms corresponding to the sample text, and inputting the synonyms, near-synonyms, and antonyms into the text encoder respectively to obtain the corresponding synonym features, near-synonyms features, and antonyms features; inputting the sample image and sample text into the image encoder and text encoder respectively to obtain the corresponding sample image features and sample text features; constructing multiple triples based on the sample image features, sample text features, synonym features, near-synonyms features, and antonyms features; wherein the negative sample in each triple is antonyms feature; determining the loss function corresponding to each triple, and determining the multi-semantic contrast loss corresponding to the sample pair based on the loss function corresponding to all triples; updating the image retrieval model based on the sub-semantic contrast loss corresponding to multiple sample pairs, including: updating the image retrieval model based on the sub-semantic contrast loss and the multi-semantic contrast loss corresponding to multiple sample pairs.

[0015] In conjunction with the eighth implementation of the first aspect, the method further includes: obtaining the negative sample text corresponding to the sample image; inputting the sample image, sample text, and negative sample text into the image encoder and text encoder respectively to obtain the corresponding sample image features, sample text features, and negative sample text features; determining the contrastive learning loss based on the sample image features, sample text features, and negative sample text features; the contrastive learning loss is used to characterize the ability of the image retrieval model to distinguish between positive and negative samples corresponding to the sample image features; and updating the image retrieval model based on the corresponding sub-semantic contrastive loss of multiple sample pairs, including: updating the image retrieval model based on the corresponding sub-semantic contrastive loss of multiple sample pairs and the contrastive learning loss.

[0016] The second aspect of this application provides an image retrieval method, comprising: acquiring text to be retrieved; inputting the text to be retrieved into a target image retrieval model trained as described in the first aspect and its possible implementations; and retrieving a target image matching the text to be retrieved from an image database using the target image retrieval model.

[0017] A third aspect of this application provides a training apparatus for an image retrieval model, comprising: an acquisition module for acquiring multiple sample pairs, wherein each sample pair includes a sample image and a sample text; a segmentation module for segmenting each sample pair into multiple sub-sample images and multiple sub-sample texts; an input module for inputting the sub-sample images into an image encoder in the image retrieval model to obtain corresponding sub-sample image features, and inputting the sub-sample text into a text encoder in the image retrieval model to obtain corresponding sub-sample text features; and a determination module for determining, for each sub-sample image feature, based on the correlation between the sub-sample image feature and each sub-sample text feature, positive and negative samples corresponding to the sub-sample image feature from all sub-sample text features; and determining a sub-image semantic contrast loss based on the sub-sample image feature and the positive and negative samples corresponding to the sub-sample image feature; wherein the correlation between the sub-sample image feature and the positive sample corresponding to the sub-sample image feature is higher than the correlation between the sub-sample image feature and the negative sample corresponding to the sub-sample image feature; and sub-image semantic pairs The subtext semantic contrast loss is used to characterize the ability of the image retrieval model to identify positive and negative samples corresponding to the subsample image features. The determination module is also used to, for each subsample text feature, determine the positive and negative samples corresponding to the subsample text feature from all subsample image features based on the correlation between the subsample text feature and each subsample image feature; and determine the subtext semantic contrast loss based on the subsample text feature and the corresponding positive and negative samples. The correlation between a subsample text feature and its corresponding positive sample is higher than the correlation between the subsample text feature and its corresponding negative sample. The subtext semantic contrast loss is used to characterize the ability of the image retrieval model to identify positive and negative samples corresponding to the subsample text features. The determination module is also used to determine the sub-semantic contrast loss corresponding to each sample pair based on the sub-image semantic contrast loss corresponding to all subsample text features and the subtext semantic contrast loss corresponding to all subsample image features. The update module is used to update the image retrieval model based on the sub-semantic contrast losses corresponding to multiple sample pairs until the image retrieval model meets the convergence condition, thus obtaining the target image retrieval model.

[0018] The fourth aspect of this application provides a training method for an image retrieval model, comprising: acquiring multiple sample pairs, wherein each sample pair includes a sample image and a sample text; for each sample pair, acquiring synonyms, near-synonyms, and antonyms corresponding to the sample text, and inputting the synonyms, near-synonyms, and antonyms into a text encoder respectively to obtain corresponding synonym features, near-synonyms features, and antonyms features; inputting the sample image and sample text into an image encoder and a text encoder respectively to obtain corresponding sample image features and sample text features; constructing multiple triples based on the sample image features, sample text features, synonyms, near-synonyms, and antonyms features; wherein the negative sample in each triple is antonyms feature; determining the loss function corresponding to each triple, and determining the multi-semantic contrast loss corresponding to the sample pair based on the loss function corresponding to all triples; updating the image retrieval model based on the multi-semantic contrast loss corresponding to multiple sample pairs until the image retrieval model satisfies the convergence condition to obtain the target image retrieval model.

[0019] A fifth aspect of this application provides an electronic device, including: one or more processors; one or more memories; wherein the one or more memories are used to store computer program code, the computer program code including computer instructions, and when the one or more processors execute the computer instructions, the electronic device executes the training method of the image retrieval model provided in the first aspect and its possible implementations, or executes the image retrieval method provided in the second aspect, or executes the training method of the image retrieval model provided in the third aspect.

[0020] The sixth aspect of this application provides a computer-readable storage medium storing computer-executable instructions. When the computer executes the instructions, the computer performs the training method of the image retrieval model provided in the first aspect and its possible implementations, or performs the image retrieval method provided in the second aspect, or performs the training method of the image retrieval model provided in the third aspect.

[0021] A seventh aspect of this application provides a computer program product, including a computer program / instruction that, when executed by a processor, implements the steps of a training method for an image retrieval model provided in the first aspect and its possible implementations, or executes an image retrieval method provided in the second aspect, or executes a training method for an image retrieval model provided in the third aspect.

[0022] The beneficial effects described in aspects two through seven can be referred to the analysis of the beneficial effects in aspect one, and will not be repeated here. Attached Figure Description

[0023] The accompanying drawings are provided to further understand the technical solutions of the present invention and constitute a part of the specification. They are used together with the embodiments of this application to explain the technical solutions of the present invention and do not constitute a limitation on the technical solutions of the present invention.

[0024] Figure 1 A flowchart illustrating an image retrieval process provided in an embodiment of this application;

[0025] Figure 2 A schematic diagram of a positive and negative sample provided in an embodiment of this application;

[0026] Figure 3 A flowchart illustrating a training method for an image retrieval model provided in this application embodiment. Figure 1 ;

[0027] Figure 4 A method flow for training an image retrieval model provided in this application embodiment Figure 1 ;

[0028] Figure 5 A schematic diagram illustrating the determination of positive and negative samples provided in an embodiment of this application;

[0029] Figure 6 A method flow for training an image retrieval model provided in this application embodiment Figure 2 ;

[0030] Figure 7 A schematic diagram illustrating a fine-tuning of a large language model provided in an embodiment of this application;

[0031] Figure 8 This is a schematic diagram of a process for segmenting sample images provided in an embodiment of this application;

[0032] Figure 9 A semantic distance diagram between positive and negative samples provided in an embodiment of this application;

[0033] Figure 10 A schematic diagram illustrating a process for determining multi-semantic contrast loss, provided for an embodiment of this application;

[0034] Figure 11 A flowchart illustrating a method for determining contrastive learning loss provided in an embodiment of this application;

[0035] Figure 12 A flowchart illustrating a training method for an image retrieval model provided in this application embodiment. Figure 2 ;

[0036] Figure 13 A flowchart of an image retrieval method provided in this application embodiment;

[0037] Figure 14 A schematic diagram of the structure of a training device for an image retrieval model provided in an embodiment of this application;

[0038] Figure 15 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0039] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0040] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0041] In the description of this application, it should be noted that, unless otherwise expressly specified and limited, the terms "connected" and "linked" should be interpreted broadly, for example, as a fixed connection, a detachable connection, or an integral connection. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances. Furthermore, when describing pipelines, the terms "connected" and "linked" as used in this application have the meaning of establishing electrical connection. The specific meaning needs to be understood in conjunction with the context.

[0042] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0043] The internet generates massive amounts of data anytime, anywhere, including multimedia information such as text, images, videos, and audio. This richness and diversity of information makes extracting the truly needed information from this vast amount of data a pressing issue.

[0044] Related technologies provide image retrieval methods. When searching, users can describe the feature information of the image they need through text and input it into a pre-trained image retrieval model. The image retrieval model aligns the features of the image and text by associating and matching information from these two different modalities, and obtains the image corresponding to the input text.

[0045] For ease of understanding, the image retrieval process is first described in the embodiments of this application. Please refer to [link / reference]. Figure 1 The image retrieval process mainly includes the following steps:

[0046] S101. Acquire images. Encode the acquired images using the image encoder of the image retrieval model to obtain the feature information corresponding to the images. Establish an image retrieval database based on the images and the corresponding feature information.

[0047] S102. When a user performs a search, they input the search text into the image retrieval model. The text encoder of the image retrieval model performs feature encoding on the search text to obtain the text feature Ft_A.

[0048] S103. The image retrieval model calculates the similarity between the text features and the image features in the image retrieval database, and selects the N images with the highest similarity to present to the user.

[0049] It is understandable that if the image retrieval model retrieves an image that meets the user's needs, then the search ends. If the image retrieval model retrieves an image that does not meet the user's needs, then the user may revise the search text A and re-enter the revised search text B into the image retrieval model.

[0050] Please continue reading. Figure 1 The image retrieval process may also include the following steps:

[0051] S104. The user determines whether the images recommended by the image retrieval model contain the target image to be retrieved.

[0052] If the result is found, the search is successful and the search ends; if the result is not found, proceed to step S205.

[0053] S105. The user corrects the search text based on the search results in S104, and inputs the corrected search text into the image retrieval model to obtain the text feature Ft_B.

[0054] S106. Repeat S103-S105 until the retrieval is successful, or until the number of retrievals exceeds the preset number of retrievals, then determine that the retrieval has failed.

[0055] If we train the model by labeling target images with positive and negative samples to achieve feature alignment, that is, to determine the positive and negative samples (text) corresponding to the target image, the model can correctly distinguish between the positive and negative samples corresponding to the target image. However, this method can only achieve global semantic alignment where it is either "positive" or "negative". When there are multiple semantics in the text or image, there may be instances where some semantics in the image and the negative sample text do not match, but other parts of the semantics match. Optimizing the model using negative samples with local semantic matching will cause the model to learn incorrect information, making the model's ability to mine fine-grained local semantics weak. It cannot mine fine-grained local semantics in the text, which may lead the model to incorrectly identify positive samples as negative samples or vice versa, making it impossible for the model to accurately associate and match images and text.

[0056] For example, please refer to Figure 2 , Figure 2 In this example, text A is labeled as a positive sample of image A, and text B is labeled as a negative sample of image A. However, text A and text B only differ in three semantics: gender, hair length, and accessories. All other semantics are the same. Therefore, it is inappropriate to "completely negate" text B as a negative sample of image A. This ignores the phenomenon that when text and image have multiple semantics, there may be a few semantic mismatches but other semantic matches, which limits the model's ability to mine fine-grained local semantics.

[0057] Based on this, embodiments of this application provide a training method for an image retrieval model. When training the image retrieval model, the sample images and sample text in the obtained sample pairs are divided into sub-sample images and sub-sample text, respectively, and input into the image encoder and text encoder of the image retrieval model to obtain corresponding sub-sample image features and sub-sample text features. Then, for each sub-sample image feature, based on the correlation between the sub-sample image feature and each sub-sample text feature, the positive and negative samples corresponding to the sub-sample image feature are determined from all the sub-sample text features. Furthermore, based on each sub-sample image feature and its corresponding positive and negative samples, the sub-image semantic contrast loss corresponding to that sub-sample image feature is determined. For each sub-sample text feature, based on the correlation between the sub-sample text feature and each sub-sample image feature, the positive and negative samples corresponding to the sub-sample text feature are determined from all the sub-sample image features. And based on each sub-sample text feature and its corresponding positive and negative samples, the sub-text semantic contrast loss is determined. In other words, the semantic contrast loss of each sub-sample image and the semantic contrast loss of each sub-sample text are determined. Then, based on the semantic contrast losses of all sub-images and all sub-texts, the semantic contrast loss of each sample pair is determined. The image retrieval model is updated based on the semantic contrast losses of multiple sample pairs until the image retrieval model meets the convergence condition, thus obtaining the target image retrieval model.

[0058] As can be seen, the training method of the image retrieval model provided in this application first divides the sample image and sample text into sub-sample images and sub-sample text, respectively, and then determines the positive and negative samples corresponding to each sub-sample image and each sub-sample text. Thus, when training the image retrieval model, the sub-image semantic contrast loss corresponding to each sub-sample image and the sub-text semantic contrast loss corresponding to each sub-sample text are determined. Then, based on all sub-image semantic contrast losses and all sub-text semantic contrast losses, the sub-semantic contrast loss corresponding to each sample pair is determined. The image retrieval model is updated based on the sub-semantic contrast losses corresponding to multiple sample pairs until the image retrieval model meets the convergence condition, thus obtaining the target image retrieval model.

[0059] Understandably, sub-image semantic contrast loss can characterize the ability of an image retrieval model to identify positive and negative samples corresponding to the features of a sub-image, and sub-text semantic contrast loss can characterize the ability of an image retrieval model to identify positive and negative samples corresponding to the features of a sub-text. Therefore, by using sub-image semantic contrast loss and sub-text semantic contrast loss to update the image retrieval model, the updated image retrieval model can better acquire the positive and negative samples of each sub-image in the image and the positive and negative samples of each sub-text in the text. This enables semantic alignment of text and image at a fine-grained level, allowing the trained target image retrieval model to accurately associate and match images and text.

[0060] It should be understood that the image retrieval model training method provided in this application is applicable to model training devices that deploy image retrieval models. The model training device can be a terminal device, such as a personal computer (PC), laptop computer, mobile device, tablet computer, etc. This application does not limit the specific form of the electronic device. Alternatively, the model training device can be a single server or a server cluster consisting of multiple servers. In some implementations, the server cluster can be a distributed cluster server. This application does not impose any restrictions on the specific form of the model training device.

[0061] Please see Figure 3 The image retrieval model training method provided in this application includes the following steps:

[0062] S301. Obtain multiple sample pairs.

[0063] Each sample pair includes a sample image and a sample text.

[0064] S302. For each sample pair, the sample image is divided into multiple sub-sample images, and the sample text is divided into multiple sub-sample texts.

[0065] Since each sample image may contain multiple parts, and each part contains a semantic meaning, the sample image is divided into multiple sub-sample images so that each sub-sample image contains only one or fewer semantic meanings, which facilitates subsequent semantic alignment.

[0066] Each sample text may contain multiple clauses, and each clause has a semantic meaning. Dividing the sample text into multiple sub-sample texts ensures that each sub-sample text contains only one or fewer semantic meanings, which facilitates subsequent semantic alignment.

[0067] S303. Input the subsample image into the image encoder in the image retrieval model to obtain the corresponding subsample image features, and input the subsample text into the text encoder in the image retrieval model to obtain the corresponding subsample text features.

[0068] Understandably, encoders typically use pooling layers to gradually reduce the size of the input data, and can be viewed as feature extraction networks capable of extracting feature information from the input content. The image retrieval model in this application includes an image encoder and a text encoder, used to extract features from images and text, respectively.

[0069] It should be understood that the image encoder in the embodiments of this application can be a Vision-Transformer model, a Swin-Transformer model, a ResNet model (such as ResNet18, ResNet34, etc.), a ResNeXt model, an SE-ResNet model, etc., and the embodiments of this application do not limit this.

[0070] As a feasible implementation method, the Visual Transformer model can be used as an image encoder in this application embodiment because it has flexible input, excellent performance, scalability, global and local feature capture capabilities, interpretability, ability to be combined with deep learning models, uniformity of feature representation, and compatibility with the field of Natural Language Processing (NLP).

[0071] As a feasible implementation, ResNet50 is an image classification algorithm based on deep convolutional neural networks (CNNs). Compared with traditional CNN models, ResNet50 has a deeper network structure and solves the gradient vanishing problem during deep network training by introducing residual connections, which can effectively improve the model's performance. Therefore, ResNet50 can be used as the image encoder in this embodiment. It should be understood that the text encoder in this embodiment can be a Bidirectional Encoder Representations from Transformers (BERT) Chinese model, an Enhanced Representation through Knowledge Integration (ERNIE) model, or a Robustly Optimized BERT approach (RoBERTa) model, etc., and this embodiment does not limit it. Because ResNet50 with relatively small parameters is selected as the image encoder and BERT as the Chinese text encoder, the hardware system matched by the method in this embodiment requires fewer hardware resources, runs faster, consumes less resources, is easier for users to deploy, and is conducive to high-frequency version iteration according to user needs.

[0072] As a feasible implementation method, the BERT Chinese model, employing a bidirectional Transformer encoder architecture, can simultaneously consider the context of the text, thereby more accurately understanding its meaning. Furthermore, the BERT Chinese model can process input sequences in parallel, resulting in faster training and inference speeds. Therefore, the BERT Chinese model can be used as the text encoder in this embodiment.

[0073] S304. For each subsample image feature, based on the correlation between the subsample image feature and each subsample text feature, determine the positive and negative samples corresponding to the subsample image feature from all subsample text features; and determine the subsample semantic contrast loss based on the subsample image feature and the positive and negative samples corresponding to the subsample image feature.

[0074] Among them, the correlation between positive samples of subsample image features and subsample image features is higher than the correlation between negative samples of subsample image features and subsample image features; subimage semantic contrast loss is used to characterize the ability of the image retrieval model to identify positive and negative samples corresponding to subsample image features.

[0075] It is understandable that for each subsample image feature, the correlation between the corresponding positive sample and itself is higher than the correlation between the negative sample and itself. In other words, the matching degree between the positive sample and the subsample image is higher than the matching degree between the negative sample and the subsample image.

[0076] This application does not impose specific limitations on the method of determining the positive and negative samples corresponding to the subsample image features. As a feasible implementation method, for each subsample image feature, the similarity between the subsample image feature and all subsample text features can be calculated. The similarity between the subsample image feature and the subsample text feature is used as the correlation between the two. Subsample text features with higher similarity are selected as positive samples, and subsample text features with lower similarity are selected as negative samples.

[0077] As another feasible approach, for each subsample image feature, the attention value vector of that subsample image feature and all subsample text features can be calculated to obtain the image attention value vector set corresponding to the subsample image feature. It can be understood that the attention value vector is mainly used to represent the correlation and importance between different information or elements, that is, the value of each attention value vector in the image attention value vector set is positively correlated with the subsample image feature and the subsample text feature corresponding to the attention value vector.

[0078] Therefore, from the set of image attention value vectors corresponding to the subsample image features, we can select the subsample text features corresponding to the attention value vectors with higher attention value vectors as positive samples, and the subsample text features corresponding to the attention value vectors with lower attention value vectors as negative samples.

[0079] In some embodiments, since the sample text and sample images selected during model training are matched, the resulting subsample images and subsample texts may have a one-to-many relationship, meaning that one subsample image may match multiple subsample texts. If, when selecting positive and negative samples corresponding to subsample texts, the subsample image features with the highest relevance are selected as positive samples, and the subsample image features with the lowest relevance are selected as negative samples, then multiple subsample texts may have the same subsample image as their corresponding positive samples. This can lead to some subsample images not being able to participate in model training, preventing the model from learning the semantic information of each subsample image, and consequently, the model cannot accurately analyze the sub-semantics in the image or text.

[0080] Therefore, as a feasible approach, when determining the positive and negative samples corresponding to each subsample image feature, two subsample text features can be randomly selected, and the correlation between the subsample image feature and the two subsample text features can be judged. The subsample text features with higher correlation are taken as positive samples of the subsample image features, and the subsample text features with lower correlation are taken as negative samples of the subsample image features.

[0081] This ensures that the positive and negative samples corresponding to the features of the sub-sample images have a certain degree of randomness, avoiding the situation where one sub-sample image matches multiple sub-sample texts, which would prevent some sub-sample images from participating in model training. This allows the model to better learn the semantic information of each sub-image, and thus accurately analyze the sub-semantics in the image or text.

[0082] It should be understood that this embodiment is only used as an example to illustrate the determination of positive and negative samples corresponding to the image features of each subsample. In S305, the determination of positive and negative samples corresponding to the text features of each subsample can also be done by random selection, which will not be elaborated here.

[0083] After obtaining the positive and negative samples corresponding to the features of the subsample image, the semantic contrast loss of the subsample image corresponding to the features of the subsample image can be determined.

[0084] As a feasible approach, when determining the semantic contrast loss of a sub-image corresponding to a sub-sample image feature, the sub-sample image feature can be used as an anchor feature, and then the triplet contrast loss can be used to determine the semantic contrast loss of the sub-image. The formula for calculating the triplet contrast loss is shown in formula (1) below:

[0085] L triplet (a,p,n)=max(d(a,p)-d(a,n)+margin, 0) Formula (1)

[0086] Where a, p, and n represent the anchor feature, the positive sample feature of the anchor feature, and the negative sample feature of the anchor feature, respectively; margin is a preset threshold used to control the difference between the positive and negative samples. This threshold ensures that the distance between the anchor feature and the negative sample is at least one margin greater than the distance to the positive sample; the function d(a,p) calculates the cosine similarity between features a and p, as shown in the following formula (2):

[0087]

[0088] As another feasible approach, we can first input the above a, p, and n into a set of trainable parameters θ. cThe cross-attention layer yields an adaptively adjustable margin value, which is then used as the margin. Thus, the triplet contrastive loss can be determined using the following formula (3):

[0089] L triplet (a,p,n;θ c )=max(d(a,p)-d(a,n)+margin(a,p,n),0) Formula (3)

[0090] Where, θ c This refers to a cross-attention layer with a set of trainable parameters, where margin(a,p,n) is the input θ of features a, p, and n. c The marginal value that can be adaptively adjusted will be obtained later.

[0091] S305. For each subsample text feature, based on the correlation between the subsample text feature and each subsample image feature, determine the positive and negative samples corresponding to the subsample text feature from all subsample image features; and determine the subsample semantic contrast loss based on the subsample text feature and the positive and negative samples corresponding to the subsample text feature.

[0092] Among them, the correlation between positive samples corresponding to sub-sample text features and sub-sample text features is higher than the correlation between negative samples corresponding to sub-sample text features and sub-sample text features; sub-text semantic contrast loss is used to characterize the ability of image retrieval model to identify positive and negative samples corresponding to sub-sample text features.

[0093] This application does not impose specific limitations on the method of determining the positive and negative samples corresponding to the subsample text features. As a feasible implementation method, for each subsample text feature, the similarity between the subsample text feature and all subsample image features can be calculated. The similarity between the subsample text feature and the subsample image feature is taken as the correlation between the two. The subsample image feature with higher similarity is selected as the positive sample, and the subsample image feature with lower similarity is selected as the negative sample.

[0094] As another feasible approach, for each subsample text feature, the attention value vector of that subsample text feature and all subsample image features can be calculated to obtain the image attention value vector set corresponding to the subsample text feature. It can be understood that the attention value vector is mainly used to represent the correlation and importance between different information or elements, that is, the value of each attention value vector in the image attention value vector set is positively correlated with the subsample text feature and the subsample image feature corresponding to the attention value vector.

[0095] Therefore, from the set of text attention value vectors corresponding to the subsample text features, we can select the subsample image features corresponding to the attention value vectors with higher attention value vectors as positive samples, and the subsample image features corresponding to the attention value vectors with lower attention value vectors as negative samples.

[0096] After obtaining the positive and negative samples corresponding to the sub-sample text features, the sub-text semantic contrast loss corresponding to the sub-sample text features can be determined. It should be understood that the method for determining the sub-text semantic contrast loss corresponding to the sub-sample text features can refer to the method for determining the sub-image semantic contrast loss in S304 above, and will not be repeated here in this embodiment.

[0097] S306. Based on the sub-image semantic contrast loss corresponding to the text features of all sub-samples and the sub-text semantic contrast loss corresponding to the image features of all sub-samples, determine the sub-semantic contrast loss corresponding to the sample pair.

[0098] Understandably, as a feasible approach, after obtaining the sub-image semantic contrast loss corresponding to the text features of all sub-samples and the sub-text semantic contrast loss corresponding to the image features of all sub-samples, their average can be used as the sub-semantic contrast loss corresponding to the sample pair.

[0099] As another feasible approach, since the importance of the content in the text or image varies, such as the background of an image which generally does not contain important information, different weights can be assigned to the text features or image features of each subsample. The sub-semantic contrast loss of the sample pair can be determined by weighted summation of the semantic contrast loss of all sub-images and the semantic contrast loss of the sub-text.

[0100] S307. Update the image retrieval model based on the sub-semantic contrast loss corresponding to multiple sample pairs until the image retrieval model meets the convergence condition, and obtain the target image retrieval model.

[0101] After obtaining the sub-semantic contrast loss corresponding to a sample pair, the partial derivatives of the second sub-semantic contrast loss with respect to the weight parameters of each neuron in the image retrieval model can be obtained through the backpropagation algorithm. This constitutes the gradient of each weight parameter, and the parameters of the image retrieval model are then updated based on the calculated gradients to complete the update of the image retrieval model.

[0102] After each update of the image retrieval model, it can be determined whether the image retrieval model meets the convergence condition. If it does, it means that the model has been trained and the target image retrieval model has been obtained.

[0103] In some embodiments, since the goal of the image retrieval model is to retrieve images that match the input text, the matching degree between the text and the image can be used to determine whether the image retrieval model meets the convergence condition.

[0104] As a feasible approach, the sample image and sample text can be input into the image encoder and text encoder of the image retrieval model, respectively, to obtain the corresponding sample image features and sample text features. Then, the cosine similarity between the sample image features and sample text features can be determined using the following formula (4):

[0105]

[0106] Where Ai and Bi represent the sample image features and sample text features, respectively.

[0107] After calculating the cosine similarity between the features of the sample image and the features of the sample text, the semantic distance between the sample image and the sample text can be represented by the magnitude of the similarity. The larger the cosine similarity, the greater the probability of the image and text being paired, and vice versa. When the determined cosine similarity exceeds the preset similarity threshold, the image retrieval model can be considered to have met the convergence condition and can be used for text-based image retrieval.

[0108] As shown in S301-S307, the training method of the image retrieval model provided in this application embodiment is as follows: First, the sample image and sample text are divided into sub-sample images and sub-sample text, respectively. Then, after feature extraction by the text encoder and image encoder in the image retrieval model, the positive and negative samples corresponding to the features of each sub-sample image and the positive and negative samples corresponding to the features of each sub-sample text are determined. Thus, when training the image retrieval model, the sub-image semantic contrast loss corresponding to each sub-sample image and the sub-text semantic contrast loss corresponding to each sub-sample text are determined. Then, based on the semantic contrast losses of all sub-images and all sub-texts, the sub-semantic contrast loss corresponding to each sample pair is determined. The image retrieval model is updated based on the sub-semantic contrast losses corresponding to multiple sample pairs until the image retrieval model meets the convergence condition, thus obtaining the target image retrieval model.

[0109] It can be understood that the sub-image semantic contrast loss can characterize the ability of the image retrieval model to distinguish positive and negative samples corresponding to the sub-sample image features, and the sub-text semantic contrast loss can characterize the ability of the image retrieval model to distinguish positive and negative samples corresponding to the sub-sample text features. Therefore, the sub-image semantic contrast loss and the sub-text semantic contrast loss are used to update the image retrieval model. The updated image retrieval model can better obtain the positive and negative samples of each sub-image in the image and the positive and negative samples of each sub-text in the text, and then can perform semantic alignment of text and image at a fine granularity, so that the trained target image retrieval model can accurately associate and match images and texts.

[0110] In some embodiments, there may be some irrelevant redundant content in the sample image and the sample text, such as the background part in the sample image, the verbs and conjunctions in the sample text, etc., such as "de" and "涂有" in "年轻的她涂有口红", which do not have effective semantic information. If this part of the content without effective information is selected as the positive sample or negative sample when determining the positive and negative samples corresponding to the sub-sample image or the positive and negative samples corresponding to the sub-sample text, it will inevitably affect the training result of the model.

[0111] Based on this, as a feasible implementation method, in the training method of the image retrieval model provided by the embodiments of the present application, when determining the positive and negative samples corresponding to the sub-sample text features among all the sub-sample image features, a screening operation is first performed to filter out the sub-sample image features with low correlation with the sub-sample text features, and then the positive and negative samples corresponding to the sub-sample text features are determined.

[0112] Specifically, in the training method of the image retrieval model provided by the embodiments of the present application, determining the positive and negative samples corresponding to the sub-sample text features based on the set of text attention value vectors corresponding to the sub-sample text features includes the following steps:

[0113] S11. Input the set of text attention value vectors corresponding to all sub-sample text features and the set of image attention value vectors corresponding to all sub-sample image features into the attention screening unit. The attention screening unit suppresses the non-maximum values in the set of text attention value vectors to obtain the screened set of text attention value vectors corresponding to the sub-sample text features, and suppresses the non-maximum values in the set of image attention value vectors to obtain the screened set of image attention value vectors corresponding to the sub-sample image features.

[0114] In attention mechanisms, non-maximum suppression can be used to filter the set of attention value vectors, retaining the maximum values ​​within local regions and suppressing non-maximum values, thereby more accurately locating key regions in images or text. In other words, by suppressing non-maximum values ​​in both the text and image attention value vector sets, we can filter out image attention values ​​with lower correlation. The resulting filtered set of image and text attention value vectors shows a higher correlation between the subsample text features and subsample image features corresponding to the attention values ​​in both sets.

[0115] It is understandable that in the initial stage of training, the filtering performance of the attention filtering unit has certain problems and may not be able to filter the text attention value vector set and the image attention value vector set. Therefore, during the model training process, the attention filtering unit can be updated synchronously, that is, the calculated multi-semantic contrast loss is used to update the attention filtering unit, text encoder and image encoder at the same time. The embodiments of this application will not be described in detail here.

[0116] S12. Based on the set of filtered text attention value vectors corresponding to the subsample text features and the set of filtered image attention value vectors corresponding to the subsample image features, determine the positive and negative samples corresponding to the subsample text features.

[0117] It should be understood that, as a feasible implementation method, when determining the positive and negative samples corresponding to the features of the subsample images, S11 can also be used to determine the set of text attention value vectors and the set of image attention value vectors after filtering, so as to reduce the influence of weakly correlated attention values ​​on the selection of positive and negative samples. This application will not elaborate further here.

[0118] It is understandable that, for the same subsample text feature and the same subsample image feature, the corresponding text attention value represents the influence of the subsample text feature on its corresponding subsample image feature, and the corresponding image attention value represents the influence of the subsample image feature on its corresponding subsample text feature.

[0119] Therefore, when determining the positive and negative samples corresponding to the text features of a subsample, it is necessary to use the filtered set of text attention value vectors and the filtered set of image attention value vectors. That is, the positive and negative samples corresponding to the text features of a subsample are determined by using both text attention values ​​and image attention values. This can more accurately determine the relationship between the image and the text, and thus more accurately determine the positive and negative samples corresponding to the text features of a subsample.

[0120] As can be seen from the above embodiments, the training method of the image retrieval model provided in this application performs a screening operation first when determining the positive and negative samples corresponding to the subsample text features from all subsample image features. This process removes subsample image features with low relevance to the subsample text features before determining the positive and negative samples corresponding to the subsample text features. This effectively improves the accuracy of selecting negative and positive samples corresponding to the subsample text features, thereby improving the accuracy of the trained image retrieval model.

[0121] As a feasible implementation, S12 can be specifically implemented as follows: A two-dimensional array is constructed based on the filtered set of text attention value vectors and the filtered set of image attention value vectors. The rows and columns of the two-dimensional array correspond to the text features and image features of each subsample, respectively. Each element in the two-dimensional array is the product of the text attention value and image attention value corresponding to its corresponding subsample text feature and subsample image feature, respectively. Furthermore, when determining the positive and negative samples corresponding to the subsample text features, two elements can be selected from the elements corresponding to that row / column, and their corresponding subsample image features can be used as the positive and negative samples corresponding to the subsample text features, respectively; similarly, when determining the positive and negative samples corresponding to the subsample image features, two elements can be selected from the elements corresponding to that column / row, and their corresponding subsample text features can be used as the positive and negative samples corresponding to the subsample image features, respectively.

[0122] As another feasible implementation method, please refer to Figure 4 S12 can be specifically implemented as follows:

[0123] S401. Determine the maximum attention value in the set of filtered text attention value vectors corresponding to each sub-sample text feature among all sub-sample text features, and obtain the set of maximum text attention values.

[0124] In this set of maximum attention values, each maximum attention value corresponds to a subsample text feature.

[0125] It is understandable that after non-maximum suppression in S11, each subsample text feature corresponds to a set of filtered text attention value vectors. Therefore, a maximum attention value can be selected from each set of text attention value vectors to obtain the maximum attention value corresponding to each subsample text feature, thus forming a set of maximum text attention values.

[0126] S402. Determine the maximum attention value in the set of filtered image attention value vectors corresponding to each sub-sample image feature among all sub-sample image features, and obtain the set of maximum image attention values.

[0127] In this set of maximum attention values, each maximum attention value corresponds to a feature of a subsample image.

[0128] It is understandable that after non-maximum suppression in S11, each subsample image feature corresponds to a set of filtered image attention value vectors. Therefore, a maximum attention value can be selected from each set of image attention value vectors to obtain the maximum attention value corresponding to each subsample image feature, so as to form a set of maximum image attention values.

[0129] It should be noted that S401 and S402 are not sequential. They can be executed simultaneously, or S401 can be executed first and then S402, or S402 can be executed first and then S401. This application embodiment does not impose any restrictions on this.

[0130] S403. For subsample text features, multiply the maximum attention value of the text corresponding to the subsample text feature by the maximum attention value of each image in the set of maximum attention values ​​of images to obtain the target set corresponding to the subsample text feature.

[0131] Since the maximum attention value of text refers to the correlation between the text features and the image features of the subsample, while the maximum attention value of image refers to the correlation between the image features and the text features of the subsample, although there is a certain relationship between the two, there are also certain differences. Therefore, in order to more accurately determine the relationship between the two, the maximum attention value of text corresponding to the subsample text features can be multiplied by the maximum attention value of each image in the set of maximum attention values ​​of images to obtain the target set corresponding to the subsample text features.

[0132] S404. Select the first vector and the second vector from the target set.

[0133] The extreme value of the first vector is greater than the extreme value of the second vector.

[0134] It is understandable that the extreme value of the first vector is greater than the extreme value of the second vector, indicating that the correlation between the subsample image features and subsample text features corresponding to the first vector is higher than the correlation between the subsample image features and subsample text features corresponding to the second vector. Therefore, in this embodiment, the subsample image features corresponding to the first vector are taken as positive samples corresponding to the subsample text features, and the subsample image features corresponding to the second vector are taken as negative samples corresponding to the subsample text features.

[0135] It should be noted that the above embodiments are based on the example of determining the positive and negative samples corresponding to the text features of a single subsample. In practical applications, the positive and negative samples corresponding to the text features of all subsamples and the positive and negative samples corresponding to the image features of all subsamples can be determined simultaneously.

[0136] Specifically, as a feasible implementation method, after obtaining the set of maximum attention values ​​for text and the set of maximum attention values ​​for images through S401 and S402, each element in the set of maximum attention values ​​for text can be multiplied by each element in the set of maximum attention values ​​for images to obtain an attention matrix.

[0137] For example, please refer to Figure 5 The relationship between the text features of N subsamples and the image features of M subsamples can be displayed using a matrix structure. For example... Figure 5 As shown, This represents the maximum attention value of the image corresponding to feature 0 in the subsample image; This represents the maximum attention value of the image corresponding to feature M-1 of the subsample image; This represents the maximum attention value of the text corresponding to feature 0 in the subsample text; This represents the maximum attention value of the text corresponding to the N-1 subsample text features. The black dots in the figure represent the product of the corresponding maximum attention value of the text and the maximum attention value of the image (the extreme value of the attention vector).

[0138] In this way, when determining the positive and negative samples corresponding to any subsample text feature, the selection can be made from the extreme values ​​of the attention vector corresponding to that subsample text feature, such as... Figure 5 As shown, when determining the positive and negative samples corresponding to the subsample text feature N-1, selection can be made within the box line indicated by the bold solid line in the figure. Two extreme values ​​of attention vectors can be arbitrarily selected. If one extreme value of attention vector is greater than the other extreme value of attention vector, the subsample image feature corresponding to the larger extreme value of attention vector can be taken as the positive sample corresponding to the subsample text feature N-1, and the subsample image feature corresponding to the smaller extreme value of attention vector can be taken as the negative sample corresponding to the subsample text feature N-1.

[0139] When determining the positive and negative samples corresponding to any subsample image feature, selections can be made from the extreme values ​​of the attention vector corresponding to that subsample image feature, such as... Figure 5 As shown, when determining the positive and negative samples corresponding to the subsample image feature N-1, selection can be made within the box line indicated by the bold dashed line in the figure. Two extreme values ​​of attention vectors can be selected arbitrarily. If one extreme value of attention vector is greater than the other extreme value of attention vector, the subsample text feature corresponding to the larger extreme value of attention vector can be taken as the positive sample corresponding to the subsample image feature N-1, and the subsample text feature corresponding to the smaller extreme value of attention vector can be taken as the negative sample corresponding to the subsample image feature N-1.

[0140] As can be seen from the above embodiments, the training method of the image retrieval model provided in this application, when determining the positive and negative samples corresponding to the subsample text features among all subsample image features, determines the maximum attention value in the filtered text attention value vector set corresponding to each subsample text feature, and uses it as the maximum text attention value corresponding to the subsample text feature, thus obtaining the text maximum attention value set. Then, the maximum text attention value corresponding to the subsample text feature is multiplied by each element in the image maximum attention value set, and each vector in the resulting target set is used as the correlation between the subsample text feature and each subsample image feature. This can more accurately determine the association between the subsample image features and the subsample text features, and further determine the positive and negative samples based on the extreme values ​​of the vectors in the target set, thus more accurately determining the positive and negative samples corresponding to the subsample text features.

[0141] In some embodiments, to enhance the filtering capability of the attention filtering unit, the attention filtering unit may include multiple multi-head attention modules. It is understood that a multi-head attention module is an attention mechanism that plays a crucial role in many deep learning models, particularly in natural language processing and computer vision tasks. The core idea of ​​a multi-head attention module is to divide the input data into multiple distinct parts (often called "heads"), apply an attention mechanism to each part independently, and finally merge the outputs of these parts.

[0142] Specifically, as a feasible implementation method, S11 above can perform the following steps: input the set of text attention value vectors corresponding to all subsample text features and the set of image attention value vectors corresponding to all subsample image features into the first multi-head attention module of the attention filtering unit, and the input of other multi-head attention modules is the output of the previous multi-head attention module.

[0143] Please see Figure 6 For each multi-head attention module, the following steps are performed:

[0144] S601. Determine the maximum attention value in the set of text attention value vectors corresponding to the text features of each subsample to obtain the first maximum attention value set.

[0145] S602. Determine the maximum attention value in the set of image attention value vectors corresponding to the features of each subsample image to obtain the second maximum attention value set.

[0146] S603. Determine the filtering threshold based on the first maximum attention value set and the second maximum attention value set.

[0147] As a feasible approach, elements from the first and second maximum attention value sets can be multiplied to obtain multiple element-wise dot products. The average of these dot products, or the mean dot product, is then calculated. This mean dot product represents the average matching degree between the text features and image features of all subsamples; therefore, it can be used as a filtering threshold.

[0148] As another feasible implementation method, S603 can be implemented through the following steps:

[0149] S21. Input the first maximum attention value set and the second maximum attention value set into the weight neural network, and determine the correlation strength between the first maximum attention value set and the second maximum attention value set through the weight neural network to obtain the attention filtering weights.

[0150] The weighted neural network can include one or more MLP layers. Each MLP layer consists of an input layer, one or more hidden layers, and an output layer. Each layer contains a certain number of neurons connected by weights. MLP layers can handle complex nonlinear relationships and are trained using the backpropagation algorithm.

[0151] To avoid the element-wise dot product of elements in the first and second maximum attention value sets being too large or too small, which would hinder subsequent gradient calculation, an attention filtering weight is first determined based on the correlation strength between the first and second maximum attention value sets. Then, a filtering threshold is determined based on the dot product mean and the attention filtering weight. This approach improves filtering efficiency and avoids erroneous filtering.

[0152] In this embodiment, the first and second maximum attention value sets can be concatenated (i.e., the common concatenate operation) to output a weighted neural network. The weighted neural network is used to determine the correlation strength between the first and second maximum attention value sets, thus obtaining the attention filtering weights.

[0153] It should be understood that before using a weighted neural network, it can be trained by obtaining a set of training parameters until the preset convergence condition is met, and then applied to determine the attention selection weights.

[0154] S22. Multiply the elements in the first maximum attention value set and the elements in the second maximum attention value set to obtain a multi-element dot product.

[0155] S23. Calculate the average of the dot products of all elements to obtain the mean of the dot products.

[0156] S24. Determine the filtering threshold based on the dot product mean and the attention filtering weight.

[0157] It should be understood that, as one feasible implementation method, the screening threshold can be set as the dot product mean plus the attention screening weight. As another feasible implementation method, different weights can be set for the dot product mean and the attention screening weight, and the screening threshold can be determined by weighted summation. This application does not limit this approach.

[0158] As can be seen, the training method provided in this embodiment, when determining the screening threshold for filtering the first and second maximum attention value sets, first uses a weighted neural network to determine the correlation strength between the first and second maximum attention value sets, and obtains the attention screening weights. Then, the screening threshold is determined based on the mean of the dot product and the attention screening weights. This avoids the dot product of the elements in the first and second maximum attention value sets being too large or too small, which would be detrimental to subsequent gradient calculation. This improves the efficiency of screening and avoids incorrect screening.

[0159] S604. In the set of text attention value vectors corresponding to all subsample text features and the set of image attention value vectors corresponding to all subsample image features, set the attention value vectors that are less than or equal to the filtering threshold to 0.

[0160] It is understandable that attention value vectors less than or equal to the filtering threshold are values ​​with low matching degree. The corresponding subsample text features do not match the subsample image features (or the subsample image features match the subsample text features). Therefore, their attention value vectors can be set to 0 to avoid the mismatched features affecting subsequent calculations.

[0161] S605. Output the set of text attention value vectors corresponding to the updated text features of all subsamples, and the set of image attention value vectors corresponding to the updated image features of all subsamples.

[0162] Each multi-head attention module outputs a set of text attention value vectors corresponding to all updated subsample text features, and a set of image attention value vectors corresponding to all updated subsample image features.

[0163] It is understood that the input to the first multi-head attention module is the set of text attention value vectors corresponding to all sub-sample text features, and the set of image attention value vectors corresponding to all sub-sample image features; the input to other multi-head attention modules is the output of the previous multi-head attention module. The set of text attention value vectors and the set of image attention value vectors output by the last multi-head attention module are used to determine the positive and negative samples corresponding to each sub-sample text feature and the positive and negative samples corresponding to each sub-sample image feature, that is, to execute the above S401-S404, which will not be elaborated further in this application.

[0164] It should be noted that, since a large number of multi-head attention modules may result in better training, the device performance may not be able to support it. Therefore, as a feasible implementation method, the number of multi-head attention modules can be set to 10-14, balancing device performance and training efficiency. In practical applications, the number can be set according to the requirements. This application embodiment does not limit the number of multi-head attention modules.

[0165] In some embodiments, Large Language Models (LLMs) are capable of deeply understanding the meaning and context of text, and can effectively understand and generate human language. Therefore, when determining subsample texts, the semantic understanding capabilities of LLMs can be leveraged to segment the sample texts.

[0166] As a feasible implementation method, dividing the sample image into multiple sub-sample images in S302 can be specifically implemented as follows: inputting the sample text and preset prompt information into the target large language model, so that the target large language model splits the sample text into multiple sub-sample texts according to the preset prompt information.

[0167] Large language models (LLMs) can handle various text tasks, but their processing capabilities are affected by the design of prompts. How to design prompts for specific tasks is a key factor when operating an LLM for text tasks. Therefore, when using a large language model to segment sample text, the prompt design paradigm cannot be directly applied. A new prompt needs to be designed to divide the sample text into multiple sub-sample texts, serving as the aforementioned pre-defined prompt.

[0168] In some embodiments, to achieve semantic understanding and clause segmentation of sample text, the following methods can be used: Figure 7 The text splitting framework shown:

[0169] We collect a large amount of descriptive text and build a descriptive text database called Face-Text. Then, we use Face-Text to fine-tune the clause splitting capability of the LLM model, enabling it to perform sentence splitting based on specific dimensions according to the input descriptive text and the designed prompts.

[0170] Based on the characteristics of different subdivisions, prompts can be designed according to specific dimensions. For example, if we want to extract the makeup-related clause from the descriptive text "The young woman wore thick-rimmed sunglasses, had long, flowing hair dyed brownish-yellow, her eyebrows were not thick but were dark in color, she wore lipstick, and wore white round earrings," then the prompt could be designed as: "You are a professional and meticulous AI assistant. Please carefully understand the input text, then extract the makeup-related descriptions from the sentence and form a new clause. Now, the input text I give you is {xxxxxxxxx}."

[0171] The prompt and description text are then input into a large language model, which breaks down the description text into multiple sub-texts. The resulting sub-texts are: Sub-text 1: "She is young," Sub-text 2: "Wearing thick-rimmed sunglasses," Sub-text 3: "Long, flowing brown hair,"... Sub-text N: "Wearing white round earrings."

[0172] Similarly, we can design prompts based on dimensions such as the geometric distribution of the images to be retrieved (e.g., mouth, eyes, mouth, nose, etc.), age, accessories, expression, posture, defects, hair, etc., thereby guiding the LLM to split clauses according to the input text.

[0173] Understandably, fine-tuning of the LLM model is an ongoing process. Users can adjust the LLM model based on the output of the image retrieval model and the descriptive text they input, so that the clauses segmented by the LLM model can better represent different subdivisions.

[0174] In some embodiments, an image may also include information in multiple dimensions, such as the makeup and clothing of the people in the image. When dividing an image, it is also necessary to divide it according to different dimensions.

[0175] Based on this, as a feasible implementation method, please refer to Figure 8 The image retrieval model training method provided in this application divides a sample image into multiple sub-sample images using the following steps:

[0176] S801. Determine the key points in the sample image.

[0177] Because people in images have their own unique characteristics—such as the positional and connection relationships between limbs and body, and eyebrows generally above the eyes, but the shape and thickness of eyebrows vary from person to person—when dividing a sample image into multiple sub-sample images, we can first determine some key points corresponding to the human body, and then determine the key points in the sample image based on the correspondence between the human body and the sample image.

[0178] S802. Input the sample image and sample text into the image encoder and text encoder respectively to obtain the corresponding sample image features and sample text features.

[0179] S803. Based on the sample image features and sample text features, perform gradient updates to obtain the heatmap corresponding to the sample image.

[0180] The sample text features are used as the optimization target of the image encoder for backpropagation. During the backpropagation, the gradient of the last convolutional layer of the image encoder is updated to obtain the response value act and gradient value grad of the last convolutional layer of the image encoder. Multiplying act and grad by a dot product and averaging by the channel dimensions yields a gradient heatmap Grad_heatmap that is smaller than the sample image size, as shown in formula (5) below:

[0181]

[0182] Furthermore, the gradient heatmap is enlarged to the same size as the sample image using nearest neighbor interpolation, and then a heatmap is drawn.

[0183] S804. The local region with the strongest response in the heat map is taken as the image sub-region to be matched with the sample text.

[0184] The strongest local responses in a heatmap typically refer to the darkest, densest areas, representing hotspots or high-density regions in the data. In a heatmap, darker colors generally indicate a stronger response, meaning the region best matches the sample text. Therefore, the strongest local response in the heatmap can be selected, and its location can be defined as the image sub-region that matches the input text.

[0185] S805. Determine the key points and key point combinations within the sub-region of the image, and set a cropping box based on the key points and key point combinations within the sub-region of the image to crop the sample image, thereby obtaining multiple sub-sample images.

[0186] After obtaining the corresponding heatmap, key points and key point combinations within the image sub-regions are identified. For each key point and key point combination, the width and height parameters Rw and Rh of the cropping boxes (two rectangles and one square) for different regions are determined according to preset rules. The preset rules include at least the following: key points are linked to face shape, such as the nose height being proportional to the face length; different key points have different regions of interest sizes, such as eyebrows being longer and the nose tip being rounder.

[0187] After obtaining the keypoints and keypoint combinations, as well as the width and height parameters Rw and Wh of the cropping box, image sub-region cropping can be performed. Specifically, assuming the selected keypoint anchor coordinates are (x, y), and its corresponding anchor width and height parameters are Rw and Rh, the coordinates of the cropping region (rectangle) corresponding to this anchor point can be: x1 = x - Rw; x2 = x + Rw; y1 = y - Rh; y2 = y + Rh. Here, x1 and y1 are the coordinates of the upper left corner of the cropping region, and x2 and y2 are the coordinates of the lower right corner of the cropping region.

[0188] As can be seen from S801-S805, the training method of the image retrieval model provided in this application divides the sample image into multiple sub-sample images. It uses the method of drawing gradient heatmaps to determine the image sub-regions that best match the sample text. Then, it sets a cropping box based on the key points and key point combinations within the image sub-regions to crop the sample image. The resulting multiple sub-sample images can be better aligned with the sample text, thereby improving the training effect of the image retrieval model.

[0189] In some embodiments, the method of labeling target images with positive and negative sample tags in related technologies for model training, and using discrete labels and loss functions to optimize text semantics and image semantics with continuous distribution characteristics, impairs fine-grained feature alignment. For example, text A "a young woman with long hair wearing glasses" forms a positive sample pair with image A, and text B "a young woman with long hair wearing a hat" forms a negative sample pair with image A. However, text A and text B actually differ by only two characters, and from the perspective of feature distance, they are mutually exclusive.

[0190] For example, please refer to Figure 9 In related technologies, when training a model, text A is labeled as a positive sample of image A, and text B is labeled as a negative sample of image A. In the features extracted by the trained image retrieval model, the features of text A are close to the features of image A, while the features of text B are far from the features of image A. However, in reality, the semantics of text A and text B are very close, and related technologies cannot handle this mutually exclusive situation well.

[0191] Based on this, as a feasible implementation method, please refer to Figure 10 The training method for the image retrieval model provided in this application embodiment further includes the following steps:

[0192] S1001. Obtain the synonyms, near-synonyms, and antonyms corresponding to the sample text, and input the synonyms, near-synonyms, and antonyms into the text encoder respectively to obtain the corresponding synonym features, near-synonyms features, and antonyms features.

[0193] As a feasible approach, a large language model can be used to obtain multi-semantic text sets. This requires designing some prompts based on the content of the sample text to guide the LLM to output the corresponding synonyms, near-synonyms, and antonyms based on the sample text.

[0194] For example, suppose the sample text is "A young woman wearing thick-rimmed sunglasses has long, flowing hair dyed brownish-yellow, with dark but not thick eyebrows, wearing lipstick, and white round earrings." The corresponding prompt can be set to "Based on the input text, output text semantically similar to the input text according to dimensions such as gender, age, accessories, facial features, facial expression, and posture. For example, if the input is <a young, beautiful woman>, the output might be ." Inputting this sample text and prompt into a large language model, the model can output based on the input sample text and prompt. The possible output would be "A woman wearing glasses has long, flowing brownish-yellow hair, dark but not thick eyebrows, wears lipstick, and has white earrings." This is essentially the synonymous text corresponding to the above sample text.

[0195] It should be understood that the methods for obtaining synonyms and antonyms also have their corresponding prompts, which will not be elaborated here.

[0196] After obtaining the synonyms, near-synonyms, and antonyms corresponding to the sample text, and inputting the synonyms, near-synonyms, and antonyms into the text encoder respectively, the corresponding synonym features, near-synonyms features, and antonyms features are obtained.

[0197] S1002. Input the sample image and sample text into the image encoder and text encoder respectively to obtain the corresponding sample image features and sample text features.

[0198] S1003. Construct multiple triples based on sample image features, sample text features, synonym text features, near-synonymous text features, and antonymous text features.

[0199] In each triplet, the negative sample is the antonymous text feature.

[0200] It is understandable that a triple consists of an anchor point, a positive sample, and a negative sample. The correlation between the anchor point and the positive sample is higher than that between the anchor point and the negative sample. Therefore, among the sample image features, sample text features, synonym text features, near-synonymous text features, and antonymous text features, only antonymous text features can be used as negative samples. The other features have high correlations and can be used as positive samples and anchor points, respectively.

[0201] As a feasible implementation method, we can first construct three triples by using sample image features as anchors, sample text features, synonym text features, and near-synonymous text features as positive samples, and antonymous text features as negative samples. Then, we construct two triples by using sample text features as anchors, synonym text features, and near-synonymous text features as positive samples, and antonymous text features as negative samples. Finally, we construct a single triple by using synonym text features as anchors, near-synonymous text features as positive samples, and antonymous text features as negative samples.

[0202] S1004. Determine the loss function corresponding to each triplet, and determine the multi-semantic contrast loss corresponding to the sample pair based on the loss functions corresponding to all triplets.

[0203] After obtaining multiple triples, the loss function corresponding to each triple can be determined according to the above formula (1) or formula (2), which will not be elaborated here.

[0204] Understandably, the loss function corresponding to each triple is used to characterize the difference between the distance between the anchor point and the positive sample and the distance between the anchor point and the negative sample. In other words, the value of the loss function corresponding to each triple reflects the model's ability to distinguish between positive and negative samples of the anchor point.

[0205] For example, by using sample image features as anchor points, sample text features as positive samples, and antonymous text features as negative samples, the calculated loss function can characterize the distance between sample image features and sample text features, relative to the degree of difference between the distance between sample image features and antonymous text features.

[0206] It is understandable that the loss function corresponding to a triple is used to represent the model's ability to distinguish between positive and negative samples corresponding to the anchor point of that triple, and the loss function corresponding to multiple triples can characterize the model's ability to distinguish between positive and negative samples corresponding to different semantic representations.

[0207] After obtaining the loss function for each triplet, a multi-semantic contrastive loss is determined based on the loss functions for all triplet pairs. This multi-semantic contrastive loss represents the model's ability to distinguish between positive and negative samples corresponding to different semantic representations. Furthermore, during model training, the model can learn semantic consistency across different contexts and expressions by comparing the distance or similarity between different semantic representations, thus learning more fine-grained semantic information. This helps the model more accurately understand the meaning in natural language, resulting in higher performance when processing complex text data and improved overall robustness.

[0208] One feasible approach is to average the loss functions corresponding to all triples to obtain the corresponding multi-semantic contrastive loss. Another feasible approach is to take a weighted average of the loss functions corresponding to all triples to obtain the corresponding multi-semantic contrastive loss. This application does not impose any limitations on this approach.

[0209] After obtaining the multi-semantic contrast loss corresponding to the sample pair, the image retrieval model can be updated using the multi-semantic contrast loss corresponding to the sample pair.

[0210] Specifically, as a feasible implementation method, updating the image retrieval model based on the sub-semantic contrast loss corresponding to multiple sample pairs in S307 can be implemented as: updating the image retrieval model based on the sub-semantic contrast loss corresponding to multiple sample pairs and the multi-semantic contrast loss.

[0211] In other words, after obtaining the sub-semantic contrast loss and multi-semantic contrast loss for each sample pair, the image retrieval model is updated based on these losses. As a feasible implementation, different weights can be assigned to the sub-semantic contrast loss and the multi-semantic contrast loss, and a weighted summation method can be used to determine the total loss function. Then, the total loss function is used to update the image retrieval model.

[0212] As can be seen from the above embodiments, the image retrieval model training method provided in this application requires, during the training of the image retrieval model, not only to determine the sub-semantic contrast loss corresponding to the sample pair, but also to determine the synonyms, near-synonyms, and antonyms corresponding to the sample text, thereby obtaining the multi-semantic contrast loss corresponding to the sample pair. It is understood that the multi-semantic contrast loss can better characterize the distance relationship between features, and thus, after model updating, the updated model can better model the feature distance of image and text features with multi-semantic relationships more accurately.

[0213] In some embodiments, in order to ensure that the image retrieval model can perform feature alignment between the input text and the image, it is also necessary to ensure that the image retrieval model can align the overall features of the input text and the overall features of the image.

[0214] As one feasible approach, please refer to Figure 11 The training method for the image retrieval model provided in this application embodiment further includes the following steps:

[0215] S1101. Obtain the negative sample text corresponding to the sample image.

[0216] It is understood that the sample text in the embodiments of this application refers to the text related to the sample image, that is, the text that matches the content of the sample image. Therefore, when determining the contrastive learning loss, the sample text can be directly used as the positive sample of the sample image, and only the negative sample text corresponding to the sample image needs to be obtained.

[0217] As a feasible approach, sample text from other sample pairs can be used as negative sample text for the sample image in that sample pair.

[0218] S1102. Input the sample image, sample text, and the corresponding negative sample text of the sample image into the image encoder and the text encoder respectively to obtain the corresponding sample image features, sample text features, and negative sample text features.

[0219] S1103. Determine the contrastive learning loss based on sample image features, sample text features, and negative sample text features.

[0220] Among them, the contrastive learning loss is used to characterize the ability of the image retrieval model to distinguish between positive and negative samples corresponding to the features of the sample image.

[0221] For example, as a feasible implementation, since the sample text is a positive sample of the sample image and the negative sample text is a negative sample of the sample image, the label of the sample text can be set to 1 and the label of the negative sample text can be set to 0, that is, the label of the sample text feature can be set to 1 and the label of the negative sample text feature can be set to 0.

[0222] Furthermore, the feature distances between sample image features and sample text features, and between sample image features and negative sample text features, can be determined. As a feasible approach, the cosine similarity between sample image features and sample text features can be determined and used as the feature distance between them. The larger the cosine similarity, the smaller the distance between the features, meaning the two features are more similar. Similarly, the cosine similarity between sample image features and negative sample text features can be determined and used as the feature distance between them. The smaller the cosine similarity, the larger the distance between the features, meaning the two features are less similar.

[0223] Then, the loss between sample image and sample text, and between sample image and negative sample text is determined by constructing a contrast loss, as shown in the following formula (6):

[0224]

[0225] Where d represents the feature distance between two features, y is the label corresponding to the feature, and N is the number of sample pairs. It can be understood that the optimization objective of this loss function is to bring the feature distance between sample image features and sample text features closer together, and to widen the feature distance between sample image features and negative sample text features.

[0226] As a feasible approach, the loss corresponding to the sample image features and sample text features, as well as the loss corresponding to the sample image features and negative sample text features, can be determined by formula (6), and then the contrastive learning loss can be determined by averaging.

[0227] Specifically, as a feasible implementation method, updating the image retrieval model based on the sub-semantic contrast loss corresponding to multiple sample pairs in S307 can be implemented as follows: updating the image retrieval model based on the sub-semantic contrast loss corresponding to multiple sample pairs and the contrast learning loss.

[0228] As can be seen, the image retrieval model training method provided in this embodiment requires not only determining the sub-semantic contrast loss corresponding to the sample pair, but also determining the contrastive learning loss corresponding to the sample pair when training the image retrieval model. This ensures that the trained image retrieval model aligns with the overall features of the input text and the overall features of the image.

[0229] In some embodiments, in order to ensure the retrieval performance of the image retrieval model, when updating the model, the model can be updated simultaneously based on the contrastive learning loss, multi-semantic contrastive loss, and sub-semantic contrastive loss determined in the above embodiments.

[0230] For example, as one feasible implementation method, please refer to Figure 12The training method for the image retrieval model provided in this application embodiment may include the following steps in practical applications:

[0231] S1201. Input the sample text into the large language model to obtain multiple sub-sample texts and the set of polysemous sentences corresponding to the sample texts; divide the sample image into multiple sub-sample images.

[0232] S1202. Input the sample text, multiple sub-sample texts, and the set of polysemous sentences corresponding to the sample text into the text encoder to obtain the corresponding text features; input the sample image and multiple sub-sample images into the image encoder to obtain the corresponding image features.

[0233] S1203. Determine the contrastive learning loss, multi-semantic contrastive loss, and sub-semantic contrastive loss.

[0234] The contrastive learning loss is determined by the features of the sample text corresponding to the sample text and the features of the sample image corresponding to the sample image; the multi-semantic contrastive loss is determined by inputting the features of each sentence in the set of polysemous sentences corresponding to the sample text and the features of the sample image into the attention filtering unit; and the sub-semantic contrastive loss is determined by the features of multiple sub-sample texts and multiple sub-sample images.

[0235] S1204, Update the image retrieval model and attention filtering unit.

[0236] After obtaining the contrastive learning loss, multi-semantic contrastive loss, and sub-semantic contrastive loss, the total loss can be determined by weighted summation or direct summation. Gradient backpropagation is then performed based on the total loss to update the image encoder, text encoder, and attention filtering unit in the image retrieval model.

[0237] Understandably, updating the attention filter unit can make the multi-semantic contrast loss determined by the attention filter unit more accurate, thereby enabling better training of the image retrieval model.

[0238] This application also provides an image retrieval method applicable to image retrieval devices that deploy the target image retrieval model provided in the above embodiments. The image retrieval device can be a terminal device, such as a personal computer, laptop computer, mobile device, tablet computer, or laptop computer. This application does not limit the specific form of the electronic device. Alternatively, the image retrieval device can be a single server or a server cluster composed of multiple servers. In some implementations, the server cluster can be a distributed cluster server. This application does not impose any restrictions on the specific form of the image retrieval device.

[0239] Please see Figure 13 The image retrieval method includes the following steps:

[0240] S1301. Obtain the text to be searched.

[0241] Understandably, the text to be retrieved is usually descriptive text generated by the user based on the image they need, which includes some characteristics of the image, such as: "a young man wearing sunglasses", etc.

[0242] S1301. Input the text to be retrieved into the target image retrieval model, and retrieve the target image that matches the text to be retrieved from the image database through the target image retrieval model.

[0243] Understandably, image databases store a large number of images. When any image is stored in an image database, it first needs to use the image encoder in the target image retrieval model to obtain the image features corresponding to the image. Then, the image and image features are stored in the image database. During retrieval, the target image retrieval model obtains the text features of the text to be retrieved through the text encoder, and matches these text features with the image features in the image database, such as by calculating similarity to determine the matching degree.

[0244] If the matching degree between image features and text features exceeds a preset threshold, the image corresponding to that image feature can be used as the target image. Alternatively, the images corresponding to the top K matching features can be selected as the target images.

[0245] It is understandable that if the image retrieved by the target image retrieval model does not meet the user's needs, the user can modify the text to be retrieved and then continue to input it into the target retrieval model for retrieval until the retrieval is successful or the number of retrievals exceeds the preset number. This application embodiment does not impose any restrictions on this.

[0246] This application also provides a training device for an image retrieval model. Please refer to [link to relevant documentation]. Figure 14 The training device 140 for the image retrieval model includes: an acquisition module 41, a segmentation module 42, an input module 43, a determination module 44, and an update module 45.

[0247] The acquisition module 41 is used to acquire multiple sample pairs, wherein each sample pair includes a sample image and a sample text.

[0248] The segmentation module 42 is used to segment the sample image into multiple sub-sample images and the sample text into multiple sub-sample text for each sample pair.

[0249] The input module 43 is used to input the subsample image into the image encoder in the image retrieval model to obtain the corresponding subsample image features, and to input the subsample text into the text encoder in the image retrieval model to obtain the corresponding subsample text features.

[0250] The determination module 44 is used to determine the positive and negative samples corresponding to the subsample image features from all subsample text features for each subsample image feature, based on the correlation between the subsample image features and each subsample text feature; and to determine the subsample semantic contrast loss based on the subsample image features and the positive and negative samples corresponding to the subsample image features; wherein, the correlation between the subsample image features and the positive samples corresponding to the subsample image features is higher than the correlation between the subsample image features and the negative samples corresponding to the subsample image features; the subsample semantic contrast loss is used to characterize the ability of the image retrieval model to identify the positive and negative samples corresponding to the subsample image features.

[0251] The determination module 44 is further configured to, for each subsample text feature, determine the positive and negative samples corresponding to the subsample text feature from all subsample image features based on the correlation between the subsample text feature and each subsample image feature; and determine the subtext semantic contrast loss based on the subsample text feature and the positive and negative samples corresponding to the subsample text feature; wherein, the correlation between the subsample text feature and the positive sample corresponding to the subsample text feature is higher than the correlation between the subsample text feature and the negative sample corresponding to the subsample text feature; the subtext semantic contrast loss is used to characterize the ability of the image retrieval model to identify the positive and negative samples corresponding to the subsample text features.

[0252] The determination module 44 is also used to determine the sub-semantic contrast loss corresponding to the sample pair based on the sub-image semantic contrast loss corresponding to the text features of all sub-samples and the sub-text semantic contrast loss corresponding to the image features of all sub-samples.

[0253] The update module 45 is used to update the image retrieval model based on the sub-semantic contrast loss corresponding to multiple sample pairs until the image retrieval model meets the convergence condition and the target image retrieval model is obtained.

[0254] In some embodiments, the determining module 44 is specifically configured to, for each subsample image feature, determine the attention value vector of the subsample image feature and each subsample text feature, to obtain a set of text attention value vectors corresponding to the subsample image feature; the value of each attention value vector in the set of text attention value vectors is positively correlated with the subsample image feature and the subsample text feature corresponding to the attention value vector; based on the set of text attention value vectors corresponding to the subsample image feature, determine the positive and negative samples corresponding to the subsample image feature; for each subsample text feature, determine the attention value vector of the subsample text feature and each subsample image feature, to obtain a set of text attention value vectors corresponding to the subsample text feature, the value of each attention value vector in the set of text attention value vectors is positively correlated with the subsample text feature and the subsample image feature corresponding to the attention value vector; based on the set of text attention value vectors corresponding to the subsample text feature, determine the positive and negative samples corresponding to the subsample text feature.

[0255] In some embodiments, the determining module 44 is specifically used to input the set of text attention value vectors corresponding to all subsample text features and the set of image attention value vectors corresponding to all subsample image features into the attention filtering unit. The attention filtering unit suppresses non-maximum values ​​in the set of text attention value vectors to obtain the filtered set of text attention value vectors corresponding to the subsample text features, and suppresses non-maximum values ​​in the set of image attention value vectors to obtain the filtered set of image attention value vectors corresponding to the subsample image features. Based on the filtered set of text attention value vectors corresponding to the subsample text features and the filtered set of image attention value vectors corresponding to the subsample image features, the positive and negative samples corresponding to the subsample text features are determined.

[0256] In some embodiments, the determining module 44 is specifically used to determine the maximum attention value in the set of filtered text attention value vectors corresponding to each sub-sample text feature in all sub-sample text features, to obtain a set of maximum text attention values; each maximum text attention value in the set of maximum text attention values ​​corresponds to a sub-sample text feature; determine the maximum attention value in the set of filtered image attention value vectors corresponding to each sub-sample image feature in all sub-sample image features, to obtain a set of maximum image attention values; each maximum image attention value in the set of maximum image attention values ​​corresponds to a sub-sample image feature; for a sub-sample text feature, multiply the maximum text attention value corresponding to the sub-sample text feature by the maximum image attention value in the set of maximum image attention values ​​to obtain a target set corresponding to the sub-sample text feature; select a first vector and a second vector in the target set, wherein the extreme value of the first vector is greater than the extreme value of the second vector; take the sub-sample image feature corresponding to the first vector as the positive sample corresponding to the sub-sample text feature; and take the sub-sample image feature corresponding to the second vector as the negative sample corresponding to the sub-sample text feature.

[0257] In some embodiments, the attention filtering unit includes multiple multi-head attention modules. The determining module 44 is specifically used to input the set of text attention value vectors corresponding to all sub-sample text features and the set of image attention value vectors corresponding to all sub-sample image features into the first multi-head attention module of the attention filtering unit. The input of other multi-head attention modules is the output of the previous multi-head attention module. Each multi-head attention module performs the following steps: determining the maximum attention value in the set of text attention value vectors corresponding to each sub-sample text feature to obtain a first maximum attention value set; determining the maximum attention value in the set of image attention value vectors corresponding to each sub-sample image feature to obtain a second maximum attention value set; determining a filtering threshold based on the first and second maximum attention value sets; setting attention value vectors less than or equal to the filtering threshold to 0 in the set of text attention value vectors corresponding to all sub-sample text features and the set of image attention value vectors corresponding to all sub-sample image features; and outputting the updated set of text attention value vectors corresponding to all sub-sample text features and the updated set of image attention value vectors corresponding to all sub-sample image features.

[0258] In some embodiments, the determining module 44 is specifically used to input the first maximum attention value set and the second maximum attention value set into a weighted neural network, determine the correlation strength between the first maximum attention value set and the second maximum attention value set through the weighted neural network, and obtain attention filtering weights; multiply the elements in the first maximum attention value set and the elements in the second maximum attention value set to obtain multiple element dot products; calculate the average of all element dot products to obtain the dot product mean; and determine the filtering threshold based on the dot product mean and the attention filtering weights.

[0259] In some embodiments, the segmentation module is specifically used to input sample text and preset prompt information into a target large language model, so that the target large language model splits the sample text into multiple sub-sample texts according to the preset prompt information; the segmentation of the sample text into multiple sub-sample texts includes: determining key points in the sample image; inputting the sample image and sample text into an image encoder and a text encoder respectively to obtain corresponding sample image features and sample text features; performing gradient updates based on the sample image features and sample text features to obtain a heatmap corresponding to the sample image; taking the local region with the strongest response in the heatmap as the image sub-region that matches the sample text; determining the key points and key point combinations within the image sub-region, and setting a cropping box based on the key points and key point combinations within the image sub-region to crop the sample image to obtain multiple sub-sample images.

[0260] In some embodiments, the training device 140 for the image retrieval model further includes: an acquisition module, configured to acquire synonyms, near-synonyms, and antonyms corresponding to sample text, and input the synonyms, near-synonyms, and antonyms into a text encoder respectively to obtain corresponding synonym features, near-synonyms features, and antonyms features; an input module, configured to input sample images and sample text into an image encoder and a text encoder respectively to obtain corresponding sample image features and sample text features; the training device 140 for the image retrieval model further includes: a construction module, configured to construct multiple triples based on sample image features, sample text features, synonyms, near-synonyms, and antonyms features; wherein, the negative sample in each triple is an antonyms feature; a determination module, configured to determine the loss function corresponding to each triple, and determine the multi-semantic contrast loss corresponding to the sample pair based on the loss function corresponding to all triples; and an update module, specifically configured to update the image retrieval model based on the sub-semantic contrast loss and multi-semantic contrast loss corresponding to multiple sample pairs.

[0261] In some embodiments, the training apparatus 140 for the image retrieval model further includes: an acquisition module for acquiring negative sample text corresponding to a sample image; an input module for inputting the sample image, sample text, and negative sample text into an image encoder and a text encoder, respectively, to obtain corresponding sample image features, sample text features, and negative sample text features; a determination module for determining a contrastive learning loss based on the sample image features, sample text features, and negative sample text features; and an update module specifically for updating the image retrieval model based on the sub-semantic contrastive loss corresponding to multiple sample pairs and the contrastive learning loss.

[0262] This application also provides an electronic device; please refer to [link / reference]. Figure 15 The electronic device 201 includes: one or more memories 111, one or more processors 112, a communication bus 113, and a communication interface 114. The processors 112 are connected to the memories 111 via the bus 113. The one or more memories 111 are used to store computer program code, which includes computer instructions. When the one or more processors 112 execute the computer instructions, the electronic device 201 performs the training method for the image retrieval model provided in the above embodiments.

[0263] Optionally, the memory 111 may be a non-transitory computer-readable storage medium, such as read-only memory (ROM), random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc., and the embodiments of this application do not impose any limitations on this.

[0264] The processor 112 may be a central processing unit (CPU), a general-purpose processor, a network processor (NP), a digital signal processor (DSP), a microprocessor, a microcontroller, a programmable logic device (PLD), or any combination thereof, and the embodiments of this application do not impose any limitations on this.

[0265] The communication bus 113 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus 113 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 15 It is represented by a single thick line, but this does not mean that there is only one bus or one type of communication bus.

[0266] Communication interface 114 uses any transceiver-like device for communicating with other devices or communication networks, such as control systems, radio access networks (RAN), wireless local area networks (WLAN), etc.

[0267] This application also provides a computer program product comprising one or more instructions, which are stored in the memory of a computer device and executed by a processor to complete the various processes described in the above embodiments.

[0268] This application also provides a computer-readable storage medium, which includes computer-executable instructions. When the computer-executable instructions are executed on the computer, the computer performs the training method for the image retrieval model provided in the above embodiments.

[0269] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0270] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another apparatus, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0271] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0272] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0273] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0274] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A training method for an image retrieval model, characterized in that, include: Multiple sample pairs are obtained, wherein each sample pair includes a sample image and a sample text; For each sample pair, the sample image is divided into multiple sub-sample images, and the sample text is divided into multiple sub-sample texts; The subsample image is input into the image encoder in the image retrieval model to obtain the corresponding subsample image features, and the subsample text is input into the text encoder in the image retrieval model to obtain the corresponding subsample text features; For each sub-sample image feature, based on the correlation between the sub-sample image feature and each sub-sample text feature, positive and negative samples corresponding to the sub-sample image feature are determined from all sub-sample text features; and a sub-image semantic contrast loss is determined based on the sub-sample image feature and the corresponding positive and negative samples; wherein, the correlation between the sub-sample image feature and the corresponding positive sample is higher than the correlation between the sub-sample image feature and the corresponding negative sample; the sub-image semantic contrast loss is used to characterize the ability of the image retrieval model to distinguish between positive and negative samples corresponding to the sub-sample image feature; For each subsample text feature, based on the correlation between the subsample text feature and each subsample image feature, positive and negative samples corresponding to the subsample text feature are determined from all subsample image features; and a subtext semantic contrast loss is determined based on the subsample text feature and the corresponding positive and negative samples; wherein, the correlation between the subsample text feature and the corresponding positive sample is higher than the correlation between the subsample text feature and the corresponding negative sample; the subtext semantic contrast loss is used to characterize the ability of the image retrieval model to distinguish between positive and negative samples corresponding to the subsample text feature; Based on the sub-image semantic contrast loss corresponding to all the sub-sample text features and the sub-text semantic contrast loss corresponding to all the sub-sample image features, determine the sub-semantic contrast loss corresponding to the sample pair; The image retrieval model is updated based on the sub-semantic contrast loss corresponding to multiple sample pairs until the image retrieval model meets the convergence condition, thus obtaining the target image retrieval model.

2. The method according to claim 1, characterized in that, The step of determining the positive and negative samples corresponding to the sub-sample image features from all the sub-sample text features based on the correlation between the sub-sample image features and each sub-sample text feature includes: The attention value vectors of the subsample image features and each subsample text feature are determined to obtain a set of text attention value vectors corresponding to the subsample image features; the value of each attention value vector in the set of text attention value vectors is positively correlated with the subsample image features and the subsample text features corresponding to the attention value vectors. Based on the set of text attention value vectors corresponding to the subsample image features, determine the positive and negative samples corresponding to the subsample image features; The step of determining the positive and negative samples corresponding to the sub-sample text features from all the sub-sample image features based on the correlation between the sub-sample text features and each sub-sample image features includes: Determine the attention value vectors of the subsample text features and each subsample image feature to obtain a set of text attention value vectors corresponding to the subsample text features. The value of each attention value vector in the set of text attention value vectors is positively correlated with the subsample text features and the subsample image features corresponding to the attention value vectors. Based on the set of text attention value vectors corresponding to the subsample text features, the positive and negative samples corresponding to the subsample text features are determined.

3. The method according to claim 2, characterized in that, The step of determining the positive and negative samples corresponding to the text features of the sub-samples based on the set of text attention value vectors corresponding to the sub-sample text features includes: The set of text attention value vectors corresponding to all the subsample text features and the set of image attention value vectors corresponding to all the subsample image features are input into the attention filtering unit. The attention filtering unit suppresses the non-maximum values ​​in the set of text attention value vectors to obtain the filtered set of text attention value vectors corresponding to the subsample text features, and suppresses the non-maximum values ​​in the set of image attention value vectors to obtain the filtered set of image attention value vectors corresponding to the subsample image features. Based on the set of filtered text attention value vectors corresponding to the subsample text features and the set of filtered image attention value vectors corresponding to the subsample image features, the positive and negative samples corresponding to the subsample text features are determined.

4. The method according to claim 3, characterized in that, The process of determining the positive and negative samples corresponding to the subsample text features based on the filtered set of text attention value vectors corresponding to the subsample text features and the filtered set of image attention value vectors corresponding to the subsample image features includes: The maximum attention value in the set of filtered text attention value vectors corresponding to each of the subsample text features is determined to obtain the set of maximum text attention values; each maximum text attention value in the set of maximum text attention values ​​corresponds to a subsample text feature. The maximum attention value in the filtered image attention value vector set corresponding to each of the subsample image features is determined to obtain the image maximum attention value set; each image maximum attention value in the image maximum attention value set corresponds to a subsample image feature; For the subsample text feature, the maximum attention value of the text corresponding to the subsample text feature is multiplied by the maximum attention value of each image in the set of maximum attention values ​​of images to obtain the target set corresponding to the subsample text feature; A first vector and a second vector are selected from the target set, wherein the extreme value of the first vector is greater than the extreme value of the second vector; The subsample image features corresponding to the first vector are used as the positive samples corresponding to the subsample text features; The subsample image features corresponding to the second vector are used as the negative samples corresponding to the subsample text features.

5. The method according to claim 3, characterized in that, The attention filtering unit includes multiple multi-head attention modules; The step of inputting the set of text attention value vectors corresponding to all the subsample text features and the set of image attention value vectors corresponding to all the subsample image features into the attention filtering unit, and suppressing the non-maximum values ​​in the set of image attention value vectors through the attention filtering unit to obtain the filtered set of text attention value vectors corresponding to the subsample text features and the filtered set of image attention value vectors corresponding to the subsample image features, includes: The set of text attention value vectors corresponding to all the subsample text features and the set of image attention value vectors corresponding to all the subsample image features are input into the first multi-head attention module of the attention filtering unit. The input of other multi-head attention modules is the output of the previous multi-head attention module. Each of the multi-head attention modules performs the following steps: Determine the maximum attention value in the set of text attention value vectors corresponding to the text features of each subsample to obtain the first maximum attention value set; Determine the maximum attention value in the set of image attention value vectors corresponding to each of the subsample image features to obtain the second maximum attention value set; The filtering threshold is determined based on the first set of maximum attention values ​​and the second set of maximum attention values; In the set of text attention value vectors corresponding to all the subsample text features and the set of image attention value vectors corresponding to all the subsample image features, attention value vectors that are less than or equal to the filtering threshold are set to 0; Output the updated set of text attention value vectors corresponding to all the subsample text features, and the updated set of image attention value vectors corresponding to all the subsample image features.

6. The method according to claim 5, characterized in that, The step of determining the filtering threshold based on the first set of maximum attention values ​​and the second set of maximum attention values ​​includes: The first set of maximum attention values ​​and the second set of maximum attention values ​​are input into a weighted neural network. The correlation strength between the first set of maximum attention values ​​and the second set of maximum attention values ​​is determined by the weighted neural network to obtain attention filtering weights. Multiply the elements in the first maximum attention value set and the elements in the second maximum attention value set to obtain multiple element-wise dot products; Calculate the average of the dot products of all the elements to obtain the mean dot product; The filtering threshold is determined based on the mean of the dot product and the attention filtering weight.

7. The method according to claim 1, characterized in that, The step of dividing the sample image into multiple sub-sample images includes: The sample text and preset prompt information are input into the target large language model, so that the target large language model splits the sample text into the multiple sub-sample texts according to the preset prompt information; The process of dividing the sample text into multiple sub-sample texts includes: Identify the key points in the sample image; The sample image and the sample text are input into the image encoder and the text encoder, respectively, to obtain the corresponding sample image features and sample text features; Gradient updates are performed based on the sample image features and the sample text features to obtain the heatmap corresponding to the sample image; The local region with the strongest response in the heatmap is taken as the image sub-region that matches the sample text. Key points and key point combinations within the image sub-region are determined, and a cropping frame is set based on the key points and key point combinations within the image sub-region to crop the sample image, thereby obtaining the multiple sub-sample images.

8. The method according to claim 1, characterized in that, The method further includes: Obtain the synonyms, near-synonyms, and antonyms corresponding to the sample text, and input the synonyms, near-synonyms, and antonyms into the text encoder respectively to obtain the corresponding synonym features, near-synonyms features, and antonyms features; The sample image and the sample text are input into the image encoder and the text encoder, respectively, to obtain the corresponding sample image features and sample text features; Multiple triples are constructed based on the sample image features, the sample text features, the synonym text features, the near-synonymous text features, and the antonymous text features; wherein, the negative sample in each triple is the antonymous text feature; Determine the loss function corresponding to each triplet, and determine the multi-semantic contrast loss corresponding to the sample pair based on the loss functions corresponding to all triplets; The step of updating the image retrieval model based on the sub-semantic contrast loss corresponding to multiple sample pairs includes: The image retrieval model is updated based on the sub-semantic contrast loss and multi-semantic contrast loss corresponding to multiple sample pairs.

9. The method according to claim 1, characterized in that, The method further includes: Obtain the negative sample text corresponding to the sample image; The sample image, the sample text, and the negative sample text are respectively input into the image encoder and the text encoder to obtain the corresponding sample image features, sample text features, and negative sample text features; The contrastive learning loss is determined based on the sample image features, the sample text features, and the negative sample text features; the contrastive learning loss is used to characterize the image retrieval model's ability to distinguish between positive and negative samples corresponding to the sample image features. The step of updating the image retrieval model based on the sub-semantic contrast loss corresponding to multiple sample pairs includes: The image retrieval model is updated based on the sub-semantic contrast loss and contrastive learning loss corresponding to multiple sample pairs.

10. An image retrieval method, characterized in that, include: Get the text to be searched; The text to be retrieved is input into the target image retrieval model trained by the image retrieval model training method as described in any one of claims 1-9, and the target image retrieval model is used to retrieve a target image that matches the text to be retrieved from the image database.

11. A training device for an image retrieval model, characterized in that, include: An acquisition module is used to acquire multiple sample pairs, wherein each sample pair includes a sample image and a sample text; The segmentation module is used to segment the sample image into multiple sub-sample images and the sample text into multiple sub-sample text for each sample pair; The input module is used to input the subsample image into the image encoder in the image retrieval model to obtain the corresponding subsample image features, and to input the subsample text into the text encoder in the image retrieval model to obtain the corresponding subsample text features; A determination module is configured to, for each sub-sample image feature, determine the positive and negative samples corresponding to the sub-sample image feature from all the sub-sample text features based on the correlation between the sub-sample image feature and each sub-sample text feature; and determine the sub-image semantic contrast loss based on the sub-sample image feature and the corresponding positive and negative samples; wherein the correlation between the sub-sample image feature and the corresponding positive sample is higher than the correlation between the sub-sample image feature and the corresponding negative sample; the sub-image semantic contrast loss is used to characterize the ability of the image retrieval model to distinguish the positive and negative samples corresponding to the sub-sample image feature; The determining module is further configured to, for each sub-sample text feature, determine the positive and negative samples corresponding to the sub-sample text feature from all the sub-sample image features based on the correlation between the sub-sample text feature and each sub-sample image feature; and determine the sub-text semantic contrast loss based on the sub-sample text feature and the positive and negative samples corresponding to the sub-sample text feature; wherein, the correlation between the sub-sample text feature and the positive sample corresponding to the sub-sample text feature is higher than the correlation between the sub-sample text feature and the negative sample corresponding to the sub-sample text feature; the sub-text semantic contrast loss is used to characterize the ability of the image retrieval model to distinguish the positive and negative samples corresponding to the sub-sample text feature; The determining module is further configured to determine the sub-semantic contrast loss corresponding to the sample pair based on the sub-image semantic contrast loss corresponding to all the sub-sample text features and the sub-text semantic contrast loss corresponding to all the sub-sample image features; The update module is used to update the image retrieval model based on the sub-semantic contrast loss corresponding to multiple sample pairs until the image retrieval model meets the convergence condition and the target image retrieval model is obtained.

12. An electronic device, characterized in that, include: One or more processors; one or more memories; Wherein, one or more of the memories are used to store computer program code, the computer program code including computer instructions, which, when executed by one or more of the processors, cause the electronic device to perform the training method of the image retrieval model as described in any one of claims 1 to 9, or to perform the image retrieval method as described in claim 10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions that, when executed on a computer, cause the computer to perform the training method of the image retrieval model as described in any one of claims 1 to 9, or to perform the image retrieval method as described in claim 10.

Images