Fine-tuning of text retrieval image model and image retrieval method and device

Abstract

The application discloses a fine tuning method and device of a text retrieval image model and an image retrieval method. The method comprises the following steps: obtaining error retrieval records of a text retrieval image model to be adjusted; establishing a positive sample data set according to retrieval texts and corresponding missed images, and establishing a negative sample data set according to misdetected images and corresponding image description texts; determining a plurality of first type samples from the positive sample data set, and determining corresponding second type samples from the positive sample data set and the negative sample data set according to the first type samples; determining a single batch processing sample according to each first type sample and the corresponding second type sample; fine tuning training the text retrieval image model to be adjusted according to a plurality of single batch processing samples, and obtaining a target text retrieval image model after fine tuning. The method can fine tune and train a model using a small amount of samples, quickly improve the image retrieval effect of a target retrieval text in a target scene, and solve the problem of missed detection and misdetection in the target scene.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method and device for fine-tuning a text retrieval image model and for image retrieval. Background Technology

[0002] Image retrieval is one of the important tasks in the field of computer vision. In recent years, many image retrieval algorithms have been proposed, among which the CLIP (Contrastive Language-Image Pre-training) model has attracted widespread attention due to its strong generalization and simple and efficient characteristics. The CLIP model extracts features from all images in the database using a pre-trained image encoder and extracts features from the user-input retrieval text using a text encoder. Then, it calculates the similarity between the text features and the image features, and finally ranks the images according to the similarity to present the retrieval results.

[0003] CLIP models are typically trained on large datasets of image-text pairs. Through contrastive learning with massive amounts of data and extremely large batch sizes, the model learns general text and visual representations and aligns text and visual features in the same spatial domain. CLIP batch sizes are often in the thousands or even tens of thousands; if the training data is limited, it's impossible to effectively train a model with good metrics. Furthermore, because contrastive training is required, the image-text pairs within a batch are usually different; otherwise, retrieval performance in general scenarios cannot be guaranteed.

[0004] The original CLIP model extracts similarity features between images and text on the same dimension from a large amount of image-text pair data through contrastive learning. However, due to the limitations of the model's capabilities, retrieval errors occur in practical applications, meaning that the top-ranked results are not necessarily the correct ones. If a small amount of data from the target scene with retrieval errors is added directly to the large amount of original training data for fine-tuning, this small portion of data will be overwhelmed by the large original sample size, failing to effectively improve the retrieval performance for the target scene. Conversely, if only target scene data is used for training, it will lead to inconsistent retrieval performance in other general scenarios. Summary of the Invention

[0005] This invention provides a text retrieval image model fine-tuning method and device for image retrieval, in order to solve the problem of text retrieval models failing to retrieve images in certain scenarios after training in general scenarios.

[0006] According to one aspect of the present invention, a method for fine-tuning a text retrieval image model is provided, comprising:

[0007] Obtain the erroneous retrieval records of the text retrieval image model to be adjusted; wherein, the erroneous retrieval records include the retrieval text and the corresponding missed and falsely detected images;

[0008] A positive sample dataset is established based on the retrieved text and the corresponding missed images, and a negative sample dataset is established based on the falsely detected images and the corresponding image description text.

[0009] Multiple first-type samples are determined from the positive sample dataset, and corresponding second-type samples are determined from the positive sample dataset and the negative sample dataset based on the first-type samples. A single batch processing sample is determined based on each first-type sample and the corresponding second-type sample.

[0010] The text retrieval image model to be adjusted is fine-tuned and trained based on multiple batch processing samples to obtain the fine-tuned target text retrieval image model.

[0011] According to one aspect of the present invention, an image retrieval method is provided, comprising:

[0012] The target retrieval text is input into the target text retrieval image model, and the output of the target text retrieval image model is the image retrieval result corresponding to the target retrieval text determined from the image library; wherein, the target text retrieval image model is trained by the fine-tuning method of the text retrieval image model according to any embodiment of the present invention.

[0013] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0014] At least one processor; and

[0015] A memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the text retrieval image model fine-tuning method or image retrieval method according to any embodiment of the present invention.

[0017] The technical solution of this invention uses multiple similar positive sample data and corresponding negative sample data in a single batch of training samples during model fine-tuning training. This enables the model to be fine-tuned using a small number of samples, thereby rapidly improving the image retrieval effect of target text in the target scene and solving the problems of missed detection and false detection in the target scene.

[0018] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart of a method for fine-tuning a text retrieval image model according to an embodiment of the present invention;

[0021] Figure 2 This is a flowchart of another method for fine-tuning a text retrieval image model according to an embodiment of the present invention;

[0022] Figure 3 This is a schematic diagram of fine-tuning training of a text retrieval image model provided in an embodiment of the present invention;

[0023] Figure 4 This is a schematic diagram of the structure of a fine-tuning device for a text retrieval image model according to an embodiment of the present invention;

[0024] Figure 5 This is a schematic diagram of the structure of an electronic device that implements the text retrieval image model fine-tuning method of the present invention. Detailed Implementation

[0025] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0026] It should be noted that the terms "candidate," "target," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0027] Figure 1 This invention provides a flowchart of a method for fine-tuning a text retrieval image model, applicable to fine-tuning training of a CLIP model. This method can be executed by a text retrieval image model fine-tuning device, which can be implemented in hardware and / or software and configured in a server with computing power. Figure 1 As shown, the method includes:

[0028] S110. Obtain the erroneous retrieval records of the text retrieval image model to be adjusted.

[0029] The erroneous retrieval records include the retrieved text and the corresponding missed and false positive images. A text retrieval image model refers to a deep learning model that uses contrastive learning techniques to determine matching images in an image database based on text including image description information. For example, a text retrieval image model could be the CLIP model. CLIP extracts features from all images in the image database using a pre-trained image encoder and extracts features from the user-input text using a text encoder. It then calculates the similarity between the text features and the image features, and finally ranks the images according to the similarity to present the retrieval results.

[0030] Although the CLIP model performs well in generalization, it still exhibits erroneous retrieval results in some scenarios during practical applications. The text retrieval image model to be tuned refers to a model that exhibits false positives and false negatives in image retrieval within the target scenario. For example, the text retrieval image model to be tuned refers to a CLIP model initially trained on a large amount of image-text data. The target scenario refers to the situation where the initially trained CLIP model exhibits false positives and false negatives during use, without limiting the specific content of the target scenario.

[0031] Error retrieval records refer to the retrieval records of missed and false detections that occurred during the initial training of the text retrieval image model. Specifically, when a user inputs search text into the model to be adjusted, the resulting image output does not match the search text. After determining that the image retrieval result output by the model does not match the input search text, the model outputs a false detection image. Simultaneously, images in the image library that match the search text but are not output by the model are considered missed detection images. An error retrieval record is constructed based on the search text, the corresponding missed detection images, and the false detection images. Multiple error retrieval records generated by the model during the retrieval process are collected. The process of identifying missed detection images that match real images in the image library can be achieved using other image comparison techniques or manually; the implementation method is not limited here.

[0032] S120. Establish a positive sample dataset based on the retrieved text and the corresponding missed images, and establish a negative sample dataset based on the falsely detected images and the corresponding image description text.

[0033] Since the missed images are the correct matching images corresponding to the search text, a positive sample dataset is built based on the search text and missed images in each erroneous search record, and the positive sample dataset includes the correspondence between the search text and the missed images.

[0034] Since the false positive images are incorrectly matched images corresponding to the search text, the search text and the false positive images do not match. Therefore, the corresponding image description text is determined based on the image features of the false positive images; that is, the image description text is the text content that correctly describes the false positive images. The determination of the corresponding image description text for the false positive images can be achieved using image recognition technology, such as using a large language model to provide text descriptions of the false positive images. The method for determining the image description text is not limited here. The corresponding image description text is determined based on the false positive images in each erroneous search record, and a negative sample dataset is built based on the false positive images and image description texts. The negative sample dataset includes the correspondence between the false positive images and the image description texts.

[0035] In one feasible embodiment, a positive sample dataset is established based on the retrieved text and the corresponding missed images, including:

[0036] The search text is expanded using a large language model to obtain the expanded search text. Based on the expansion relationship, a correspondence is established between the expanded search text and the missed images.

[0037] A positive sample dataset is built based on the search text, the expanded search text, and the corresponding missed images.

[0038] The reasons why the text retrieval image model to be adjusted may fail to match the retrieved text may include unclear semantic description of the retrieved text. Therefore, in order to improve the generalization of the text retrieval image model in recognizing the retrieved text input by the user, when fine-tuning the text retrieval image model to be adjusted in the target scene, the semantics of the retrieved text in the target scene are expanded to improve the generalization of the retrieved text.

[0039] Specifically, a large language model is used to expand each search text in the erroneous search records with synonyms. This involves changing the descriptive form of the search text without altering its semantic information, resulting in expanded search text. Each search text undergoes at least one synonym expansion. A correspondence is established between the expanded search texts and the missed images based on the expansion relationships. For example, expanding the first search text with synonyms yields the first expanded search text. Since the first expanded search text shares the same semantic information as the first search text, it also corresponds to the missed image.

[0040] A positive sample dataset is built based on the search text, missed images, and expanded search text of each erroneous search record. The positive sample dataset includes the correspondence between search text and missed images, as well as the correspondence between expanded search text and missed images.

[0041] For example, when using a large language model for synonym expansion, the prompt word could be "Please rewrite the following sentence randomly, answer only with one rewritten sentence, do not explain, and do not change the meaning of the sentence." Taking "a woman wearing red clothes, black pants, and carrying a yellow shoulder bag" as the original search text, the expanded search text examples for synonym expansion could be: a woman wearing black pants, red clothes, and carrying a yellow shoulder bag, or a woman carrying a yellow shoulder bag, wearing red clothes, and black pants.

[0042] For all the missed positive sample datasets, the corresponding search texts are augmented with synonyms using a large language model to randomly modify and enhance them, ensuring the generalization of the model's text input.

[0043] S130. Determine multiple first-type samples from the positive sample dataset, and determine corresponding second-type samples from the positive sample dataset and negative sample dataset based on the first-type samples, and determine single-batch processing samples based on each first-type sample and the corresponding second-type sample.

[0044] Since text retrieval image models use text information to retrieve image information, it involves matching two different types of information. During fine-tuning training, the model is trained by comparing different types of data in both the positive and negative sample datasets. Because the model uses batch training during fine-tuning, parameters are adjusted once per batch of samples. Each batch of samples constitutes a single batch, and the parameters are adjusted multiple times across multiple batches of samples until the model training is complete.

[0045] Specifically, from the positive sample dataset, one type of data is determined as the first type of sample. Based on the first type of sample, corresponding data from the positive sample dataset and data from the negative sample dataset that differ from the first type are determined as the second type of sample. For example, text is designated as the first type and images as the second type. The search text from the positive sample dataset is used as the first type of sample, the missed images from the positive sample dataset corresponding to the search text, and multiple randomly selected false positive images from the negative sample dataset are used as the second type of sample corresponding to the first type of sample. Alternatively, images are designated as the first type and text as the second type. Missed images from the positive sample dataset are used as the first type of sample, the search text from the positive sample dataset corresponding to the missed images, and multiple randomly selected image description texts from the negative sample dataset are used as the second type of sample corresponding to the first type of sample.

[0046] The batch size for model training is B, meaning that parameters are adjusted once per batch of samples during training. The number of positive samples in a batch is B. During each batch training, B first-type samples are determined from the positive sample dataset, and multiple corresponding second-type samples are determined based on these B first-type samples. These serve as training samples for that batch. After training the samples in that batch is complete, the next batch of samples is determined, and this process continues until the conditions for model training are met. For example, during each batch training, B search texts are randomly selected from the positive sample data as first-type samples. For each search text, corresponding false negative images are determined from the positive sample dataset, and K false positive images are randomly selected from the negative sample dataset as their corresponding second-type samples. That is, each batch includes B first-type samples, and each first-type sample corresponds to K+1 second-type samples.

[0047] S140. Fine-tune the text retrieval image model to be adjusted based on multiple single batch processing samples to obtain the fine-tuned target text retrieval image model.

[0048] The loss value is determined based on the similarity calculation results between each first-type sample and the corresponding second-type sample in multiple batches of samples. The text retrieval image model to be adjusted is fine-tuned according to the changes in the loss value until the loss converges and the fine-tuning training is completed, resulting in the fine-tuned target text retrieval image model.

[0049] For example, if the first type is text and the second type is image, then the similarity between each retrieved text and the corresponding second type sample in each batch is determined. The text retrieval image model to be adjusted is fine-tuned based on the cross-entropy loss of the similarity corresponding to all retrieved texts to obtain the fine-tuned target text retrieval image model.

[0050] The technical solution of this embodiment uses multiple similar positive sample data and corresponding negative sample data in a single batch of training samples during model fine-tuning training. This enables the model to be fine-tuned using a small number of samples, thereby rapidly improving the image retrieval effect of target text in the target scene and solving the problems of missed detection and false detection in the target scene.

[0051] Figure 2 This is a flowchart illustrating another method for fine-tuning a text retrieval image model provided by an embodiment of the present invention. This embodiment further refines the construction of single-batch processing samples and the fine-tuning training process in the above embodiments. Figure 2 As shown, the method includes:

[0052] S210. Obtain the erroneous retrieval records of the text retrieval image model to be adjusted.

[0053] The erroneous retrieval records include the retrieval text and the corresponding missed and falsely detected images.

[0054] S220. Establish a positive sample dataset based on the retrieved text and the corresponding missed images, and establish a negative sample dataset based on the falsely detected images and the corresponding image description text.

[0055] S230. Take multiple first search texts and / or first missed images in the positive sample dataset as first type samples.

[0056] Multiple first search texts from the positive sample dataset can be used as first-type samples. For example, with a batch size of B, B search texts are randomly selected from the positive sample dataset as first search texts, i.e., first-type samples. Alternatively, multiple first missed images from the positive sample dataset can be used as first-type samples. For example, with a batch size of B, B missed images are randomly selected from the positive sample dataset as first missed images, i.e., first-type samples. Alternatively, multiple first missed images from the positive sample dataset and multiple first search texts corresponding to the first missed images can be used as first-type samples. For example, B missed images are randomly selected from the positive sample dataset as first missed images, and the search text corresponding to the first missed images is determined from the positive sample dataset as the first search text. The first search text and the first missed images constitute the first-type samples.

[0057] S240. Based on the first type of sample, determine the corresponding positive sample data from the positive sample dataset as the second type of first sample.

[0058] If the first type of sample consists of multiple first search texts, then the second type of first sample consists of the missed images in the positive sample dataset corresponding to each first search text; if the first type of sample consists of multiple first missed images, then the second type of first sample consists of the search text in the positive sample dataset corresponding to each first missed image; if the first type of sample includes multiple first search texts and multiple first missed images, then the second type of first sample consists of the missed images in the positive sample dataset corresponding to each first search text and the search text corresponding to each first missed image.

[0059] In one feasible embodiment, the first type of sample includes a plurality of first missed detection images; S240 includes:

[0060] The search text corresponding to the first missed image is determined to be the first text.

[0061] Multiple non-synonymous text replacements are performed on the first text to obtain multiple second texts;

[0062] The first sample of the second type is formed based on the first text and the second text.

[0063] If the first type of sample includes multiple first missed detection images, then the second type of first sample includes at least the search text corresponding to each first missed detection image in the positive sample dataset. This search text is taken as the first text. In this embodiment, the first text is replaced with non-synonymous text to obtain multiple second texts. The number of second texts can be determined according to the actual scenario. For example, the number of second texts can be on the order of ten. This embodiment does not limit the number of second texts. The first text and the second text together constitute the second type of first sample. Similarly, if the first type of sample includes multiple first missed detection images and multiple corresponding first search texts, the first search text is the first text. Multiple non-synonymous text replacements are performed on the first text to obtain multiple second texts. The second type of first sample is constituted based on the first text, the second texts, and the missed detection images corresponding to the first search texts.

[0064] For example, a noun database can be pre-established, and replacement nouns can be randomly selected from the database to perform multiple non-synonymous text replacements on the first text; or a large language model can be used to perform multiple non-synonymous text replacements on the first text. Taking the first text as "a woman wearing a red dress, black pants, and carrying a yellow shoulder bag" as an example, the second text after non-synonymous text replacements could be: a woman wearing a black dress, red pants, and carrying a white shoulder bag; a woman wearing a red dress, yellow pants, and carrying a black shoulder bag, etc.

[0065] This embodiment improves the model's understanding of the input text by adding a second text that is a non-synonymous replacement to the first sample of the second type. This effectively enhances the text retrieval image model's ability to recognize difficult texts and improves the model's accuracy in understanding the correct attributes of objects with multi-attribute texts. It also increases the training amount of negative samples.

[0066] S250. Based on the type of the first type of sample, randomly select multiple negative sample data of different types from the negative sample dataset as the second type of second sample. The second type of first sample and the second type of second sample constitute the second type of sample corresponding to the first type of sample.

[0067] If the first type of sample consists of multiple first search texts, then the second type of sample consists of multiple false positive images randomly selected from the negative sample dataset corresponding to each first search text; if the first type of sample consists of multiple false negative images, then the second type of sample consists of multiple image description texts randomly selected from the negative sample dataset corresponding to each false negative image; if the first type of sample includes multiple first search texts and multiple false negative images, then the second type of sample consists of multiple false positive images randomly selected from the negative sample dataset and their corresponding image description texts. The second type of first sample and the second type of second sample constitute the second type of sample corresponding to the first type of sample.

[0068] Specifically, if the first type of sample consists of multiple first search texts, then the second type of sample consists of the missed images corresponding to each first search text in the positive sample dataset, and multiple false positive images randomly selected from the negative sample dataset corresponding to each first search text. If the first type of sample consists of multiple first missed images, then the second type of sample consists of the search text corresponding to each first missed image in the positive sample dataset, and multiple image description text randomly selected from the negative sample dataset corresponding to each first search text. If the first type of sample includes multiple first search texts and multiple first missed images, then the second type of sample includes the missed images corresponding to each first search text in the positive sample dataset corresponding to the first search text, and multiple false positive images randomly selected from the negative sample dataset corresponding to each first search text; as well as the search text corresponding to each first missed image in the positive sample dataset corresponding to the first missed image, and multiple image description text randomly selected from the negative sample dataset corresponding to each first search text.

[0069] S260. Determine the batch processing samples based on each first type of sample and the corresponding second type of sample.

[0070] S270. Determine the cross-entropy loss parameter for each first-type sample based on the cross-entropy loss of the similarity parameter between each first-type sample and its corresponding second-type sample in each batch of samples.

[0071] If the first type of sample consists of multiple first search texts, then the image samples corresponding to each first search text in the second type of sample are determined. These image samples are the missed detection images corresponding to each first search text, and multiple false detection images randomly selected from the negative sample dataset corresponding to each first search text. That is, the image samples corresponding to each first search text are the corresponding missed detection images and multiple false detection images. The first similarity between the text features and image features between each first search text and the corresponding missed detection images and multiple false detection images is determined. The cross-entropy loss parameter of the first search text is determined based on the cross-entropy loss of the multiple first similarities.

[0072] If the first type of sample consists of multiple first missed detection images, then the text sample corresponding to each first missed detection image in the second type of sample is determined. This text sample is the retrieval text corresponding to each first missed detection image, and multiple image description texts randomly selected from the negative sample dataset corresponding to each first missed detection image. That is, the text sample corresponding to each first missed detection image is the corresponding retrieval text and multiple image description texts. The second similarity between the image features and text features between each first missed detection image and the corresponding retrieval text and multiple image description texts is determined. The cross-entropy loss parameter of the first missed detection image is determined based on the cross-entropy loss of multiple second similarities.

[0073] If the first type of sample consists of multiple first search texts and multiple first missed detection images, then the corresponding cross-entropy loss parameters are determined for each first search text and each first missed detection image in sequence according to the above method.

[0074] In one feasible embodiment, S270 includes:

[0075] Determine the similarity parameters between the first type of sample and each corresponding second type of sample;

[0076] If the similarity parameter is greater than the preset similarity threshold, then the similarity parameter between the first type of sample and the corresponding second type of sample is determined to be the preset similarity threshold.

[0077] The cross-entropy loss parameter for each first-type sample is determined based on the cross-entropy loss of the product of the similarity parameter and the dynamic temperature coefficient to be adjusted; wherein, the value of the dynamic temperature coefficient changes dynamically during the fine-tuning training of the text retrieval image model to be adjusted.

[0078] Since the similarity between positive and negative samples is generally high during training, in order to maintain the consistency of similarity, a preset similarity threshold is set by setting the maximum similarity and a dynamic temperature coefficient is set to keep the similarity of positive samples within the target domain and reduce the similarity of negative samples accordingly, so that the similarity distribution identified by the model after fine-tuning training is consistent with that before training.

[0079] Specifically, when determining the similarity parameter between each first-type sample and each corresponding second-type sample, it is determined whether the similarity parameter is greater than a preset similarity threshold. If it is greater, the preset similarity threshold is directly used as the similarity parameter between the first-type sample and the second-type sample; that is, the upper limit of the similarity between the first-type sample and the second-type sample is limited.

[0080] The similarity parameter after being adjusted by a preset similarity threshold is multiplied by the dynamic temperature coefficient to obtain the dynamic similarity parameter. The cross-entropy loss of the dynamic similarity parameter between the first type of sample and the corresponding second type of sample is used as the cross-entropy loss parameter of the first type of sample.

[0081] Since the samples used in the fine-tuning process are from the target scene, the similarity between positive and negative samples is generally high. However, the samples within each batch of the large batch used in the initial training of the text retrieval image model generally have low similarity. Therefore, in order to ensure that the similarity calculation of the fine-tuned model is limited to a controllable range and to maintain the consistency of the image retrieval results of the fine-tuned model in other general scenarios, a preset similarity threshold and a dynamic temperature coefficient are used to adjust the similarity parameters, which effectively improves the image retrieval effect of specific search terms in the target scene.

[0082] S280. Determine the average cross-entropy loss based on the cross-entropy loss parameters of all first-type samples in a single batch of samples, and fine-tune the text retrieval image model to be adjusted based on the average cross-entropy loss to obtain the fine-tuned target text retrieval image model.

[0083] After determining the cross-entropy loss parameter for each first-type sample in a single batch, the average value of the cross-entropy loss parameters of all first-type samples is used as the average cross-entropy loss. Based on the change in the average cross-entropy loss, the text retrieval image model to be adjusted is fine-tuned until the average cross-entropy loss converges and the fine-tuning training is completed, thus obtaining the fine-tuned target text retrieval image model.

[0084] Specifically, if the first type of sample consists of multiple first search texts The second type of sample consists of multiple image samples, and each one is determined. Corresponding image samples Determine each and each corresponding Similarity between , , , No. The cross-entropy loss parameter for each first-type sample is: The average cross-entropy loss is ,in, This indicates a preset similarity threshold. Indicates the dynamic temperature coefficient. This represents the similarity loss from text to image.

[0085] If the first type of sample consists of multiple images that were initially missed. The second type of sample consists of multiple text samples, and each... The corresponding text samples include multiple image description texts randomly selected from the negative sample dataset and the retrieval text corresponding to the first missed image. and the second text obtained by non-synonymous substitution Determine each Similarity between and each corresponding text sample , No. The cross-entropy loss of each first-type sample is described as follows: The average cross-entropy loss is ,in, This indicates a preset similarity threshold. Indicates the dynamic temperature coefficient. This represents the similarity loss from image to text.

[0086] In one feasible embodiment, the first type of sample includes multiple first search texts and corresponding first missed detection images;

[0087] The average cross-entropy loss is determined based on the cross-entropy loss parameters of all Type I samples in a single batch of samples, including:

[0088] The first average cross-entropy loss is determined based on the average value of the cross-entropy loss parameter of the first search text in the first type of samples;

[0089] The second average cross-entropy loss is determined based on the average value of the cross-entropy loss parameter of the first missed images in the first type of samples;

[0090] The average cross-entropy loss is determined by the sum of the first average cross-entropy loss and the second average cross-entropy loss.

[0091] If the first type of sample includes multiple first retrieval texts and corresponding first missed detection images, then the calculation is performed separately for different types of first type samples. The calculation method is the same as in the above embodiment and will not be repeated here. The first average cross-entropy loss represents the similarity loss from text to image, and the second average cross-entropy loss represents the similarity loss from image to text. The sum of the first average cross-entropy loss and the second average cross-entropy loss is used as the final training average cross-entropy loss. For example, the average cross-entropy loss... .

[0092] By incorporating the cross-entropy loss of a text against multiple images and the cross-entropy loss of an image against multiple texts in the model fine-tuning, and by using positive samples in a training batch that are all similar positive samples in the target scene, the fine-tuned model can improve the retrieval performance in the target scene.

[0093] In one feasible embodiment, the method further includes, prior to fine-tuning the text retrieval image model to be tuned based on the average cross-entropy loss:

[0094] The search text in the first type of sample is identified as the first search text;

[0095] Multiple other search texts are randomly selected from the positive sample dataset as the second search texts corresponding to the first search text;

[0096] The divergence loss is determined based on the KL divergence between each first search text and its corresponding second search text.

[0097] The text retrieval image model to be tuned is trained by summing the average cross-entropy loss and divergence loss.

[0098] For positive sample text in the first type of samples To maintain the generality of the retrieved text, S other positive sample texts were randomly sampled from the positive sample dataset. ,Sure and The KL divergence loss is added to the total loss of the model fine-tuning to ensure consistent feature distributions among positive sample texts. Specifically, the divergence loss is... The final total loss was .

[0099] For example, the fine-tuned CLIP model can effectively solve the problems of missed detections and false detections in the target scene. When retrieving images from text, it can adjust the image-text similarity of missed images to the expected range. At the same time, it can maintain the good generalization of the CLIP model without adding a large amount of general training materials.

[0100] like Figure 3 The diagram illustrates the fine-tuning training of the text retrieval image model. Multiple first-type search texts (yellow P1, P2, ...) and / or first-type false negative images (yellow I1, I2, ...) are determined from the positive sample dataset as first-type samples. If the first-type samples consist of multiple first-type search texts (yellow P1, P2, ...), then the second-type first-type samples are determined as the first-type false negative images (yellow I1, I2, ...) corresponding to the first-type search texts. Optionally, the first-type search texts may also include expanded search texts augmented using a large language model. Similarly, if the first-type samples consist of multiple false negative images (yellow I1, I2, ...), then the second-type first-type samples are determined as the first-type search texts (yellow P1, P2, ...). Simultaneously, multiple negative sample data of different types are randomly selected from the negative sample dataset as second-type second-type samples. That is, if the first-type samples consist of multiple first-type search texts (yellow P1, P2, ...), then the second-type second-type samples are multiple false positive images (gray I1, I2, ... I) randomly selected from the negative sample dataset. n ), gray I1, I2, ... I n Together with the yellow I1, I2, ..., they constitute the second type of samples corresponding to the first type of samples. The cross-entropy loss parameter for each first type of sample is determined based on the cross-entropy loss of the similarity parameter between each first type of sample and its corresponding second type of sample.

[0101] Furthermore, if the first type of sample consists of multiple missed images (yellow I1, I2, ...), then the second type of first sample (yellow P1, P2, ...) undergoes multiple non-synonymous text replacements to obtain the second text (green P1, P2, ... P). mThe second type of sample consists of the first sample of the second type, the second sample of the second type, and the second text. Further, the first sample of the second type in each second type of sample is determined as the first search text, and multiple other search texts are randomly selected from the positive sample dataset as the second search texts corresponding to the first search text. That is, if yellow P1 is the first search text, then yellow P2, ..., P... i If yellow P2 is the first search text, then yellow P1, ..., P2 are the second search text. i For the second search text, the divergence loss is determined based on the KL divergence between each first search text and the corresponding second search text. The text retrieval image model to be adjusted is then fine-tuned based on the sum of the average cross-entropy loss and the divergence loss.

[0102] The technical solution of this embodiment sets the model loss to cross-entropy loss between one text and multiple images and between one image and multiple text during the fine-tuning training process of the text retrieval image model. Furthermore, by adding technical features such as dynamic temperature coefficient, the similarity is limited to the expected range. This effectively optimizes the retrieval effect in the target scene while ensuring the consistency of retrieval results in general scenarios.

[0103] This invention provides an image retrieval method, which includes:

[0104] The target search text is input into the target text retrieval image model, and the output of the target text retrieval image model is the image retrieval result that is determined from the image library to correspond to the target search text.

[0105] The target text retrieval image model is a fine-tuned text retrieval image model obtained from any of the above embodiments. This fine-tuned target text retrieval image model can effectively solve the problems of missed detections and false detections in the target scenario. When retrieving text images, it can adjust the image-text similarity of missed images to the expected range, thereby improving the accuracy of the retrieval results.

[0106] Specifically, the fine-tuned target text retrieval image model is used to extract features from all images in the image library. The target retrieval text is input, and the target text retrieval image model extracts the target text features from the target retrieval text. The similarity between the target text features and each image feature in the image library is calculated, and the image retrieval result is determined based on the similarity calculation result.

[0107] Figure 4 This is a schematic diagram of a fine-tuning device for a text retrieval image model provided in an embodiment of the present invention. Figure 4 As shown, the device includes:

[0108] The retrieval record acquisition module 310 is used to acquire the erroneous retrieval records of the text retrieval image model to be adjusted; wherein, the erroneous retrieval records include the retrieval text and the corresponding missed and false detection images;

[0109] The dataset creation module 320 is used to create a positive sample dataset based on the search text and the corresponding missed images, and to create a negative sample dataset based on the falsely detected images and the corresponding image description text.

[0110] The batch processing sample determination module 330 is used to determine multiple first type samples from the positive sample dataset, and to determine corresponding second type samples from the positive sample dataset and the negative sample dataset based on the first type samples, and to determine a single batch processing sample based on each first type sample and the corresponding second type sample;

[0111] The model fine-tuning training module 340 is used to fine-tune the text retrieval image model to be adjusted based on multiple single batch processing samples to obtain the fine-tuned target text retrieval image model.

[0112] The technical solution of this embodiment uses multiple similar positive sample data and corresponding negative sample data in a single batch of training samples during model fine-tuning training. This enables the model to be fine-tuned using a small number of samples, thereby rapidly improving the image retrieval effect of target text in the target scene and solving the problems of missed detection and false detection in the target scene.

[0113] Optional, the batch sample determination module includes:

[0114] The first type of sample determination unit is used to identify multiple first search texts and / or first missed images in the positive sample dataset as first type samples.

[0115] The second type first determining unit is used to determine the corresponding positive sample data as the second type first sample from the positive sample dataset based on the first type sample;

[0116] The second type second determining unit is used to randomly select multiple negative sample data of different types from the negative sample dataset according to the type of the first type sample as the second type second sample, and the second type first sample and the second type second sample constitute the second type sample corresponding to the first type sample.

[0117] Optional, the model fine-tuning training module includes:

[0118] The cross-entropy loss parameter determination unit is used to determine the cross-entropy loss parameter of each first type sample based on the cross-entropy loss of the similarity parameter between each first type sample and the corresponding second type sample in each batch of samples.

[0119] The average cross-entropy loss fine-tuning unit is used to determine the average cross-entropy loss based on the cross-entropy loss parameters of all first-type samples in the single batch of processing samples, and to fine-tune the text retrieval image model to be adjusted based on the average cross-entropy loss.

[0120] Optional, the cross-entropy loss parameter determination unit, specifically used for:

[0121] Determine the similarity parameters between the first type of sample and each corresponding second type of sample;

[0122] If the similarity parameter is greater than the preset similarity threshold, then the similarity parameter between the first type of sample and the corresponding second type of sample is determined to be the preset similarity threshold;

[0123] The cross-entropy loss parameter for each of the first type of samples is determined based on the cross-entropy loss of the product of the similarity parameter and the dynamic temperature coefficient to be adjusted; wherein the value of the dynamic temperature coefficient changes dynamically during the fine-tuning training of the text retrieval image model to be adjusted.

[0124] Optionally, the first type of sample includes multiple first search texts and corresponding first missed detection images;

[0125] The average cross-entropy loss fine-tuning unit includes an average cross-entropy loss determination subunit, specifically used for:

[0126] The first average cross-entropy loss is determined based on the average value of the cross-entropy loss parameter of the first search text in the first type of samples;

[0127] The second average cross-entropy loss is determined based on the average value of the cross-entropy loss parameter of the first missed images in the first type of samples.

[0128] The average cross-entropy loss is determined by the sum of the first average cross-entropy loss and the second average cross-entropy loss.

[0129] Optionally, the device further includes a divergence loss training module for fine-tuning the text retrieval image model to be adjusted before training it based on the average cross-entropy loss, including:

[0130] The first search text determination unit is used to determine the search text in the first type of sample as the first search text;

[0131] The second search text determination unit is used to randomly select multiple other search texts from the positive sample dataset as the second search text corresponding to the first search text;

[0132] The divergence loss determination unit is used to determine the divergence loss based on the KL divergence between each of the first search texts and the corresponding second search texts;

[0133] The divergence loss fine-tuning unit is used to fine-tune the text retrieval image model to be adjusted based on the sum of the average cross-entropy loss and the divergence loss.

[0134] Optionally, the first type of sample includes a plurality of first missed images;

[0135] The second type of first determining unit is specifically used for:

[0136] The search text corresponding to the first missed image is determined to be the first text.

[0137] The first text is replaced with multiple non-synonymous texts to obtain multiple second texts;

[0138] The first sample of the second type is constituted based on the first text and the second text.

[0139] Optionally, the dataset creation module includes a positive sample dataset creation unit, specifically used for:

[0140] The search text is expanded using a large language model to obtain expanded search text. A correspondence between the expanded search text and the missed images is established based on the expansion relationship.

[0141] A positive sample dataset is established based on the search text, the expanded search text, and the corresponding missed detection images.

[0142] The text retrieval image model fine-tuning device provided in the embodiments of the present invention can execute the text retrieval image model fine-tuning method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0143] An image retrieval device provided in this embodiment of the invention is specifically used for: inputting target retrieval text into a target text retrieval image model, and obtaining the output of the target text retrieval image model as an image retrieval result determined from an image library corresponding to the target retrieval text; wherein the target text retrieval image model is trained using the text retrieval image model fine-tuning method according to any embodiment of the invention.

[0144] The acquisition, storage, use, and processing of data in this application comply with relevant laws and regulations and do not violate public order and good morals.

[0145] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0146] Figure 5 A schematic diagram of an electronic device 10, which can be used to implement embodiments of the present invention, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0147] like Figure 5 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0148] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0149] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as fine-tuning methods for text retrieval image models or image retrieval methods.

[0150] In some embodiments, the text retrieval image model fine-tuning method or image retrieval method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the text retrieval image model fine-tuning method or image retrieval method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the text retrieval image model fine-tuning method or image retrieval method by any other suitable means (e.g., by means of firmware).

[0151] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific reference products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0152] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0153] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0154] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0155] The systems and technologies described herein can be implemented in computing systems that include back-end components (e.g., as data servers), or computing systems that include switching components (e.g., application servers), or computing systems that include front-end components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such back-end, switching, or front-end components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0156] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0157] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication unit 19, or installed from storage unit 18, or installed from ROM 12. When the computer program is executed by processor 11, it performs the functions defined in the methods of the embodiments of the present invention.

[0158] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0159] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for fine-tuning a text retrieval image model, characterized in that, The method includes: Obtain the erroneous retrieval records of the text retrieval image model to be adjusted; wherein, the erroneous retrieval records include the retrieval text and the corresponding missed and falsely detected images; A positive sample dataset is established based on the retrieved text and the corresponding missed images, and a negative sample dataset is established based on the falsely detected images and the corresponding image description text. Multiple first search texts and / or first missed images in the positive sample dataset are used as first type samples; based on the first type samples, corresponding positive sample data are determined from the positive sample dataset as second type first samples; based on the type of the first type samples, multiple negative sample data of different types are randomly selected from the negative sample dataset as second type second samples; the second type first samples and the second type second samples constitute the second type samples corresponding to the first type samples; and a single batch processing sample is determined based on each first type sample and the corresponding second type sample. Based on the cross-entropy loss of the similarity parameter between each first type sample and the corresponding second type sample in each batch of samples, the cross-entropy loss parameter of each first type sample is determined. The average cross-entropy loss is determined based on the cross-entropy loss parameters of all first-type samples in the single batch of processing samples, and the text retrieval image model to be adjusted is fine-tuned based on the average cross-entropy loss to obtain the fine-tuned target text retrieval image model. Specifically, the cross-entropy loss parameter for each first-type sample is determined based on the cross-entropy loss of the similarity parameter between each first-type sample and its corresponding second-type sample in each batch of processed samples, including: Determine the similarity parameters between the first type of sample and each corresponding second type of sample; If the similarity parameter is greater than the preset similarity threshold, then the similarity parameter between the first type of sample and the corresponding second type of sample is determined to be the preset similarity threshold; The cross-entropy loss parameter for each of the first type of samples is determined based on the cross-entropy loss of the product of the similarity parameter and the dynamic temperature coefficient to be adjusted; wherein the value of the dynamic temperature coefficient changes dynamically during the fine-tuning training of the text retrieval image model to be adjusted.

2. The method according to claim 1, characterized in that, The first type of sample includes multiple first search texts and corresponding first missed detection images; The average cross-entropy loss is determined based on the cross-entropy loss parameters of all first-type samples in the single batch of processing samples, including: The first average cross-entropy loss is determined based on the average value of the cross-entropy loss parameter of the first search text in the first type of samples; The second average cross-entropy loss is determined based on the average value of the cross-entropy loss parameter of the first missed images in the first type of samples. The average cross-entropy loss is determined by the sum of the first average cross-entropy loss and the second average cross-entropy loss.

3. The method according to claim 1, characterized in that, Before fine-tuning the text retrieval image model to be adjusted based on the average cross-entropy loss, the method further includes: The search text in the first type of sample is identified as the first search text; Randomly select multiple other search texts from the positive sample dataset as the second search texts corresponding to the first search text; The divergence loss is determined based on the KL divergence between each of the first search texts and the corresponding second search texts; The text retrieval image model to be adjusted is fine-tuned based on the sum of the average cross-entropy loss and the divergence loss.

4. The method according to claim 1, characterized in that, in, The first type of sample includes multiple first missed detection images; Determining the corresponding positive sample data as the second type of first sample from the positive sample dataset based on the first type of sample includes: The search text corresponding to the first missed image is determined to be the first text. The first text is replaced with multiple non-synonymous texts to obtain multiple second texts; The first sample of the second type is constituted based on the first text and the second text.

5. The method according to claim 1, characterized in that, A positive sample dataset is established based on the retrieved text and the corresponding missed images, including: The search text is expanded using a large language model to obtain expanded search text. A correspondence between the expanded search text and the missed images is established based on the expansion relationship. A positive sample dataset is established based on the search text, the expanded search text, and the corresponding missed detection images.

6. An image retrieval method, characterized in that, The method includes: The target retrieval text is input into the target text retrieval image model, and the output of the target text retrieval image model is the image retrieval result corresponding to the target retrieval text determined from the image library; wherein, the target text retrieval image model is trained by the fine-tuning method of the text retrieval image model according to any one of claims 1-5.

7. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the text retrieval image model fine-tuning method of any one of claims 1-5 or the image retrieval method of claim 6.

Images