Target retrieval method, device, equipment and storage medium

Abstract

The present application discloses a target retrieval method, apparatus, device and storage medium, which relates to the field of electronic information technology and discloses a target retrieval method, including: obtaining a pre-trained target retrieval model; wherein the training data set of the target retrieval model includes simple image-text pairs corresponding to business data, as well as detailed description image-text pairs constructed based on business data, positive sample pool image-text pairs and negative sample pool image-text pairs; inputting the content to be retrieved input by the user into the target retrieval model to obtain the retrieval results output by the target retrieval model; wherein the target retrieval model is used to encode the content to be retrieved into a first feature, and to filter out content corresponding to at least one second feature that matches the first feature in the underlying database used for target retrieval as the retrieval result. The present application effectively expands the amount of data in the dataset for model training, improves the effect of model training, and thereby improves the accuracy of retrieval by the target retrieval model.

Description

Technical Field

[0001] The present application relates to the field of electronic information technology, and in particular to a target retrieval method, apparatus, device and storage medium. Background Art

[0002] Cross-modal search refers to a technique that uses multiple input forms (such as text, images, and audio) to retrieve and display information in other modalities. It has numerous applications in everyday life: from search engines on common social media platforms, product queries on e-commerce platforms, search and diagnosis in the medical field, to smart home devices, efficient and accurate cross-modal search is required to achieve its primary function, helping people easily access information, interact, and solve problems.

[0003] At present, existing cross-modal target retrieval technologies are usually based on CLIP (Contrastive Language-Image Pre-training), and different target retrieval models are optimized and designed according to the characteristics of the task.

[0004] However, such models often perform poorly when the image-text information is limited, resulting in low target retrieval accuracy.

[0005] The above content is only used to assist in understanding the technical solution of this application and does not constitute an admission that the above content is prior art. Summary of the Invention

[0006] The main purpose of this application is to provide a target retrieval method, device, equipment and storage medium, aiming to solve the technical problem in related technologies that CLIP-based models often perform poorly when image-text information is limited, resulting in low target retrieval accuracy.

[0007] To achieve the above objectives, the present application proposes a target retrieval method, which includes:

[0008] Obtain a pre-trained target retrieval model; wherein the training data set of the target retrieval model includes simple image-text pairs corresponding to the business data, as well as detailed description image-text pairs constructed based on the business data, positive sample pool image-text pairs, and negative sample pool image-text pairs;

[0009] The content to be retrieved input by the user is input into the target retrieval model to obtain the retrieval result output by the target retrieval model; wherein the target retrieval model is used to encode the content to be retrieved into a first feature, and filter out content corresponding to at least one second feature that matches the first feature in the underlying database used for target retrieval as the retrieval result.

[0010] In one embodiment, before the step of obtaining a pre-trained target retrieval model, the method further includes:

[0011] Generate simple graphic and text pairs corresponding to the business data by applying templates;

[0012] The business data is passed through a preset image-text model to construct the detailed description image-text pair;

[0013] The business data is applied to the service data by a template or positive example construction method to construct the positive sample pool image-text pairs; wherein the positive example construction method is to first generate seed data by passing the business data through a preset large language model, and then generate the positive sample pool image-text pairs by passing the seed data through a preset fine-tuned language model;

[0014] The positive sample pool image-text pairs are subjected to a preset text matching model to generate and match negative sample pool image-text pairs whose similarity with the positive sample pool image-text pairs is within a preset similarity range;

[0015] The simple image-text pairs, the detailed description image-text pairs, the positive sample pool image-text pairs, and the negative sample pool image-text pairs are used to train a pre-set candidate model to obtain a trained target retrieval model.

[0016] In one embodiment, the candidate model includes an image encoding module, a text encoding module, and an image-to-text decoding module; and the step of training the preset candidate model includes:

[0017] For each image-text pair in the training dataset, perform the following steps:

[0018] Inputting the image in the image-text pair into the image encoding module to obtain a third feature;

[0019] Inputting the third feature into the image-to-text decoding module to obtain first text description information corresponding to the third feature;

[0020] Inputting the second text description information in the image-text pair into the text encoding module to obtain a fourth feature;

[0021] Determining a loss function corresponding to the image-text pair based on the third feature, the fourth feature, the first text description information, and the second text description information;

[0022] The candidate model is trained based on the loss function corresponding to each image-text pair to obtain the target retrieval model.

[0023] In one embodiment, the loss function includes at least one of the following:

[0024] A first loss function is used to characterize the image-text contrastive learning loss corresponding to the simple image-text pair;

[0025] A second loss function is used to characterize the description loss corresponding to the detailed description image-text pair;

[0026] A third loss function is used to characterize the loss function of the positive sample pool corresponding to the positive sample pool image-text pair;

[0027] The fourth loss function is used to characterize the loss function of the negative sample pool corresponding to the negative sample pool image-text pair.

[0028] In one embodiment, the step of inputting the to-be-searched content input by the user into the target search model to obtain the search results output by the target search model includes:

[0029] The content to be searched input by the user is input into the target search model, and the target search model performs the following steps:

[0030] Encoding the content to be retrieved into a first feature;

[0031] Calculating the similarity between the first feature corresponding to the content to be retrieved and at least one second feature in the underlying database;

[0032] In descending order of similarity, a preset number of pictures corresponding to the second features are selected as the search results.

[0033] In one embodiment, before the step of inputting the to-be-searched content input by the user into the target search model to obtain the search results output by the target search model, the method further includes:

[0034] Inputting the picture in the underlying database into the picture encoding module of the target retrieval model to obtain the at least one second feature corresponding to the picture in the underlying database;

[0035] The step of encoding the content to be retrieved into a first feature includes:

[0036] In a case where the retrieval type of the content to be retrieved is an image, inputting the content to be retrieved into the image encoding module of the target retrieval model for encoding to obtain an image feature as a first feature corresponding to the content to be retrieved;

[0037] In the case where the retrieval type of the content to be retrieved is text description, the content to be retrieved is input into the text encoding module of the target retrieval model for encoding to obtain text features as the first features corresponding to the content to be retrieved.

[0038] In one embodiment, when the retrieval type of the content to be retrieved is an image, the step of inputting the content to be retrieved into the image encoding module of the target retrieval model for encoding to obtain an image feature as the first feature corresponding to the content to be retrieved includes:

[0039] In a case where the retrieval type of the content to be retrieved is image, determining whether the content to be retrieved can be read by the target retrieval model;

[0040] In the case that the content to be retrieved can be read by the target retrieval model, the content to be retrieved is input into the image encoding module of the target retrieval model for encoding to obtain image features as the first features corresponding to the content to be retrieved.

[0041] In one embodiment, when the search type of the content to be retrieved is text description, the step of inputting the content to be retrieved into the text encoding module of the target retrieval model for encoding to obtain a text feature as the first feature corresponding to the content to be retrieved includes:

[0042] In the case where the search type of the content to be retrieved is text description, determining whether the content to be retrieved is a normal query statement;

[0043] In the case that the content to be retrieved is a normal query statement, the content to be retrieved is input into the text encoding module of the target retrieval model to obtain text features as the first features corresponding to the content to be retrieved.

[0044] In addition, to achieve the above-mentioned purpose, the present application also proposes a target retrieval device, which includes:

[0045] An acquisition module is used to acquire a pre-trained target retrieval model; wherein the training data set of the target retrieval model includes simple image-text pairs corresponding to the business data, as well as detailed description image-text pairs constructed based on the business data, positive sample pool image-text pairs, and negative sample pool image-text pairs;

[0046] A retrieval module is used to input the content to be retrieved input by the user into the target retrieval model to obtain the retrieval results output by the target retrieval model; wherein, the target retrieval model is used to encode the content to be retrieved into a first feature, and to filter out content corresponding to at least one second feature that matches the first feature in the underlying database used for target retrieval as the retrieval result.

[0047] In addition, to achieve the above-mentioned purpose, the present application also proposes a target retrieval device, which includes: a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the computer program is configured to implement the steps of the target retrieval method as described above.

[0048] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium. A computer program is stored on the storage medium, and when the computer program is executed by the processor, the steps of the target retrieval method described above are implemented.

[0049] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the steps of the target retrieval method described above.

[0050] One or more technical solutions proposed in this application have at least the following technical effects:

[0051] When business data is limited, the training data set used for model training in this application includes not only simple data pairs corresponding to the business data, but also other training data constructed from different angles based on the business data, specifically detailed descriptions of image-text pairs, positive sample pool image-text pairs, and negative sample pool image-text pairs, which effectively expands the amount of data in the data set for model training from different angles, effectively improves the effect of model training, and thus effectively improves the accuracy of retrieval by the target retrieval model; compared with the existing training data that has the problems of noise and low data utilization, this application makes more full use of business data. This method is particularly suitable for situations where the amount of data is small and the quality is limited. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0053] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0054] Figure 1 This is one of the flow charts of the target retrieval method provided in this application;

[0055] Figure 2 This is a schematic diagram of the distinction between simple image-text pairs, detailed description image-text pairs, positive sample pool image-text pairs, and negative sample pool image-text pairs in the target retrieval method provided in this application;

[0056] Figure 3 This is the second flowchart of the target retrieval method provided by this application;

[0057] Figure 4 This is a flow chart of training data processing in the target retrieval method provided in this application;

[0058] Figure 5 It is a schematic diagram of the model structure in the target retrieval method provided in this application;

[0059] Figure 6 This is a flowchart of the model reasoning in the target retrieval method provided in this application;

[0060] Figure 7 This is the third flow chart of the target retrieval method provided by this application;

[0061] Figure 8 It is a structural diagram of the target retrieval device provided by this application;

[0062] Figure 9 It is a structural diagram of the electronic device provided in this application.

[0063] The purpose, features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0064] It should be understood that the specific embodiments described herein are merely used to explain the technical solutions of the present application and are not intended to limit the present application.

[0065] Unless otherwise defined, all technical and scientific terms used in this application have the same meanings as commonly understood by technicians in the technical field to which this application belongs; the terms used in this application are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" in the specification and claims of this application and the above-mentioned figure descriptions and any variations thereof are intended to cover non-exclusive inclusions.

[0066] In the description of the embodiments of this application, the technical terms "first" and "second" are used only to distinguish different objects and should not be understood to indicate or imply relative importance or implicitly indicate the quantity, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, the meaning of "plurality" is two or more, unless otherwise specifically defined.

[0067] References to "embodiments" in this application mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0068] In the description of the embodiments of this application, the term "and / or" is simply a description of the association relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent the following three situations: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in this document generally indicates that the associated objects are in an "or" relationship.

[0069] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0070] In related technologies, cross-modal search refers to the retrieval behavior between different modalities, such as retrieving images, sounds, or videos based on text information.

[0071] Existing cross-modal object retrieval techniques are typically based on CLIP, optimizing and designing different object retrieval models based on task characteristics. CLIP, a text-image pre-training model using contrastive learning, often suffers from suboptimal performance when limited information is available between the image and text pairs.

[0072] To address the above issues, this application provides a target retrieval method, apparatus, device, and storage medium. These methods utilize the concept of BLIP (Bootstrapping Language-Image Pre-training) to add text generation tasks to the original image-text comparison training. This method then optimizes the training data from various angles, such as generating dense captions corresponding to images and pooling positive and negative samples, thereby improving algorithm performance. This method makes full use of all training data, making it particularly suitable for training target retrieval models when the amount of data is limited and the quality of the data is limited.

[0073] Among them, BLIP is an improvement based on CLIP, which unifies visual language understanding and generation, has stronger capabilities in understanding long texts and complex semantics, and can achieve more accurate and in-depth language understanding.

[0074] It should be noted that the execution subject of the embodiments of the present application can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, mobile phone, etc., or an electronic device capable of performing the above functions, a target retrieval device, etc. The following uses the target retrieval device as an example to illustrate the embodiments of the present application and the following embodiments.

[0075] The following embodiments of the present application and the following embodiments are described in detail.

[0076] The present application embodiment provides a target retrieval method, referring to Figure 1 , Figure 1 This is one of the flowcharts of the target retrieval method provided in this application. The method includes steps S101 to S102:

[0077] Step S101, obtaining a pre-trained target retrieval model;

[0078] Among them, the training data set of the target retrieval model includes simple image-text pairs corresponding to business data, as well as detailed description image-text pairs constructed based on business data, positive sample pool image-text pairs and negative sample pool image-text pairs.

[0079] It should be noted that the positive and negative sample pools containing image-text pairs and negative sample pools can be understood as decomposing the dense caption corresponding to a particular image into different attributes, further assembling them into different positive text samples, and further modifying the corresponding attributes to form / assemble different negative text samples. All positive text samples constitute the positive sample pool, and all negative text samples constitute the negative sample pool. The total is called the positive and negative sample pools.

[0080] For example, consider an image of a man wearing a red scarf. The corresponding positive pool image-text pairs for this example might be: "a person wearing a red scarf," "a man." The corresponding negative pool image-text pairs might be: "a person wearing a black scarf," "a woman." In other words, the positive pool image-text pairs are those that match the image's target, while the negative pool image-text pairs are those that don't.

[0081] Step S102: input the content to be searched input by the user into the target search model to obtain the search results output by the target search model;

[0082] The target retrieval model is used to encode the content to be retrieved into a first feature, and filter out content corresponding to at least one second feature that matches the first feature in the underlying database used for target retrieval as the retrieval result.

[0083] It should be noted that the content to be retrieved input by the user can be pictures, text, audio and other content, and this application does not impose any restrictions on this.

[0084] It should also be noted that a large number of images, texts, audios, etc. can be stored in advance in the underlying database, which is used to retrieve the search results that match the search content input by the user in the underlying database and return them to the user.

[0085] Specifically, the target retrieval model can be pre-trained based on simple image-text pairs corresponding to business data, as well as detailed description image-text pairs constructed based on business data, image-text pairs in the positive sample pool, and image-text pairs in the negative sample pool. The training data covers different angles to facilitate improving the model training effect. The trained target retrieval model is then used to retrieve the content to be retrieved entered by the user, specifically performing feature-level retrieval in the underlying database. The target retrieval model first encodes the content to be retrieved entered by the user into a first feature, and then filters out content corresponding to at least one second feature that matches the first feature in the underlying database, returning it to the user as the search result.

[0086] It should be noted that in order to achieve cross-modal retrieval, the present application can encode the content to be retrieved input by the user into a first feature through the target retrieval model, and perform feature-level matching with the second feature encoded in advance in the underlying database, so as to avoid the situation where the text input by the user is difficult to match with the image in the underlying database. By matching through features, it can effectively adapt to cross-modal retrieval.

[0087] In some embodiments, the training data set may also include general knowledge data, which represents common sense knowledge in the existing world, and business data represents data in the field of focus of the target retrieval task, such as security, medical care, agriculture, etc. The data type of general knowledge data is image-text pairs, and the data can be collected from the Internet, usually in large quantities; while business data mainly comes from business, usually in small quantities and stored in attribute dictionary format. The present application adds general knowledge data for image-text comparison learning in model training, and task-related data for image-text comparison learning, positive and negative sample pool comparison learning, and detailed description generation tasks, so that the learning of the model can be more purposeful without losing its versatility.

[0088] An embodiment of the present application provides a target retrieval method. When business data is limited, the training data set used for model training in the present application includes not only simple data pairs corresponding to the business data, but also other training data constructed from different angles based on the business data, specifically including detailed descriptions of image-text pairs, positive sample pool image-text pairs, and negative sample pool image-text pairs, which effectively expands the amount of data in the data set for model training from different angles, effectively improves the effect of model training, and thereby effectively improves the accuracy of retrieval by the target retrieval model; compared with the existing training data that has the problems of noise and low data utilization, the present application makes more full use of business data, and this method is particularly suitable for situations where the amount of data is small and the quality is limited.

[0089] In some embodiments, a specific implementation method for constructing a training data set and training a model is provided. Before step S101, the following steps may be included:

[0090] S1-1, generating a simple image-text pair corresponding to the business data by applying a template;

[0091] It should be noted that the text description containing only attribute information can be obtained through template application and form a simple picture-text pair with the picture. The template used in template application can be generated through a large model according to specific needs.

[0092] S1-2, constructing the detailed description image-text pair by applying the business data to a preset image-text model;

[0093] In some embodiments, the large image-to-text model is, for example, the Intern-vl (Intern Vision-Language) model, which is a multimodal large model mainly used for vision-language understanding and generation tasks. The large image-to-text model can also be BLIP-2 (Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models), PaLI (A Jointly-Scaled Multilingual Language-Image Model), etc. This application does not impose any restrictions on this.

[0094] Specifically, a detailed text description of the picture can be obtained through the Intern-vl model, and a detailed description picture-text pair can be formed between the picture and the text.

[0095] S1-3, constructing the image-text pairs in the positive sample pool by applying a template or constructing a positive example to the business data; wherein the positive example construction method is to first generate seed data by passing the business data through a preset large language model, and then generate the image-text pairs in the positive sample pool by passing the seed data through a preset fine-tuned language model;

[0096] In some embodiments, the large language model is, for example, GPT-4v, Deepseek, Doubao, Wenxin Yiyan, Qianwen, and other models, and the fine-tuned language model is, for example, the qwen model, which is not limited in this application.

[0097] Specifically, for the positive and negative sample pools, the following two methods can be used to construct the positive sample pool image-text pairs:

[0098] <1> By applying the template, a text description containing only attribute information is obtained, and it is combined with the image to form a positive sample pool of image-text pairs.

[0099] It should be noted that the granularity of the template used to construct the positive sample pool image-text pairs here can be different from the granularity of the template used to generate the simple image-text pairs in S1-1, so that the text description in the simple image-text pairs can be mainly used to describe the attribute information of the entire image, while the positive sample pool image-text pairs can be used to describe the attribute information of some details in the image.

[0100] <2> Generate seed data through a model such as the GPT-4v model, and then generate the required detailed text description through a model such as the qwen model. The detailed text description is then broken down into different attribute information, and further assembled with images to form different positive sample pool image-text pairs.

[0101] S1-4, using a preset text matching model to generate and match the positive sample pool image-text pairs with negative sample pool image-text pairs whose similarity to the positive sample pool image-text pairs is within a preset similarity range;

[0102] In some embodiments, the text matching model is, for example, BGE-M3, BERT (Bidirectional Encoder Representations from Transformers), RoBERTa, ALBERT (A Lite BERT), DistilBERT, ELECTRA, T5 model, etc., and this application does not impose any restrictions on this.

[0103] It should be noted that the preset similarity range can be set according to actual conditions. After constructing the sample pool image-text pairs, for example, the sample pool image-text pairs whose similarity with the positive sample pool image-text pairs is less than a certain threshold are set as negative sample pool image-text pairs. For example, the sample pool image-text pairs with a similarity less than 0.4 are used as negative sample pool image-text pairs, or the sample pool image-text pairs with a similarity less than 0.75 are selected as their negative sample pool image-text pairs, so as to avoid the high similarity between the constructed negative sample pool image-text pairs and the positive sample pool image-text pairs, which makes it difficult for the constructed negative sample pool image-text pairs to effectively improve the effect of model training.

[0104] S1-5, using the simple image-text pairs, the detailed description image-text pairs, the positive sample pool image-text pairs and the negative sample pool image-text pairs, a preset candidate model is trained to obtain a trained target retrieval model.

[0105] It should be noted that the embodiment of the present application does not limit the order of executing the above S1-1, S1-2, S1-3 and S1-4, that is, it does not limit the order of obtaining various data in the training data set.

[0106] Figure 2 This is a schematic diagram of the distinction between simple image-text pairs, detailed description image-text pairs, positive sample pool image-text pairs, and negative sample pool image-text pairs in the target retrieval method provided by this application, such as Figure 2 As shown, the following examples illustrate the differences between simple image-text pairs, detailed description image-text pairs, positive sample pool image-text pairs, and negative sample pool image-text pairs:

[0107] 1) Simple picture-text pair: A man wearing a brown polo shirt and white shorts;

[0108] 2) Detailed description of the image / text pair: A man wearing a brown polo shirt and white shorts, with his right hand in his pocket and his left hand above his pants, is wearing white shoes. He is wearing sunglasses and has short hair. There are green trees in the background, and he appears to be standing on the road.

[0109] 3) Positive sample pool image-text pairs: a person wearing a brown polo shirt, ... (other content), a man wearing white shorts.

[0110] 4) Negative sample pool image-text pairs: a person wearing a black polo shirt, ... (other content), a woman wearing white shorts.

[0111] In the embodiment of the present application, although there is only one source of business data, the present application maximizes the utilization of its information according to different construction methods, effectively alleviating the problem of insufficient business data affecting the model training effect, improving the performance of the trained target retrieval model, and facilitating more accurate target retrieval.

[0112] In some embodiments, a specific implementation of a candidate model is provided. The candidate model may include a picture encoding module, a text encoding module, and a picture-to-text decoding module;

[0113] Accordingly, the above step of training the preset candidate model may include the following sub-steps:

[0114] S2-1, for each image-text pair in the training dataset, perform the following steps:

[0115] Inputting the image in the image-text pair into the image encoding module to obtain a third feature;

[0116] Inputting the third feature into the image-to-text decoding module to obtain first text description information corresponding to the third feature;

[0117] Inputting the second text description information in the image-text pair into the text encoding module to obtain a fourth feature;

[0118] Determining a loss function corresponding to the image-text pair based on the third feature, the fourth feature, the first text description information, and the second text description information;

[0119] S2-2, training the candidate model based on the loss function corresponding to each image-text pair to obtain the target retrieval model.

[0120] In some embodiments, the candidate model is, for example, a VIT (Vision Transformer) model, a BERT model, a transformer model, etc., and this application does not impose any restrictions on this.

[0121] In related technologies, candidate models usually only include an image encoding module and a text encoding module. During the model training process, for a picture-text pair, the picture is encoded into picture features through the picture encoding module, and the corresponding text description is encoded into text features through the text encoding module. Then, the loss function is determined based on the picture-text comparison of the picture features and the text features to perform supervised training on the model, expecting the features between the picture and the text to be encoded into a unified space and aligned in this space.

[0122] In this application, the candidate model also includes an image-to-text decoding module. During the model training process, for each image-to-text pair in the training data set, the third feature output by the image encoding module can be input into the image-to-text decoding module to obtain the first text description information corresponding to the third feature, and based on the first text description information and the second text description information corresponding to the image, the loss function corresponding to the image-to-text pair is determined to train the candidate model. The image-to-text decoding module is mainly used to assist the image encoding module and the text encoding module in training during the model training phase. After the target retrieval model is obtained through training, only the image encoding module and the text encoding module are used for feature encoding and similarity comparison.

[0123] In the embodiment of the present application, the trained target retrieval model is end-to-end cross-modal retrieval. On the basis of the comparative learning task constructed by the dual-tower model (i.e., the structure of image encoding and text encoding), an image-to-text decoding module is added to realize the text generation task, so that the image encoding module and text encoding module of the dual-tower model can further learn the fine-grained alignment relationship between images and texts. The addition of such tasks makes the alignment between images and texts more fine-grained, and also solves various problems such as poor multi-attribute constraints and poor coherent semantic modeling.

[0124] In some embodiments, a specific implementation of a loss function is provided. The loss function may include at least one of the following:

[0125] 1) A first loss function, used to represent the image-text contrastive learning loss corresponding to the simple image-text pair;

[0126] 2) A second loss function is used to represent the caption loss corresponding to the detailed description image and text pair;

[0127] 3) A third loss function, used to represent the loss function of the positive sample pool corresponding to the image-text pair in the positive sample pool;

[0128] 4) A fourth loss function is used to represent the loss function of the negative sample pool corresponding to the negative sample pool image-text pair.

[0129] In the embodiment of the present application, compared with the related art that only uses simple image-text pairs to train the model, the present application adds multiple sets of image-to-text losses for the newly added positive and negative sample pool image-text pairs, which effectively improves the training effect. At the same time, due to the addition of the loss function corresponding to the negative sample, it also solves the problem of negative query difference of the related model, and can also perform target retrieval more accurately for the negative type of search content input by the user.

[0130] In some embodiments, a specific implementation method of target retrieval model is provided. The above step S102 may include the following sub-steps:

[0131] S3-1, input the content to be searched input by the user into the target search model, and the target search model performs the following steps:

[0132] Encoding the content to be retrieved into a first feature;

[0133] Calculating the similarity between the first feature corresponding to the content to be retrieved and at least one second feature in the underlying database;

[0134] In descending order of similarity, a preset number of pictures corresponding to the second features are selected as the search results.

[0135] Specifically, after receiving the content to be retrieved input by the user, the target retrieval model can first encode the content to be retrieved into a first feature, and compare it with at least one second feature in the underlying database, calculate the similarity between the first feature and at least one second feature, and select a preset number of pictures corresponding to the second features in order of similarity from large to small, for example, select the top 100 pictures and return them to the terminal device on the user side for display.

[0136] It should be noted that the preset number of the second feature returned can be set according to actual conditions and is not limited in this application.

[0137] In some embodiments, the similarity calculated above is, for example, cosine similarity.

[0138] In some embodiments, a specific implementation of image or text search is provided. Before the above step S102, the following steps may also be included:

[0139] Inputting the picture in the underlying database into the picture encoding module of the target retrieval model to obtain the at least one second feature corresponding to the picture in the underlying database;

[0140] A specific implementation of encoding the to-be-retrieved content into the first feature may include:

[0141] 1) When the search type of the content to be retrieved is an image, the content to be retrieved is input into the image encoding module of the target retrieval model for encoding to obtain an image feature as the first feature corresponding to the content to be retrieved;

[0142] Specifically, if the content to be retrieved is a picture, it can be encoded by the picture encoding module of the target retrieval model, and after being encoded into picture features, it is matched with at least one second feature in the underlying database to perform target retrieval.

[0143] In some embodiments, when the retrieval type of the content to be retrieved is an image, the content to be retrieved is input into the image encoding module of the target retrieval model for encoding to obtain an image feature as the first feature corresponding to the content to be retrieved. A specific implementation method may include:

[0144] In a case where the retrieval type of the content to be retrieved is image, determining whether the content to be retrieved can be read by the target retrieval model;

[0145] In the case that the content to be retrieved can be read by the target retrieval model, the content to be retrieved is input into the image encoding module of the target retrieval model for encoding to obtain image features as the first features corresponding to the content to be retrieved.

[0146] Specifically, if the content to be retrieved is an image, the legitimacy of the content to be retrieved also needs to be determined, specifically whether the image input by the user can be read by the target retrieval model. If it is determined that it can be read by the target retrieval model, the target retrieval model can perform subsequent target retrieval steps;

[0147] In some embodiments, if it is determined that the content cannot be read by the target retrieval model, a prompt message for re-entering may be returned to the user until the content to be retrieved input by the user is legal, and then subsequent target retrieval steps are performed.

[0148] 2) When the search type of the content to be retrieved is text description, the content to be retrieved is input into the text encoding module of the target retrieval model for encoding to obtain text features as the first features corresponding to the content to be retrieved.

[0149] Specifically, if the content to be retrieved is a text description, it can be encoded by the text encoding module of the target retrieval model, and after being encoded into text features, it is matched with at least one second feature in the underlying database to perform target retrieval.

[0150] In some embodiments, when the search type of the content to be retrieved is text description, the content to be retrieved is input into the text encoding module of the target retrieval model for encoding to obtain text features as the first features corresponding to the content to be retrieved. A specific implementation method may include:

[0151] In the case where the search type of the content to be retrieved is text description, determining whether the content to be retrieved is a normal query statement;

[0152] In the case that the content to be retrieved is a normal query statement, the content to be retrieved is input into the text encoding module of the target retrieval model to obtain text features as the first features corresponding to the content to be retrieved.

[0153] Specifically, if the content to be retrieved is a text description, it is also necessary to determine the legitimacy of the content to be retrieved, specifically whether the text description entered by the user is a normal query statement. If it is determined to be a normal query statement, the target retrieval model can perform subsequent target retrieval steps;

[0154] In some embodiments, if it is determined that the text description input by the user is not a normal query statement, a prompt message for re-entering the query can be returned to the user until the content to be retrieved input by the user is legal, and then subsequent target retrieval steps are performed.

[0155] The following examples illustrate the target retrieval method provided by the embodiments of the present application. The overall goal of the present method is to use a system built with a trained model to perform retrieval based on user needs. For example, based on the text description provided by the user, images that match the user's description can be retrieved from the underlying database.

[0156] In general, the training data is first processed, then the corresponding loss function is designed to train the model, and finally the model is applied, i.e., model inference is performed. Before the retrieval system is built, all images are encoded using the model's image encoding module based on the underlying database provided by the user, ultimately matching the text features provided by the user, which are encoded by the model's text encoding module.

[0157] Figure 3 This is the second flow chart of the target retrieval method provided by this application, such as Figure 3 As shown, it mainly includes the following stages:

[0158] 1) Training data processing: The business data of the task is processed through the business attribute data processing module to construct simple image-text pairs, detailed description image-text pairs, and positive and negative sample pool image-text pairs (including positive sample pool image-text pairs and negative sample pool image-text pairs). The image-text pairs of general knowledge data are then input into the model for training.

[0159] 2) Model training: The training data uses simple image-text pairs, positive and negative sample pool image-text pairs, detailed description image-text pairs, and general knowledge data image-text pairs for training. The model uses a dual-tower structure similar to BLIP plus a text understanding generation module, which is the above-mentioned image-to-text decoding module. The model loss uses the original image-text contrast learning loss and the self-developed loss function based on simple business image-text pairs, positive and negative sample pools, and the caption loss corresponding to the next token predict task. Here, the next token predict task refers to the task of generating the next word, and the data used is the detailed description image-text pair. According to Figure 1The task is called NTP (next token prediction), and the corresponding loss is called caption loss (also known as the description loss mentioned above).

[0160] 3) Model reasoning: In the application phase, the text or image content to be queried is input and integrated into the model's text or image encoding module for encoding. The similarity is then calculated with the pre-encoded image features in the underlying database to obtain the retrieval results.

[0161] The following examples illustrate the specific implementation details of the above parts:

[0162] <1> Training data processing:

[0163] This application divides the overall training data into general knowledge data and business data. General knowledge data represents common sense knowledge in the existing world, while business data represents data in the field of focus of the task, such as security, healthcare, agriculture, etc. General knowledge data is a type of image-text pair collected from the internet, usually in large quantities; business data mainly comes from business, is usually smaller in quantity, and is stored in an attribute dictionary format. The goal of this stage is to obtain training data for the model.

[0164] Figure 4 This is a flow chart of the training data processing in the target retrieval method provided by this application, such as Figure 4 As shown, for business data whose data is attribute data, this application uses two training transformation methods, namely image-text pairs and positive and negative sample pools.

[0165] Among them, image-text pairs include two categories. One is a text description containing only attribute information obtained by applying a template to generate a simple image-text pair; the other is a detailed text description of the image obtained by a large image-text model (such as Intern-vl) to construct a detailed description image-text pair.

[0166] The positive and negative sample pools also include two categories: the Image-Attr positive and negative sample pool and the Image-Dense positive and negative sample pool. Positive samples are obtained differently: first, a text description containing only attribute information is obtained by applying a template (the positive samples in the Image-Attr positive and negative sample pool mentioned above); second, a large language model (such as GPT-4v) is used to generate seed data for positive samples, and then a fine-tuned language model (such as qwen) is used to generate the required detailed text description (the positive samples in the Image-Dense positive and negative sample pool mentioned above). Negative samples are matched using a text matching model (such as BGE-M3), and text within a certain similarity threshold is selected as negative sample text.

[0167] Through the above training data processing process, we can obtain general knowledge data in the form of image-text pairs and business data of two types of image-text pairs and two types of positive and negative sample pools. Although there is only one source of business data, the different construction methods of this application maximize the utilization of its information, alleviating the problem of insufficient business data.

[0168] <2> Model training:

[0169] Figure 5 This is a schematic diagram of the model structure in the target retrieval method provided by this application, such as Figure 5 As shown in Figure 2, the goal of this stage is to obtain a model that performs image-text alignment to complete the retrieval task.

[0170] The model inputs the two types of data obtained above (image-text pairs and structured data related to general knowledge data and business data). The model consists of three main modules, including an image encoding module, a text encoding module, and an image-to-text decoding module. The models used can be VIT, Bert, and transformer, respectively. There are two main types of tasks, namely text generation tasks and image-text comparison tasks. Among them, the image-text comparison task is set up to expect the features between the image and the text to be encoded into a unified space and aligned in this space. The text generation task is an image-to-text task. The addition of this type of task makes the alignment between the image and the text more fine-grained, and also solves various problems such as poor multi-attribute constraints and poor coherent semantic modeling.

[0171] In order to meet the needs of the new data form - the addition of positive and negative sample pools, this application designs a loss calculation method, that is, calculating the positive and negative sample losses within the sample, such as the losses corresponding to 5 positive and 5 negative text positive and negative examples for a picture. Compared with the previous method, multiple sets of image-to-text losses are added, which improves the training effect and solves the problem of negative query difference.

[0172] <3> Model Inference:

[0173] Figure 6 This is a flow chart of the model reasoning process in the target retrieval method provided in this application, such as Figure 6 As shown, the goal of this stage is to use the above <2> The model obtained in the experiment is input through the user's query text, and then the model is inferred to obtain the final retrieval results.

[0174] Figure 7 This is the third flow chart of the target retrieval method provided by this application, such as Figure 7 As shown, the specific process is as follows:

[0175] a) Base database image feature extraction: When the system is built, images in the existing base database (i.e., the underlying database) are encoded using the model's image encoding module. As the base database expands, the expanded images can be further encoded using the same image encoding module to be added to the image feature base database.

[0176] b) The terminal device obtains the text or image to be searched: The system obtains the text description or similar image input by the user and then determines the legitimacy of the search. For text, this determines whether it is a valid query statement; for images, this determines whether the model can read the image.

[0177] c) Extracting features of the text or image to be retrieved: If the input in step b) is deemed valid, the text encoding module in the model is used to encode the text to obtain text features, or the image encoding module in the model is used to encode the image to be retrieved to obtain image features. If the input is invalid, the process returns to step b) and repeats the input verification until it is valid.

[0178] d) Calculate the similarity between the search feature and the image features in the base database: Calculate the similarity, such as cosine similarity, using the search feature obtained in c) and all the image features in the base database.

[0179] e) Select candidate images and return the top 100 candidate images as the search results to the terminal device: Sort the similarity values obtained in d) and finally return the top 100 similarity images and display them to the terminal.

[0180] In the embodiments of the present application, there are at least the following beneficial effects:

[0181] 1) To address the problems of noise in existing data and low utilization of data information, a data processing method is used to construct a positive and negative sample pool for each sample and generate a detailed description of each sample. Different loss functions are designed for the model according to different data forms, making maximum use of the information of the existing training data. More positive and negative samples are added on the basis of the original positive and negative samples, thereby improving data utilization and training efficiency.

[0182] 2) This application is an end-to-end cross-modal retrieval method. It adds a text generation task based on the contrastive learning task constructed by the dual-tower model, so that the image and text encoding modules of the dual-tower model can further learn the fine-grained alignment relationship between images and text, making the knowledge learned by the image and text encoding modules more comprehensive.

[0183] 3) The application proposes to conduct image-text comparison learning on general knowledge data, image-text comparison learning on task-related data, positive and negative sample pool comparison learning, and detailed description generation tasks, so that the model learning can be more purposeful without losing its generality, so that the model maintains the general knowledge of generality and takes the task itself into consideration.

[0184] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the target retrieval method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0185] This application also provides a target retrieval device, Figure 8 This is a schematic diagram of the structure of the target retrieval device provided by this application. Figure 8 As shown, the target retrieval device includes:

[0186] Acquisition module 801 is used to acquire a pre-trained target retrieval model; wherein the training data set of the target retrieval model includes simple image-text pairs corresponding to business data, detailed description image-text pairs constructed based on business data, positive sample pool image-text pairs, and negative sample pool image-text pairs;

[0187] The retrieval module 802 is used to input the content to be retrieved input by the user into the target retrieval model to obtain the retrieval results output by the target retrieval model; wherein, the target retrieval model is used to encode the content to be retrieved into a first feature, and filter out content corresponding to at least one second feature that matches the first feature in the underlying database used for target retrieval as the retrieval result.

[0188] The target retrieval device provided in this application, which utilizes the target retrieval method described in the aforementioned embodiments, can address the technical issue in related art where CLIP-based models often perform poorly when limited image-text information is available, resulting in low target retrieval accuracy. Compared to the prior art, the target retrieval device provided in this application has the same beneficial effects as the target retrieval method described in the aforementioned embodiments. Other technical features of the target retrieval device are the same as those disclosed in the aforementioned embodiments and are not further elaborated here.

[0189] The present application provides a target retrieval device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the target retrieval method in the above-mentioned embodiment.

[0190] Reference below Figure 9 , Figure 9is a schematic diagram of the structure of an electronic device provided in the present application, which illustrates a schematic diagram of the structure of a target retrieval device suitable for implementing embodiments of the present application. The target retrieval device in the embodiments of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 9 The target retrieval device shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0191] like Figure 9 As shown, the target retrieval device may include a processing device 901 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes based on programs stored in a read-only memory (ROM) 902 or programs loaded from a storage device 903 into a random access memory (RAM) 904. RAM 904 also stores various programs and data required for the operation of the target retrieval device. Processing device 901, ROM 902, and RAM 904 are interconnected via a bus 905. An input / output (I / O) interface 906 is also connected to the bus. Typically, the following systems can be connected to I / O interface 906: input devices 907, such as a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 908, such as a liquid crystal display (LCD), speaker, vibrator, etc.; storage device 903, such as a magnetic tape or hard disk; and communication device 909. Communication device 909 can allow target retrieval equipment to carry out wireless or wired communication with other equipment to exchange data. Although shown in the figure, target retrieval equipment with various systems, it should be understood that it is not required to implement or possess all the systems shown. Can alternatively implement or possess more or less systems.

[0192] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device 903, or installed from a ROM 902. When the computer program is executed by the processing device 901, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0193] The target retrieval device provided in this application, which utilizes the target retrieval method of the aforementioned embodiment, can address the technical problem in related art where CLIP-based models often perform poorly when limited image-text information is available, resulting in low target retrieval accuracy. Compared to the prior art, the beneficial effects of the target retrieval device provided in this application are the same as those of the target retrieval method provided in the aforementioned embodiment, and the other technical features of the target retrieval device are the same as those disclosed in the aforementioned embodiment, and are not further described here.

[0194] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0195] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

[0196] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer program) stored thereon, wherein the computer-readable program instructions are used to execute the target retrieval method in the above embodiment.

[0197] The computer-readable storage medium provided herein may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including, but not limited to, wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0198] The computer-readable storage medium may be included in the target retrieval device, or may exist independently without being assembled into the target retrieval device.

[0199] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the target retrieval device, the target retrieval device performs the following steps:

[0200] Obtain a pre-trained target retrieval model; wherein the training data set of the target retrieval model includes simple image-text pairs corresponding to the business data, as well as detailed description image-text pairs constructed based on the business data, positive sample pool image-text pairs, and negative sample pool image-text pairs;

[0201] The content to be retrieved input by the user is input into the target retrieval model to obtain the retrieval result output by the target retrieval model; wherein the target retrieval model is used to encode the content to be retrieved into a first feature, and filter out content corresponding to at least one second feature that matches the first feature in the underlying database used for target retrieval as the retrieval result.

[0202] Computer program code for performing the operations of the present application may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0203] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0204] The modules described in the embodiments of the present application may be implemented in software or hardware, wherein the name of a module does not necessarily limit the unit itself.

[0205] The computer-readable storage medium provided in this application stores computer-readable program instructions (i.e., a computer program) for executing the aforementioned target retrieval method. This computer-readable storage medium can address the technical issue in related art where CLIP-based models often perform poorly when limited image-text information is available, resulting in low target retrieval accuracy. Compared to the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the target retrieval method provided in the aforementioned embodiments, and are not further elaborated here.

[0206] The present application also provides a computer program product, comprising a computer program, which implements the steps of the target retrieval method described above when executed by a processor.

[0207] The computer program product provided in this application can address the technical problem in related art where CLIP-based models often perform poorly when limited image-text information is available, resulting in low target retrieval accuracy. Compared to the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the target retrieval method provided in the aforementioned embodiments, and are not further elaborated here.

[0208] The above description is only part of the embodiments of the present application and does not limit the patent scope of the present application. All equivalent structural transformations made by using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A target retrieval method, characterized in that: The method comprises: Obtain a pre-trained target retrieval model; wherein the training data set of the target retrieval model includes simple image-text pairs corresponding to the business data, as well as detailed description image-text pairs constructed based on the business data, positive sample pool image-text pairs, and negative sample pool image-text pairs; Inputting the content to be searched, input by the user, into the target search model to obtain a search result output by the target search model; wherein the target search model is used to encode the content to be searched into a first feature, and filter out content corresponding to at least one second feature that matches the first feature in the underlying database used for target search as the search result; Before the step of obtaining the pre-trained target retrieval model, the method further includes: Generate simple graphic and text pairs corresponding to the business data by applying templates; The business data is passed through a preset image-text model to construct the detailed description image-text pair; The business data is applied to the service data by a template or positive example construction method to construct the positive sample pool image-text pairs; wherein the positive example construction method is to first generate seed data by passing the business data through a preset large language model, and then generate the positive sample pool image-text pairs by passing the seed data through a preset fine-tuned language model; The positive sample pool image-text pairs are subjected to a preset text matching model to generate and match negative sample pool image-text pairs whose similarity with the positive sample pool image-text pairs is within a preset similarity range; The simple image-text pairs, the detailed description image-text pairs, the positive sample pool image-text pairs, and the negative sample pool image-text pairs are used to train a pre-set candidate model to obtain a trained target retrieval model.

2. The method according to claim 1, wherein The candidate model includes an image encoding module, a text encoding module, and an image-to-text decoding module; the step of training the preset candidate model includes: For each image-text pair in the training dataset, perform the following steps: Inputting the image in the image-text pair into the image encoding module to obtain a third feature; Inputting the third feature into the image-to-text decoding module to obtain first text description information corresponding to the third feature; Inputting the second text description information in the image-text pair into the text encoding module to obtain a fourth feature; Determining a loss function corresponding to the image-text pair based on the third feature, the fourth feature, the first text description information, and the second text description information; The candidate model is trained based on the loss function corresponding to each image-text pair to obtain the target retrieval model.

3. The method according to claim 2, wherein The loss function includes at least one of the following: A first loss function is used to characterize the image-text contrastive learning loss corresponding to the simple image-text pair; A second loss function is used to characterize the description loss corresponding to the detailed description image-text pair; A third loss function is used to characterize the loss function of the positive sample pool corresponding to the positive sample pool image-text pair; The fourth loss function is used to characterize the loss function of the negative sample pool corresponding to the negative sample pool image-text pair.

4. The method according to any one of claims 1 to 3, wherein The step of inputting the to-be-searched content input by the user into the target search model to obtain the search results output by the target search model includes: The content to be searched input by the user is input into the target search model, and the target search model performs the following steps: Encoding the content to be retrieved into a first feature; Calculating the similarity between the first feature corresponding to the content to be retrieved and at least one second feature in the underlying database; In descending order of similarity, a preset number of pictures corresponding to the second features are selected as the search results.

5. The method according to claim 4, wherein Before the step of inputting the to-be-searched content input by the user into the target search model to obtain the search results output by the target search model, the method further includes: Inputting the picture in the underlying database into the picture encoding module of the target retrieval model to obtain the at least one second feature corresponding to the picture in the underlying database; The step of encoding the content to be retrieved into a first feature includes: In a case where the retrieval type of the content to be retrieved is an image, inputting the content to be retrieved into the image encoding module of the target retrieval model for encoding to obtain an image feature as a first feature corresponding to the content to be retrieved; In the case where the retrieval type of the content to be retrieved is text description, the content to be retrieved is input into the text encoding module of the target retrieval model for encoding to obtain text features as the first features corresponding to the content to be retrieved.

6. The method according to claim 5, wherein When the retrieval type of the content to be retrieved is an image, the step of inputting the content to be retrieved into the image encoding module of the target retrieval model for encoding to obtain an image feature as the first feature corresponding to the content to be retrieved includes: In a case where the retrieval type of the content to be retrieved is image, determining whether the content to be retrieved can be read by the target retrieval model; In the case that the content to be retrieved can be read by the target retrieval model, the content to be retrieved is input into the image encoding module of the target retrieval model for encoding to obtain image features as the first features corresponding to the content to be retrieved.

7. The method according to claim 5 or 6, wherein: When the search type of the content to be retrieved is text description, the step of inputting the content to be retrieved into the text encoding module of the target retrieval model for encoding to obtain a text feature as the first feature corresponding to the content to be retrieved includes: In the case where the search type of the content to be retrieved is text description, determining whether the content to be retrieved is a normal query statement; In the case that the content to be retrieved is a normal query statement, the content to be retrieved is input into the text encoding module of the target retrieval model to obtain text features as the first features corresponding to the content to be retrieved.

8. A target retrieval device, characterized in that: The device comprises: An acquisition module is used to acquire a pre-trained target retrieval model; wherein the training data set of the target retrieval model includes simple image-text pairs corresponding to business data, as well as detailed description image-text pairs constructed based on business data, positive sample pool image-text pairs, and negative sample pool image-text pairs; A retrieval module, configured to input the to-be-retrieved content input by the user into the target retrieval model, and obtain a retrieval result output by the target retrieval model; wherein the target retrieval model is configured to encode the to-be-retrieved content into a first feature, and to filter out content corresponding to at least one second feature that matches the first feature from an underlying database for target retrieval, as the retrieval result; The acquisition module is further used for: Generate simple graphic and text pairs corresponding to the business data by applying templates; The business data is passed through a preset image-text model to construct the detailed description image-text pair; The business data is applied to the service data by a template or positive example construction method to construct the positive sample pool image-text pairs; wherein the positive example construction method is to first generate seed data by passing the business data through a preset large language model, and then generate the positive sample pool image-text pairs by passing the seed data through a preset fine-tuned language model; The positive sample pool image-text pairs are subjected to a preset text matching model to generate and match negative sample pool image-text pairs whose similarity with the positive sample pool image-text pairs is within a preset similarity range; The simple image-text pairs, the detailed description image-text pairs, the positive sample pool image-text pairs, and the negative sample pool image-text pairs are used to train a pre-set candidate model to obtain a trained target retrieval model.

9. A target retrieval device, characterized in that: The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the target retrieval method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the target retrieval method according to any one of claims 1 to 7 are implemented.

11. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the target search method according to any one of claims 1 to 7 are implemented.

Images