Information retrieval model generation method and device, information retrieval model application method and device, equipment and medium

Abstract

The embodiment of the invention discloses an information retrieval model generation method and device, an information retrieval model application method and device, equipment and a medium. The method comprises the steps of determining a reference text image pair adopted by information retrieval model training, wherein an information retrieval model is configured and constructed based on a contrast learning model; performing text expansion on a text in the reference text image through a text generation model to obtain a target text sample set, and performing image expansion on an image in the reference text image through an image generation model to obtain a target image sample set; and performing parameter adjustment on the information retrieval model based on the reference text image pair, the target text sample set and the target image sample set to generate a convergent information retrieval model. According to the scheme, the generalization ability of the information retrieval model is kept, and meanwhile, the resolution performance of the information retrieval model during processing of tasks needing highly refined resolution is improved.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of machine learning, and in particular to a method for generating an information retrieval model, an application method, an apparatus, a device, and a medium. Background Art

[0002] In similar business scenarios that require highly refined understanding capabilities, it is usually necessary to search for corresponding similar text or images based on text descriptions or given images.

[0003] In order to achieve text or image retrieval, a large multimodal model can be used to collect hundreds of millions of multimodal data on the Internet and then train the model. However, since the data on the Internet is too complex, many multimodal data pairs (such as image-text pairs) are not completely aligned, and the long tail effect of the data is obvious, resulting in the multimodal model having excellent generalization ability but not satisfactory performance when processing high-precision tasks, or having high precision but weak generalization ability for specific tasks. Therefore, it is particularly important to maintain the generalization ability of information retrieval and the performance when processing high-precision tasks at the same time. Summary of the invention

[0004] The embodiments of the present disclosure provide a method for generating, an application method, an apparatus, a device and a medium for an information retrieval model to solve the problem of low generalization ability and low ability to process high-precision tasks of the information retrieval model.

[0005] In a first aspect, an embodiment of the present disclosure provides a method for generating an information retrieval model, the method comprising:

[0006] Determining reference text-image pairs for training an information retrieval model, wherein the information retrieval model is configured and constructed based on a contrastive learning model;

[0007] Performing text expansion on the text in the reference text image through a text generation model to obtain a target text sample set, and performing image expansion on the image in the reference text image through an image generation model to obtain a target image sample set;

[0008] The parameters of the information retrieval model are adjusted based on the reference text-image pair, the target text sample set and the target image sample set to generate a converged information retrieval model.

[0009] In a second aspect, an embodiment of the present disclosure further provides an application method of an information retrieval model, wherein the information retrieval model is obtained by using the generation method of the information retrieval model described in any of the above embodiments, and the application method comprises:

[0010] Determining information to be processed, wherein the information to be processed is text to be processed or an image to be processed;

[0011] Using an information retrieval model to search for images corresponding to the text to be processed from a preset image library; or,

[0012] An information retrieval model is used to search for text corresponding to the image to be processed from a preset text library.

[0013] In a third aspect, an embodiment of the present disclosure further provides a device for generating an information retrieval model, the device comprising:

[0014] A determination module, used to determine reference text-image pairs used for training an information retrieval model, wherein the information retrieval model is configured and constructed based on a contrastive learning model;

[0015] An expansion module, configured to perform text expansion on the text in the reference text image through a text generation model to obtain a target text sample set, and to perform image expansion on the image in the reference text image through an image generation model to obtain a target image sample set;

[0016] A generation module is used to adjust parameters of the information retrieval model based on the reference text-image pair, the target text sample set and the target image sample set to generate a converged information retrieval model.

[0017] In a fourth aspect, an application device of an information retrieval model is also provided in an embodiment of the present disclosure. The information retrieval model is obtained by using the information retrieval model generation method described in any of the above embodiments. The application device includes:

[0018] An input module, used to determine information to be processed, wherein the information to be processed is text to be processed or an image to be processed;

[0019] The search module is used to use an information retrieval model to search for images corresponding to the text to be processed from a preset image library; or to use an information retrieval model to search for texts corresponding to the images to be processed from a preset text library.

[0020] In a fifth aspect, an electronic device is also provided in an embodiment of the present disclosure, and the electronic device includes:

[0021] at least one processor; and

[0022] a memory communicatively connected to the at least one processor; wherein,

[0023] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the method for generating an information retrieval model or the method for applying an information retrieval model described in any one of the above embodiments.

[0024] In a sixth aspect, a computer-readable medium is also provided in an embodiment of the present disclosure, wherein the computer-readable medium stores computer instructions, and the computer instructions are used to enable a processor to implement a method for generating an information retrieval model or an application method of an information retrieval model as described in any one of the above embodiments when executed.

[0025] The technical solution of the disclosed embodiment obtains the reference text-image pairs used in the training of the information retrieval model, introduces the text generation model and the image generation model, performs text expansion and image expansion on the basis of the text and image included in the reference text-image pairs, effectively utilizes the generation capabilities of the text generation model and the image generation model, generates more text descriptions or enhances context information through text expansion and image expansion, and provides more training sample data for the training of the information retrieval model. At the same time, the information retrieval model is configured and constructed based on the contrastive learning model, and the contrastive learning model is used to improve the fine resolution capability of specific tasks, and can provide more accurate results in tasks that require highly fine resolution capabilities, thereby achieving the goal of maintaining the generalization capability of the information retrieval model while improving its resolution performance when processing tasks that require highly fine resolution.

[0026] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0028] Figure 1 is a flow chart of a method for generating an information retrieval model provided by an embodiment of the present disclosure;

[0029] Figure 2 It is a schematic diagram of the structure and training of the information retrieval model applicable to the embodiments of the present disclosure;

[0030] Figure 3 is a flowchart of another method for generating an information retrieval model provided by an embodiment of the present disclosure;

[0031] Figure 4a is a schematic diagram of the structure of the fusion layer in the information retrieval model applicable to the embodiments of the present disclosure;

[0032] Figure 4b is another structural diagram of the fusion layer in the information retrieval model applicable to the embodiments of the present disclosure;

[0033] Figure 5 is a flow chart of a method for an information retrieval model provided by an embodiment of the present disclosure;

[0034] Figure 6 It is a schematic diagram of the structure of a device for generating an information retrieval model provided by an embodiment of the present disclosure;

[0035] Figure 7 It is a schematic diagram of the structure of an application device of an information retrieval model provided by an embodiment of the present disclosure;

[0036] Figure 8 It is a structural schematic diagram of an electronic device for implementing a method for generating an information retrieval model or a method for applying an information retrieval model provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0037] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0038] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0039] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0040] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0041] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0042] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0043] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0044] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0045] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0046] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0047] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0048] Figure 1 A flowchart of a method for generating an information retrieval model provided in an embodiment of the present disclosure. The technical solution of this embodiment can be applied to situations where tasks with highly refined understanding capabilities are performed, especially situations where corresponding similar text or images need to be found based on text descriptions or given images. The method can be executed by a device for generating an information retrieval model, which can be implemented by software and / or hardware and is generally integrated on any electronic device with network communication capabilities, which can be a mobile terminal, PC or server, etc.

[0049] like Figure 1 As shown, the method for generating the information retrieval model of this embodiment may include but is not limited to the following processes S110-S130:

[0050] S110, determining reference text-image pairs used for information retrieval model training, the information retrieval model being configured and constructed based on a contrastive learning model.

[0051] For some specific business scenarios that require models to have highly refined understanding capabilities, such as the need to perform information retrieval tasks, find images similar to the input text description or text descriptions similar to the given image from a large database based on the input text description or the given image. For example, an e-commerce website may need to find products that meet a specific description among millions of products, and it needs to perform corresponding information retrieval tasks to complete it.

[0052] Since information retrieval requires highly refined understanding capabilities, in this embodiment, a contrastive learning model can be used as a basic framework to configure and construct the information retrieval model to be trained, and the highly refined resolution capabilities of the contrastive learning model for specific tasks can be fully utilized to ensure that the information retrieval model can maintain better refined resolution performance when performing specific tasks. Among them, the contrastive learning model can perform fine classification tasks on images and fine classification tasks on texts associated with images. The information retrieval model is used to realize cross-modal retrieval of text or images, or for cross-modal text or image recommendation.

[0053] In order to train the information retrieval model built based on the contrastive learning model, a reference text-image pair consisting of an image and a text that are related in description content can be obtained. In other words, if the content described by the text is consistent with the content described by the image or meets a preset similarity, the text and the image can be used as a reference text-image pair for training the information retrieval model.

[0054] As an optional but non-limiting implementation, determining the reference text-image pair used for information retrieval model training may include the following steps A1-A2:

[0055] Step A1: determine at least one reference image from a preset image classification data set, and use the category of the reference image in the preset image classification data set as a reference text corresponding to the reference image.

[0056] Step A2: Determine a reference text-image pair used for information retrieval model training by combining a reference image with a corresponding reference text, wherein the reference image included in the reference text-image pair is related to the corresponding reference text in terms of content.

[0057] For the reference text-image pairs used in information retrieval model training, we can first determine a preset image classification database with fine classification, select at least one suitable image from the preset image classification database as a reference image, and then use the category of the reference image recorded in the preset image classification database as the reference text of the reference image, and / or derive a corresponding text description as the corresponding reference text based on the category of the reference image and the description content recorded in the reference image.

[0058] After obtaining a reference image and a reference text paired with the reference image from a preset image classification database, the reference image and the corresponding reference text can be directly paired to form a reference text image pair used in information retrieval model training. The reference text image pairs can be divided into different groups of reference text image pairs, which are used as training sets, verification sets and test sets, respectively, for use in the training stage, verification stage and test stage of the information retrieval model. Optionally, after determining the reference text image pairs used in the information retrieval model training, it also includes: deduplication of the reference text image pairs.

[0059] S120, performing text expansion on the text in the reference text image through a text generation model to obtain a target text sample set, and performing image expansion on the image in the reference text image through an image generation model to obtain a target image sample set.

[0060] The multimodal data pairs on the Internet are too complex, and many multimodal data pairs (such as image-text pairs) are not completely aligned. In fine-scale classification data, such as CUB birds and Stanford cars, there are large differences within the class and small differences between classes. Specifically, fine classification is often divided into different sub-classes under the big class, such as the big class is birds, and the sub-class is which specific bird it is. The multimodal data pairs collected from the Internet are very rough and usually cannot cover such fine image classification.

[0061] At the same time, there is an obvious long-tail effect in the multimodal data pairs on the Internet, which is specifically reflected in the fact that there are very few images in some categories, while there are many images in some head categories. It is not difficult to see that in order to ensure the high-precision task requirements of the information retrieval model, a large amount of training data can be annotated, but due to the very rough multimodal data pairs and the long-tail effect, the generalization ability of the model is still limited, and there will still be the problem of sparse training sample data.

[0062] To this end, a generative big model is introduced in this embodiment, which includes a text generation model and an image generation model. In scenarios where data is sparse or training data is insufficient, the data generation capability of the generative big model is fully utilized. Based on the text and images included in the reference text-image pair, the generative big model is used to enhance and expand the text and images required for information retrieval model training. In this way, the text generation model and the image generation model can provide more and more difficult texts and images for model training.

[0063] By adopting the above method, through the expansion and enhancement of text and images, richer training sample data can be generated to expand data coverage, ensuring that the model used to perform specific tasks can still be trained on a large-scale data set, solving the problem of insufficient or sparse training data for the model, and avoiding as much as possible the problem of creating more training data by means of image rotation, scaling or flipping or text synonym replacement, sentence rearrangement, etc., which requires a lot of computing resources and cannot cover all situations; at the same time, it can reduce the need for manual annotation and data enhancement, improve the accuracy of training data and improve model training efficiency.

[0064] As an optional but non-limiting implementation, the target text sample set includes at least one positive sample that is associated with the text in the reference text image in terms of descriptive content and at least one negative sample that is not associated with the text in the reference text image in terms of descriptive content; the target image sample set includes at least one positive sample that is associated with the image in the reference text image in terms of descriptive content and at least one negative sample that is not associated with the image in the reference text image in terms of descriptive content.

[0065] For example, if the image in the reference text image pair corresponds to an image of "a corgi", and the text in the reference text image pair corresponds to a text description of "a corgi", if the text generation model generates a text description of "a running corgi", it can be seen that the content described is "corgi", and the generated text can be used as a positive sample of the text in the reference text image pair; and if the text generation model generates a text description of "a standing border collie", it can be seen that one of the contents described is "corgi" and the other is "border collie", and the generated text can be used as a negative sample of the text in the reference text image pair. Similarly, if the image generation model generates an image of "a running corgi", it can be seen that the content described is "corgi", and the generated image can be used as a positive sample of the image in the reference text image pair; and if the image generation model generates an image of "a standing border collie", it can be seen that one of the contents described is "corgi" and the other is "border collie", and the generated image can be used as a negative sample of the image in the reference text image pair.

[0066] As an optional but non-limiting implementation, performing text expansion on the text in the reference text image by using a text generation model to obtain a target text sample set may include steps B1-B2:

[0067] Step B1, determine the text generation prompt template used by the text generation model, the text generation prompt template includes a first prompt template and a second prompt template, the first prompt template is used to guide the generation of a text description that is related to the text in the reference text image in terms of descriptive content when the text generation model performs a text generation task, and the second prompt template is used to guide the generation of a text description that is not related to the text in the reference text image in terms of descriptive content when the text generation model performs a text generation task.

[0068] Step B2: Based on the text in the reference text image, a target text sample set is generated by performing text expansion through a text generation model according to a text generation prompt template. The text generation model is pre-configured and generated based on a large language model.

[0069] See also Figure 2 , which is a schematic diagram of the contrastive learning framework assisted by large model generation. The upper part is the input text part, and the lower part is the input image part. For a given image I 0 and its corresponding text (text description of the image) D 0 , and a coding layer F included in the information retrieval model whose parameters are frozen in advance, and the coding layer F has an image coding layer F for the image i and the text encoding layer F for text t , the text data of the corresponding text modality and the image data of the image modality are respectively passed through the text encoding layer and image encoding layer corresponding to their respective modalities, and the embedding vector output of the corresponding modality is obtained, that is:

[0070] T 0 =F t (D 0 )

[0071] V 0 =F i (I 0 )

[0072] Among them, the text generation model of the generative large model can be pre-configured and generated based on a large language model (LLM) that can understand and generate human language, and the image generation model of the generative large model can generate images according to text prompts and modify images according to text descriptions to generate images.

[0073] See also Figure 2 , for the text D in the reference text image pair 0, the text generation model can be used to perform text enhancement under the prompts of the first prompt template and the second prompt template in the text generation prompt template, specifically:

[0074]

[0075]

[0076] In the above formula, θ t represents the model parameters of the text generation model, and P + and P - Represents a text generation prompt template that guides the text generation model to generate text positively and negatively. For the positive prompt template, i.e., the first prompt template, after the text generation model is prompted by the first prompt template, the meaning of the text before and after the rewriting is the same, i.e., the text before and after the rewriting are related in description content, and the positive sample for subsequent comparative learning is equivalent to only data sample enhancement. For the negative prompt template, i.e., the second prompt template, after the text generation model is prompted by the second prompt template, the meaning of the text before and after the rewriting is different, and the meaning of the expression is opposite, i.e., the text before and after the rewriting is no longer related in description content, which is equivalent to the negative sample for subsequent comparative learning.

[0077] As an optional but non-limiting implementation, performing image expansion on the image in the reference text image by using the image generation model to obtain the target image sample set may include steps C1-C2:

[0078] Step C1: Based on the image in the reference text image and at least one positive sample in the target text sample set that is associated with the text in the reference text image in terms of descriptive content, image expansion is performed through an image generation model to generate at least one positive sample in the target image sample set that is associated with the image in the reference text image in terms of descriptive content.

[0079] Step C2: Based on the image in the reference text image and at least one negative sample in the target text sample set that has no relevance in description content to the text in the reference text image, image expansion is performed through an image generation model to generate at least one negative sample in the target image sample set that has no relevance in description content to the image in the reference text image.

[0080] See also Figure 2 , for the reference text image pair image I 0 , the image generation model can be used to generate image I under the guidance of positive and negative samples in the target text sample set generated by the text generation model. 0 Adjustments are made to enhance the image data, specifically:

[0081]

[0082]

[0083] In the above formula, SD() represents the image generation model, θ v Represents the model parameters of the image generation model, a positive sample generated by the text generation model after being prompted by the first prompt template, after the image generation model rewrites the image with the prompt of the positive sample, the images before and after the rewriting remain consistent, that is, the images before and after the rewriting are related in terms of description content, and this is a positive sample for subsequent comparative learning; a negative sample generated by the text generation model after being prompted by the first prompt template, after the image generation model rewrites the image with the prompt of the negative sample, the images before and after the rewriting are not consistent, and the meaning of the expression is opposite, that is, the images before and after the rewriting are no longer related in terms of description content, which is equivalent to a negative sample for subsequent comparative learning.

[0084] In the above manner, the text generation model of the generative large model can perform text data enhancement and expansion based on the text of the reference text image pair according to the prompt to generate positive and negative samples. Similarly, the image generation model of the generative large model can generate images according to the text prompt and modify the image according to the text description to generate images for image data enhancement and expansion to generate positive and negative samples. In this way, an image and its corresponding text description can be expanded into N after the large model. + positive samples and N - Negative samples can provide sufficient data for the training of information retrieval models, solve the problem of sparsity of training data, and further improve the generalization ability and stability of the model by generating richer training samples and expanding data coverage.

[0085] S130 , adjusting parameters of the information retrieval model based on the reference text-image pair, the target text sample set, and the target image sample set to generate a converged information retrieval model.

[0086] After obtaining the reference text-image pairs, the target text sample set and the target image sample set, the text in the reference text-image pairs and the target text sample set can be used as text training data, and the images in the reference text-image pairs and the target image sample set can be used as image training data. On this basis, the corresponding contrastive learning model in the information retrieval model can be trained, and then the parameters of the information retrieval model can be adjusted.

[0087] The technical solution of the disclosed embodiment obtains the reference text-image pairs used in the training of the information retrieval model, introduces the text generation model and the image generation model, performs text expansion and image expansion on the basis of the text and image included in the reference text-image pairs, effectively utilizes the generation capabilities of the text generation model and the image generation model, generates more text descriptions or enhances context information through text expansion and image expansion, and provides more training sample data for the training of the information retrieval model. At the same time, the information retrieval model is configured and constructed based on the contrastive learning model, and the contrastive learning model is used to improve the discrimination ability of the fine classification tasks of specific tasks, and can provide more accurate results in tasks that require the discrimination ability of highly fine classification tasks, thereby achieving the goal of maintaining the generalization ability of the information retrieval model while improving its discrimination performance when processing tasks that require highly fine discrimination, thereby improving the refinement of the model and enhancing the generalization ability of the model.

[0088] Figure 3 A flowchart of another method for generating an information retrieval model provided in an embodiment of the present disclosure. The technical solution of this embodiment further optimizes the process of adjusting parameters of the information retrieval model based on the reference text image pairs, the target text sample set and the target image sample set in the aforementioned embodiment on the basis of the above embodiment. This embodiment can be combined with various optional schemes in one or more of the above embodiments.

[0089] like Figure 3 As shown, the method for generating the information retrieval model of this embodiment may include but is not limited to the following processes S310-S330:

[0090] S310, determining reference text-image pairs used for information retrieval model training, the information retrieval model being configured and constructed based on a contrastive learning model.

[0091] S320, performing text expansion on the text in the reference text image through a text generation model to obtain a target text sample set, and performing image expansion on the image in the reference text image through an image generation model to obtain a target image sample set.

[0092] S330. Input the text in the reference text image pair and the text in the target text sample set into the text encoding layer of the information retrieval model to obtain a first text vector and a second text vector, wherein the first text vector is a vectorized representation of a positive sample that is associated with the text in the reference text image in terms of descriptive content, and the second text vector is a vectorized representation of a negative sample that is not associated with the text in the reference text image in terms of descriptive content.

[0093] See also Figure 2 The above description states that the information retrieval model includes a coding layer F for images. i and the text encoding layer F for textt , the text data of the corresponding text modality and the image data of the image modality are respectively passed through the text encoding layer and image encoding layer corresponding to their respective modalities to obtain the embedding vector output of the corresponding modality.

[0094] See also Figure 2 The text in the reference text image pair and the target text sample set, including at least one positive sample text with relevance in the description content and at least one negative sample text with no relevance in the description content, can be respectively input into the text encoding layer F t The encoding layer is used to extract feature vectors, and at least one first text vector of the vectorized representation of the positive sample that is associated with the text in the reference text image in terms of description content and at least one second text vector of the vectorized representation of the negative sample that is not associated with the text in the reference text image in terms of description content are obtained, as follows:

[0095]

[0096] S340. Input the image in the reference text image pair and the image in the target image sample set into the image coding layer of the information retrieval model to obtain a first image vector and a second image vector, wherein the first image vector is a vectorized representation of a positive sample that is associated with the image in the reference text image in terms of description content, and the second image vector is a vectorized representation of a negative sample that is not associated with the image in the reference text image in terms of description content.

[0097] See also Figure 2 The image in the reference text image pair and the target image sample set, including at least one positive sample image with relevance in the description content and at least one negative sample image with no relevance in the description content, can be input to the image coding layer F respectively. i The encoding layer is used to extract feature vectors, and at least one first image vector of the vectorized representation of the positive sample that is associated with the image in the reference text image in terms of description content and at least one second image vector of the vectorized representation of the negative sample that is not associated with the image in the reference text image in terms of description content are obtained, as follows:

[0098]

[0099] S350, adjusting parameters of a fusion layer in the information retrieval model according to the first text vector and the second text vector and the first image vector and the second image vector to generate a converged information retrieval model.

[0100] The contrast loss of the contrastive learning model of the information retrieval model in contrastive learning training is calculated using the first text vector and the second text vector corresponding to the text generation model, and the first image vector and the second image vector corresponding to the image generation model. The calculated contrastive loss is used to allow the information retrieval model to learn the feature information of the data by shortening the distance of positive samples and displacing negative samples in the feature embedding space, thereby obtaining a converged information retrieval model.

[0101] As an optional but non-limiting implementation, adjusting parameters of the fusion layer in the information retrieval model according to the first text vector and the second text vector and the first image vector and the second image vector may include the following steps D1-D3:

[0102] Step D1, fusing first text vectors that are related in description content through the text fusion layer in the information retrieval model to obtain a first fused vector, and fusing second text vectors that are not related in description content through the text fusion layer in the information retrieval model to obtain a second fused vector.

[0103] Step D2, fusing the first image vectors that are associated in the description content through the image fusion layer in the information retrieval model to obtain a third fusion vector, and

[0104] The image fusion layer in the model fuses the second image vectors that are not related in description content to obtain a fourth fusion vector.

[0105] Step D3: Based on the first fusion vector, the second fusion vector, the third fusion vector and the fourth fusion vector, adjust the parameters of the fusion layer in the information retrieval model.

[0106] See also Figure 2 and Figure 4a One problem with generative models is that uncontrollable errors may occur, that is, the text generated by the text generation model and the image generated by the image generation model may be inaccurate, so incorrect training may occur when using the corresponding text vectors and image vectors. To this end, corresponding text fusion layers and image fusion layers can be added after the text encoding layer and the image encoding layer, respectively. By fusing each text vector through the text fusion layer and fusing each image vector through the image fusion layer, the impact of errors in some text or images on model training can be reduced, and the error problem of some text or images can be improved by fusing multiple vectors.

[0107] See Figure Figure 4b, shows the structure of Transformer Encoder. For the whole fusion layer, there can be L Transformer Encoders stacked together. For the convenience of training, L can be set to 1, that is, one layer of Encoder is sufficient. Figure 4a The input of the text fusion layer can be K first text vectors of the same category (all belonging to positive samples), that is, K first text vectors with correlation in description content; the input of the text fusion layer can also be K second text vectors of the same category (all belonging to negative samples), that is, K second text vectors with correlation in description content. The input of the image fusion layer can be K first image vectors of the same category (all belonging to positive samples), that is, K first image vectors with correlation in description content; the input of the image fusion layer can also be K second image vectors of the same category (all belonging to negative samples), that is, K second image vectors with correlation in description content. After the text fusion layer or the image fusion layer, the fused text vector T output by the MLP Head can be obtained. ′ (including the first fusion vector corresponding to the positive sample text and the first fusion vector corresponding to the negative sample text) or the image vector V ′ (including the third fusion vector corresponding to the positive sample image and the fourth fusion vector corresponding to the negative sample image). The parameters of the text fusion layer and the image fusion layer can be dynamically adjusted as the model is trained, and the model's ability to distinguish fine classification tasks can be enhanced after training.

[0108] As an optional but non-limiting implementation, adjusting parameters of the fusion layer in the information retrieval model based on the first fusion vector, the second fusion vector, the third fusion vector, and the fourth fusion vector may include the following steps E1-E3:

[0109] Step E1, determining first loss information based on the difference between the first fusion vector and the third fusion vector, the first loss information is used to make the text fusion layer and the image fusion layer in the information retrieval model shorten the distance between the positive sample corresponding to the first fusion vector and the positive sample corresponding to the third fusion vector in the feature space.

[0110] Step E2, determining second loss information based on the difference between the second fusion vector and the fourth fusion vector, the second loss information is used to enable the text fusion layer and the image fusion layer in the information retrieval model to push the distance between the negative sample corresponding to the second fusion vector and the negative sample corresponding to the fourth fusion vector in the feature space.

[0111] Step E3: Based on the first loss information and the second loss information, the text fusion layer and the image fusion layer in the information retrieval model are controlled to perform comparative learning training tasks and adjust parameters.

[0112] For extracting the first fusion vector and the third fusion vector, or the second fusion vector and the fourth fusion vector, the first loss information and the second loss information can be calculated by comparing the learned loss function, such as the InfoNCE loss function. In this process, the parameter value is updated by pulling the features of the positive sample pair closer and pushing the features of the negative sample pair farther away.

[0113] Optionally, the loss function used to train the information retrieval model can be the InfoNCE loss in the contrastive learning task, as follows:

[0114]

[0115] Among them, the loss function shown in the above formula actually brings the positive samples of different modes in training closer and the negative samples of different modes farther away, so as to enhance the generalization ability of the model. In the above formula, τ is the temperature coefficient. Specifically, the first loss information is used to bring the distance between the positive sample corresponding to the first fusion vector and the positive sample corresponding to the third fusion vector closer in the feature space, and the second loss information is used to push the distance between the negative sample corresponding to the second fusion vector and the negative sample corresponding to the fourth fusion vector farther in the feature space.

[0116] Optionally, the training environment involved in model training is as follows: operating system: Debian GNU / Linux 10 (buster) x86_64; CPU: Intel Xeon Platinum 8260 (96) @ 3.900 GH; memory: 640; GPU: 8 Tesla A100, each with 80 GB of video memory; CUDA version: 11.0. The training environment is very powerful and suitable for training large-scale and complex visual-language models. There are eight Tesla A100 graphics cards, each with 80 GB of video memory, which can support larger batch sizes and model sizes, speeding up training. At the same time, there is a large memory, which helps to read and process data efficiently.

[0117] Optionally, the training parameters involved in model training are set as follows: Epochs: 30; Batch size: 256; Initial learning rate: 5e-4; Learning rate decay: decay once every 10 epochs, with a decay factor of 0.1; Weight decay: 1e-5; Gradient clipping: 1.0; Input image size: 224x224; Input text length: up to 2048 tokens; Optimization algorithm: Adam.

[0118] The technical solution of the embodiment of the present disclosure obtains the reference text-image pairs used in the training of the information retrieval model, introduces the text generation model and the image generation model, performs text expansion and image expansion on the basis of the text and image included in the reference text-image pairs, effectively utilizes the generation capabilities of the text generation model and the image generation model, generates more text descriptions or enhances context information through text expansion and image expansion to provide more training sample data for the training of the information retrieval model, and at the same time, the information retrieval model is configured and constructed based on the contrastive learning model, and the contrastive learning model is used to improve the discrimination ability of the fine classification tasks of specific tasks, and can provide more accurate results in tasks that require the discrimination ability of highly fine classification tasks. The large model generates rich data to solve the problem of data sparsity, and the contrastive learning model improves the ability to handle high-precision tasks, thereby achieving the goal of maintaining the generalization ability of the information retrieval model while improving its resolution performance when processing tasks that require highly refined resolution, thereby improving the precision of the model and enhancing the generalization ability of the model.

[0119] Figure 5 A flowchart of an application method of an information retrieval model provided in an embodiment of the present disclosure. The technical solution of this embodiment can be applied to situations where tasks with highly refined understanding capabilities are performed, especially situations where corresponding similar text or images need to be found based on text descriptions or given images. The method can be executed by an application device of the information retrieval model, which can be implemented by software and / or hardware and is generally integrated on any electronic device with network communication capabilities, which can be a mobile terminal, PC or server, etc.

[0120] like Figure 5 As shown, the application method of the information retrieval model of this embodiment may include but is not limited to the following processes S510-S520:

[0121] S510: Determine information to be processed, where the information to be processed is text to be processed or an image to be processed.

[0122] The information retrieval model in this embodiment is an information retrieval model obtained by using the information retrieval model generation method described in any of the aforementioned embodiments.

[0123] S520: Using an information retrieval model to search for an image corresponding to the text to be processed from a preset image library; or, using an information retrieval model to search for a text corresponding to the image to be processed from a preset text library.

[0124] The technical solution of the disclosed embodiment obtains the reference text-image pairs used in the training of the information retrieval model, introduces the text generation model and the image generation model, performs text expansion and image expansion on the basis of the text and image included in the reference text-image pairs, effectively utilizes the generation capabilities of the text generation model and the image generation model, generates more text descriptions or enhances context information through text expansion and image expansion, and provides more training sample data for the training of the information retrieval model. At the same time, the information retrieval model is configured and constructed based on the contrastive learning model, and the contrastive learning model is used to improve the discrimination ability of the fine classification tasks of specific tasks, and can provide more accurate results in tasks that require the discrimination ability of highly fine classification tasks, thereby achieving the goal of maintaining the generalization ability of the information retrieval model while improving its discrimination performance when processing tasks that require highly fine discrimination, thereby improving the refinement of the model and enhancing the generalization ability of the model.

[0125] Figure 6 This is a structural block diagram of a device for generating an information retrieval model provided in an embodiment of the present disclosure. The technical solution of this embodiment can be applied to situations where tasks with highly refined understanding capabilities are performed, especially situations where it is necessary to search for corresponding similar text or images based on text descriptions or given images. The device can be implemented by software and / or hardware and is generally integrated on any electronic device with network communication capabilities, which can be a mobile terminal, PC or server, etc.

[0126] like Figure 6 As shown, the information retrieval model generation device of this embodiment may include the following: a determination module 610, an expansion module 620 and a generation module 630. Among them:

[0127] A determination module 610 is used to determine reference text-image pairs used for training an information retrieval model, wherein the information retrieval model is configured and constructed based on a contrastive learning model;

[0128] An expansion module 620, configured to perform text expansion on the text in the reference text image through a text generation model to obtain a target text sample set, and perform image expansion on the image in the reference text image through an image generation model to obtain a target image sample set;

[0129] The generation module 630 is used to adjust the parameters of the information retrieval model based on the reference text-image pair, the target text sample set and the target image sample set to generate a converged information retrieval model.

[0130] Based on the above embodiment, optionally, determining a reference text-image pair used for information retrieval model training includes:

[0131] Determine at least one reference image from a preset image classification data set, and use the category of the reference image in the preset image classification data set as a reference text corresponding to the reference image;

[0132] By combining the reference image with the corresponding reference text, a reference text-image pair used for training the information retrieval model is determined, and the reference image and the corresponding reference text included in the reference text-image pair are related in content.

[0133] Based on the above embodiment, optionally, after determining the reference text-image pair used for information retrieval model training, the method further includes:

[0134] Deduplication of data pairs is performed on the reference text image pairs.

[0135] Based on the above embodiment, optionally, the target text sample set includes at least one positive sample that is associated with the text in the reference text image in terms of descriptive content and at least one negative sample that is not associated with the text in the reference text image in terms of descriptive content; the target image sample set includes at least one positive sample that is associated with the image in the reference text image in terms of descriptive content and at least one negative sample that is not associated with the image in the reference text image in terms of descriptive content.

[0136] Based on the above embodiment, optionally, performing text expansion on the text in the reference text image through a text generation model to obtain a target text sample set includes:

[0137] Determining a text generation prompt template used by the text generation model, the text generation prompt template comprising a first prompt template and a second prompt template, the first prompt template being used to guide the generation of a text description that is associated with the text in the reference text image in terms of description content when the text generation model performs a text generation task, and the second prompt template being used to guide the generation of a text description that is not associated with the text in the reference text image in terms of description content when the text generation model performs a text generation task;

[0138] Based on the text in the reference text image, a target text sample set is generated by performing text expansion through the text generation model according to the text generation prompt template, and the text generation model is pre-configured and generated based on a large language model.

[0139] Based on the above embodiment, optionally, performing image expansion on the image in the reference text image through an image generation model to obtain a target image sample set includes:

[0140] Based on the image in the reference text image and at least one positive sample in the target text sample set that is associated with the text in the reference text image in terms of description content, performing image expansion by the image generation model to generate at least one positive sample in the target image sample set that is associated with the image in the reference text image in terms of description content;

[0141] Based on the image in the reference text image and at least one negative sample in the target text sample set that has no relevance in description content to the text in the reference text image, image expansion is performed through the image generation model to generate at least one negative sample in the target image sample set that has no relevance in description content to the image in the reference text image.

[0142] On the basis of the above embodiment, optionally, adjusting parameters of the information retrieval model based on the reference text-image pair, the target text sample set and the target image sample set includes:

[0143] Inputting the text in the reference text image pair and the text in the target text sample set into the text encoding layer of the information retrieval model to obtain a first text vector and a second text vector, wherein the first text vector is a vectorized representation of a positive sample that is associated with the text in the reference text image in terms of description content, and the second text vector is a vectorized representation of a negative sample that is not associated with the text in the reference text image in terms of description content;

[0144] Inputting the image in the reference text image pair and the image in the target image sample set into the image coding layer of the information retrieval model to obtain a first image vector and a second image vector, wherein the first image vector is a vectorized representation of a positive sample that is associated with the image in the reference text image in terms of description content, and the second image vector is a vectorized representation of a negative sample that is not associated with the image in the reference text image in terms of description content;

[0145] According to the first text vector and the second text vector and the first image vector and the second image vector, parameters of the fusion layer in the information retrieval model are adjusted.

[0146] Based on the above embodiment, optionally, adjusting parameters of the fusion layer in the information retrieval model according to the first text vector and the second text vector and the first image vector and the second image vector includes:

[0147] The first text vectors having correlation in description content are fused through the text fusion layer in the information retrieval model to obtain a first fused vector, and the second text vectors having no correlation in description content are fused through the text fusion layer in the information retrieval model to obtain a second fused vector;

[0148] fusing first image vectors having correlation in description content through the image fusion layer in the information retrieval model to obtain a third fused vector, and fusing second image vectors having no correlation in description content through the image fusion layer in the information retrieval model to obtain a fourth fused vector;

[0149] Based on the first fusion vector, the second fusion vector, the third fusion vector and the fourth fusion vector, parameters of the fusion layer in the information retrieval model are adjusted.

[0150] Based on the above embodiment, optionally, adjusting parameters of the fusion layer in the information retrieval model based on the first fusion vector, the second fusion vector, the third fusion vector, and the fourth fusion vector includes:

[0151] Determine first loss information based on the difference between the first fusion vector and the third fusion vector, wherein the first loss information is used to shorten the distance between the positive sample corresponding to the first fusion vector and the positive sample corresponding to the third fusion vector in the feature space between the text fusion layer and the image fusion layer in the information retrieval model;

[0152] Determine second loss information based on the difference between the second fusion vector and the fourth fusion vector, wherein the second loss information is used to push the distance between the negative sample corresponding to the second fusion vector and the negative sample corresponding to the fourth fusion vector in the feature space between the text fusion layer and the image fusion layer in the information retrieval model;

[0153] According to the first loss information and the second loss information, the text fusion layer and the image fusion layer in the information retrieval model are controlled to perform comparative learning training tasks and adjust parameters.

[0154] The technical solution of the disclosed embodiment obtains the reference text-image pairs used in the training of the information retrieval model, introduces the text generation model and the image generation model, performs text expansion and image expansion on the basis of the text and image included in the reference text-image pairs, effectively utilizes the generation capabilities of the text generation model and the image generation model, generates more text descriptions or enhances context information through text expansion and image expansion, and provides more training sample data for the training of the information retrieval model. At the same time, the information retrieval model is configured and constructed based on the contrastive learning model, and the contrastive learning model is used to improve the discrimination ability of the fine classification tasks of specific tasks, and can provide more accurate results in tasks that require the discrimination ability of highly fine classification tasks, thereby achieving the goal of maintaining the generalization ability of the information retrieval model while improving its discrimination performance when processing tasks that require highly fine discrimination, thereby improving the refinement of the model and enhancing the generalization ability of the model.

[0155] The information retrieval model generation device provided in the embodiments of the present disclosure can execute the information retrieval model generation method provided in any of the above embodiments of the present disclosure, and has the corresponding functions and beneficial effects of executing the information retrieval model generation method. For detailed process, please refer to the relevant operations of the information retrieval model generation method in the above embodiments.

[0156] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present disclosure.

[0157] Figure 7 This is a structural block diagram of an application device of an information retrieval model provided in an embodiment of the present disclosure. The technical solution of this embodiment can be applied to situations where tasks with highly refined understanding capabilities are performed, especially situations where corresponding similar text or images need to be found based on text descriptions or given images. The device can be implemented by software and / or hardware and is generally integrated on any electronic device with network communication capabilities, which can be a mobile terminal, PC or server, etc.

[0158] like Figure 7 As shown, the application device of the information retrieval model of this embodiment may include the following: an input module 710 and a search module 720. Among them:

[0159] The input module 710 is used to determine the information to be processed, and the information to be processed is text to be processed or an image to be processed; wherein, the information retrieval model in this embodiment is an information retrieval model obtained by using the information retrieval model generation method described in any of the aforementioned embodiments.

[0160] The search module 720 is used to use an information retrieval model to search for images corresponding to the text to be processed from a preset image library; or, use an information retrieval model to search for text corresponding to the image to be processed from a preset text library.

[0161] The technical solution of the disclosed embodiment obtains the reference text-image pairs used in the training of the information retrieval model, introduces the text generation model and the image generation model, performs text expansion and image expansion on the basis of the text and image included in the reference text-image pairs, effectively utilizes the generation capabilities of the text generation model and the image generation model, generates more text descriptions or enhances context information through text expansion and image expansion, and provides more training sample data for the training of the information retrieval model. At the same time, the information retrieval model is configured and constructed based on the contrastive learning model, and the contrastive learning model is used to improve the discrimination ability of the fine classification tasks of specific tasks, and can provide more accurate results in tasks that require the discrimination ability of highly fine classification tasks, thereby achieving the goal of maintaining the generalization ability of the information retrieval model while improving its discrimination performance when processing tasks that require highly fine discrimination, thereby improving the refinement of the model and enhancing the generalization ability of the model.

[0162] The application device of the information retrieval model provided in the embodiments of the present disclosure can execute the application method of the information retrieval model provided in any of the embodiments of the present disclosure, and has the corresponding functions and beneficial effects of executing the application method of the information retrieval model. For detailed process, please refer to the relevant operations of the application method of the information retrieval model in the aforementioned embodiments.

[0163] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present disclosure.

[0164] Reference below Figure 8 , which shows a schematic diagram of the structure of an electronic device 800 suitable for implementing the embodiment of the present disclosure. The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 8 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0165] like Figure 8As shown, the electronic device 800 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage device 806 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the electronic device 800 are also stored. The processing device 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0166] Typically, the following devices may be connected to the I / O interface 805: input devices 806 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 807 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 806 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 809. The communication devices 809 may allow the electronic device 800 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 8 The electronic device 800 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0167] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0168] The electronic device provided by the embodiment of the present disclosure and the method for generating an information retrieval model or the method for applying an information retrieval model provided by the above-mentioned embodiment belong to the same inventive concept. For technical details not fully described in this embodiment, reference can be made to the above-mentioned embodiment, and this embodiment has the same beneficial effects as the above-mentioned embodiment.

[0169] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains a program code for executing the method for generating the information retrieval model or the method for applying the information retrieval model shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 809, or installed from the storage device 806, or installed from the ROM 802. When the computer program is executed by the processing device 801, the above-mentioned functions defined in the method for generating the information retrieval model or the method for applying the information retrieval model of the embodiment of the present disclosure are executed.

[0170] The embodiments of the present disclosure provide a computer storage medium on which a computer program is stored. When the program is executed by a processor, the method for generating an information retrieval model or the method for applying an information retrieval model provided in the above embodiments is implemented.

[0171] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0172] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0173] The computer-readable medium may be included in the electronic device, or may exist independently without being installed in the electronic device.

[0174] The above-mentioned computer-readable medium carries one or more programs. When the above-mentioned one or more programs are executed by the electronic device, the electronic device: determines the reference text image pair used for information retrieval model training, and the information retrieval model is configured and constructed based on the contrastive learning model; performs text expansion on the text in the reference text image through a text generation model to obtain a target text sample set, and performs image expansion on the image in the reference text image through an image generation model to obtain a target image sample set; adjusts the parameters of the information retrieval model based on the reference text image pair, the target text sample set and the target image sample set to generate a converged information retrieval model.

[0175] Alternatively, the computer-readable medium carries one or more programs, and when the one or more programs are executed by the electronic device, the electronic device executes the method for generating an information retrieval model or the method for applying an information retrieval model described in any one of the above embodiments.

[0176] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0177] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0178] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a unit does not limit the unit itself in some cases. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses".

[0179] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0180] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0181] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0182] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0183] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A method for generating an information retrieval model, characterized in that: The method comprises: Determining reference text-image pairs for training an information retrieval model, wherein the information retrieval model is configured and constructed based on a contrastive learning model; Performing text expansion on the text in the reference text image through a text generation model to obtain a target text sample set, and performing image expansion on the image in the reference text image through an image generation model to obtain a target image sample set; The parameters of the information retrieval model are adjusted based on the reference text-image pair, the target text sample set and the target image sample set to generate a converged information retrieval model.

2. The method according to claim 1, characterized in that Determine the reference text-image pairs used for information retrieval model training, including: Determine at least one reference image from a preset image classification data set, and use the category of the reference image in the preset image classification data set as a reference text corresponding to the reference image; By combining the reference image with the corresponding reference text, a reference text-image pair used for training the information retrieval model is determined, and the reference image and the corresponding reference text included in the reference text-image pair are related in content.

3. The method according to claim 1 or 2, characterized in that: After determining the reference text-image pairs used for information retrieval model training, it also includes: Deduplication of data pairs is performed on the reference text image pairs.

4. The method according to claim 1, characterized in that: The target text sample set includes at least one positive sample that is associated with the text in the reference text image in terms of descriptive content and at least one negative sample that is not associated with the text in the reference text image in terms of descriptive content; the target image sample set includes at least one positive sample that is associated with the image in the reference text image in terms of descriptive content and at least one negative sample that is not associated with the image in the reference text image in terms of descriptive content.

5. The method according to claim 1 or 4, characterized in that: The text in the reference text image is expanded by a text generation model to obtain a target text sample set, including: Determining a text generation prompt template used by the text generation model, the text generation prompt template comprising a first prompt template and a second prompt template, the first prompt template being used to guide the generation of a text description that is associated with the text in the reference text image in terms of description content when the text generation model performs a text generation task, and the second prompt template being used to guide the generation of a text description that is not associated with the text in the reference text image in terms of description content when the text generation model performs a text generation task; Based on the text in the reference text image, a target text sample set is generated by performing text expansion through the text generation model according to the text generation prompt template, and the text generation model is pre-configured and generated based on a large language model.

6. The method according to claim 1 or 4, characterized in that: The image in the reference text image is expanded by using an image generation model to obtain a target image sample set, including: Based on the image in the reference text image and at least one positive sample in the target text sample set that is associated with the text in the reference text image in terms of description content, performing image expansion by the image generation model to generate at least one positive sample in the target image sample set that is associated with the image in the reference text image in terms of description content; Based on the image in the reference text image and at least one negative sample in the target text sample set that has no relevance in description content to the text in the reference text image, image expansion is performed through the image generation model to generate at least one negative sample in the target image sample set that has no relevance in description content to the image in the reference text image.

7. The method according to claim 1 or 4, characterized in that: Adjusting parameters of the information retrieval model based on the reference text-image pair, the target text sample set, and the target image sample set includes: Inputting the text in the reference text image pair and the text in the target text sample set into the text encoding layer of the information retrieval model to obtain a first text vector and a second text vector, wherein the first text vector is a vectorized representation of a positive sample that is associated with the text in the reference text image in terms of description content, and the second text vector is a vectorized representation of a negative sample that is not associated with the text in the reference text image in terms of description content; Inputting the image in the reference text image pair and the image in the target image sample set into the image coding layer of the information retrieval model to obtain a first image vector and a second image vector, wherein the first image vector is a vectorized representation of a positive sample that is associated with the image in the reference text image in terms of description content, and the second image vector is a vectorized representation of a negative sample that is not associated with the image in the reference text image in terms of description content; According to the first text vector and the second text vector and the first image vector and the second image vector, parameters of the fusion layer in the information retrieval model are adjusted.

8. The method according to claim 7, characterized in that According to the first text vector and the second text vector and the first image vector and the second image vector, adjusting parameters of the fusion layer in the information retrieval model includes: The first text vectors having correlation in description content are fused through the text fusion layer in the information retrieval model to obtain a first fused vector, and the second text vectors having no correlation in description content are fused through the text fusion layer in the information retrieval model to obtain a second fused vector; fusing first image vectors having correlation in description content through the image fusion layer in the information retrieval model to obtain a third fused vector, and fusing second image vectors having no correlation in description content through the image fusion layer in the information retrieval model to obtain a fourth fused vector; Based on the first fusion vector, the second fusion vector, the third fusion vector and the fourth fusion vector, parameters of the fusion layer in the information retrieval model are adjusted.

9. The method according to claim 8, characterized in that Adjusting parameters of a fusion layer in the information retrieval model based on the first fusion vector, the second fusion vector, the third fusion vector, and the fourth fusion vector includes: Determine first loss information based on the difference between the first fusion vector and the third fusion vector, wherein the first loss information is used to shorten the distance between the positive sample corresponding to the first fusion vector and the positive sample corresponding to the third fusion vector in the feature space between the text fusion layer and the image fusion layer in the information retrieval model; Determine second loss information based on the difference between the second fusion vector and the fourth fusion vector, wherein the second loss information is used to push the distance between the negative sample corresponding to the second fusion vector and the negative sample corresponding to the fourth fusion vector in the feature space between the text fusion layer and the image fusion layer in the information retrieval model; According to the first loss information and the second loss information, the text fusion layer and the image fusion layer in the information retrieval model are controlled to perform comparative learning training tasks and adjust parameters.

10. An application method of an information retrieval model, characterized in that: An information retrieval model obtained by using the method for generating an information retrieval model according to any one of claims 1 to 9, the method comprising: Determining information to be processed, wherein the information to be processed is text to be processed or an image to be processed; Using an information retrieval model to search for images corresponding to the text to be processed from a preset image library; or, An information retrieval model is used to search for text corresponding to the image to be processed from a preset text library.

11. A device for generating an information retrieval model, characterized in that: The device comprises: A determination module, used to determine reference text-image pairs used for training an information retrieval model, wherein the information retrieval model is configured and constructed based on a contrastive learning model; An expansion module, configured to perform text expansion on the text in the reference text image through a text generation model to obtain a target text sample set, and to perform image expansion on the image in the reference text image through an image generation model to obtain a target image sample set; A generation module is used to adjust parameters of the information retrieval model based on the reference text-image pair, the target text sample set and the target image sample set to generate a converged information retrieval model.

12. An application device of an information retrieval model, characterized in that: An information retrieval model obtained by using the method for generating an information retrieval model according to any one of claims 1 to 9, wherein the device comprises: An input module, used to determine information to be processed, wherein the information to be processed is text to be processed or an image to be processed; The search module is used to use an information retrieval model to search for images corresponding to the text to be processed from a preset image library; or to use an information retrieval model to search for text corresponding to the image to be processed from a preset text library.

13. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the method for generating an information retrieval model described in any one of claims 1 to 9 or the method for applying an information retrieval model described in claim 10.

14. A computer readable medium, characterized in that The computer-readable medium stores computer instructions, and the computer instructions are used to enable a processor to implement the method for generating an information retrieval model according to any one of claims 1 to 9 or the method for applying an information retrieval model according to claim 10 when executed.

Images