Model training method and device, computer equipment and storage medium

Abstract

The invention discloses a model training method and device, computer equipment and a storage medium, and the method comprises the steps: carrying out the mask processing of a sample image according to the entity information in the sample image, and obtaining a mask image; masking a field corresponding to the entity information in the sample text to obtain a masked text; performing text mapping processing on the mask image to obtain a first mapping text; combining the first mapping text and the mask text to obtain a combined text; and training the first model according to the sample image and the combined text to obtain a second model for image retrieval. According to the method, the mask image and the mask text are obtained by performing mask processing on the sample image and the sample text, so that the mask text and the mapping text corresponding to the model processing image can be combined, the combined text and the original sample image before mask are utilized to train the model, and the model training efficiency is improved on the basis of ensuring the model training effect. The collection difficulty of training samples is reduced, and the training effect of the model is improved.

Description

Technical Field

[0001] The present application relates to the field of image retrieval, and specifically to a model training method, apparatus, computer equipment, and storage medium. Background Art

[0002] Image retrieval refers to a retrieval technology that retrieves images related to user input information from an image database. In order to improve the effect of image retrieval, a combined image retrieval (CIR) based on image and text has been proposed. CIR mainly uses a text mapping network and a pre-trained visual language model to achieve the effect of retrieving the target image through the query image and query text.

[0003] Currently, most combined image retrieval technologies rely on supervised training strategies, which train text-mapping models by constructing a ternary training set consisting of a query image, query text, and target image. However, collecting ternary training sets is not convenient because, in most cases, combined image retrieval aims to perform image retrieval using images and text in a complementary manner. This means that the query image and query text information must be able to correspond to the target image after complementation. Therefore, if the query image and query text in the ternary training set represent the same content or contain too much information unrelated to the target image, this will affect the training of the text-mapping network during supervised training, resulting in the trained text-mapping network being unable to be effectively applied to subsequent combined image retrieval. Summary of the Invention

[0004] In response to the above technical problems, the embodiments of the present application provide a model training method, apparatus, computer equipment and storage medium, aiming to solve the problem of unsatisfactory text mapping network training results in related technologies.

[0005] In a first aspect, an embodiment of the present application provides a model training method, comprising:

[0006] performing mask processing on the sample image according to entity information in the sample image to obtain a mask image containing the entity information;

[0007] Masking the fields corresponding to the entity information in the sample text to obtain masked text, wherein the sample text is used to describe the sample image;

[0008] Performing text mapping processing on the first image feature of the mask image to obtain a first mapping text corresponding to the mask image;

[0009] Combining the first mapping text and the mask text to obtain a combined text;

[0010] The first model is trained according to the sample image and the combined text to obtain a second model for image retrieval.

[0011] In a second aspect, an embodiment of the present application provides a model training device, comprising:

[0012] a mask module, configured to perform mask processing on the sample image according to entity information in the sample image to obtain a mask image containing the entity information;

[0013] The masking module is further configured to mask the fields corresponding to the entity information in the sample text to obtain masked text, wherein the sample text is used to describe the sample image;

[0014] a processing module, configured to perform text mapping processing on the first image feature of the mask image to obtain a first mapped text corresponding to the mask image;

[0015] a combining module, configured to combine the first mapping text and the mask text to obtain a combined text;

[0016] The training module is used to train the first model according to the sample image and the combined text to obtain a second model for image retrieval.

[0017] In a third aspect, an embodiment of the present application further provides a computer device comprising a processor and a memory, wherein the memory stores a plurality of instructions; the processor loads instructions from the memory to execute the steps of model training as described above.

[0018] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores a plurality of instructions, and the instructions are suitable for loading by a processor to execute the steps of model training as described above.

[0019] In a fifth aspect, an embodiment of the present application further provides a computer program product, which includes a computer program, and the computer program is executed by a processor to perform the steps of model training as described above.

[0020] Compared to using a ternary training group that is more difficult to collect to train the model, in this application, the model training will be completed by selecting a binary training group that is easier to collect, that is, an image-text pair, and by further using the entity information in the sample image to perform corresponding masking on the sample image and the sample text, a mask image containing entity information and a mask text that masks the field of the corresponding entity information are obtained, so as to form a mask image-text pair that is complementary in content. At the same time, the original sample image before masking can also be used as a complementary target image to correspond to the mask image-text pair that is complementary in content, so that in the subsequent training process, by combining the mask text and the mapping text corresponding to the masked image, the model can be trained using the combined text and the original sample image before masking, so that the model can focus on information related to the retrieval task, thereby extracting information useful for retrieval, and effectively completing the training of the model through binary training samples that are easier to collect. Through the above method, the difficulty of collecting training samples can be reduced on the basis of effectively ensuring the training effect of the model, thereby improving the training effect of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 A schematic flow chart of the steps of a model training method provided in an embodiment of the present application;

[0023] Figure 2 A schematic flow chart of the steps of performing mask processing on a sample image provided in an embodiment of the present application;

[0024] Figure 3 A schematic diagram of a process flow for training a first model provided in an embodiment of the present application;

[0025] Figure 4 A schematic diagram of a flow chart of steps for calculating a model loss value based on similarity between an image and text provided in an embodiment of the present application;

[0026] Figure 5 Another example of a process flow for training a model based on a model loss value is provided in an embodiment of the present application;

[0027] Figure 6 A schematic diagram of a process flow for performing image retrieval using a trained model provided in an embodiment of the present application;

[0028] Figure 7 A schematic diagram of the structure of a model training device provided in an embodiment of the present application;

[0029] Figure 8 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of this application. At the same time, in the description of the embodiments of the present application, the terms "first", "second", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more features. In the description of the embodiments of the present application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0031] In order to better understand the model training method, device, computer equipment and storage medium provided by this application, the relevant application background of the model training method is first explained. Specifically, the model training method provided by this application is mainly used in the field of image retrieval, and is particularly suitable for the field of combined image retrieval, that is, the retrieval of images is completed through the query image and query text input by the user. Among them, combined image retrieval is usually achieved through a text mapping network and a pre-trained visual language model. Among them, the pre-trained visual language model can adopt a pre-trained image encoding model or a text encoding model, and the text mapping network is mainly used to map the image into a pseudo-word text (text without substantive meaning) in order to combine with the query text to complete the retrieval of the image. Therefore, in the combined image retrieval technology, the training effect of the text mapping network is crucial.

[0032] In current technology, most combined image retrieval techniques rely on supervised training strategies, that is, training the text mapping model by constructing a ternary training set containing the query image, query text, and target image. However, in actual application, it is found that collecting ternary training sets is not convenient because, in most cases, combined image retrieval techniques tend to improve retrieval results by complementing images and text. For example, for unique colors or other information without common names, it is often more direct to use images to display them, while the relationship between two objects in the image is more clear through the use of verbs or prepositions in the text. Therefore, the constructed triples usually require that the query image and query text correspond to the target image after complementation. If the query image and query text represent the same content or contain too much information unrelated to the target image, the training effect of the text mapping network will be affected to a certain extent, thereby affecting the subsequent combined image retrieval effect.

[0033] It is precisely to solve the above problems that this application provides a model training method. By using image-text pairs that are easier to collect and performing appropriate masking on the image-text, the masked image-text information is made complementary, and after complementation, it can correspond to the original sample image to complete the training of the text mapping network. While effectively ensuring the training effect of the text mapping network, it reduces the difficulty of collecting sample training data, thereby effectively improving the efficiency of model training. Specifically, to facilitate understanding of the above content, the following will provide a specific explanation of the model training method.

[0034] For details, please refer to Figure 1 , Figure 1 A schematic flow chart of a model training method provided in an embodiment of the present application, specifically comprising steps S110 to S150:

[0035] S110 , performing mask processing on the sample image according to entity information in the sample image to obtain a mask image containing the entity information.

[0036] S120: Mask the fields corresponding to the entity information in the sample text to obtain masked text.

[0037] The sample image in the embodiment of the present application generally refers to the image in the image-text pair used to train the model, that is, the sample image and the sample text correspond to the same content, that is, the sample text is used to describe the sample image. Compared with the conventional method of constructing a ternary training group, the method of collecting sample image-text pairs in the embodiment of the present application is more convenient. For example, as a feasible implementation scheme, sample images can be collected from a network platform through network technology means, such as crawler technology, and then the descriptive text corresponding to the image is obtained as sample text by manual annotation or input into a trained large model. Of course, it is also possible to directly extract the labeled image-text pairs directly from the model training database disclosed by the network platform. The embodiment of the present application does not limit the method of obtaining sample images and sample texts.

[0038] In order to realize the training of the text mapping network in the combined image retrieval, in the embodiment of the present application, the sample image will be masked by the entity information in the sample image to obtain a mask image containing the entity information. Specifically, there are many ways to obtain the entity information in the sample image. For example, the entity in the image can be identified by image recognition, so as to further determine the entity area containing the entity in the image and the background area that does not contain the entity. At this time, by making a random mask in the background area that does not contain the entity in the sample image, for example, setting the pixel value of a random part of the pixel points in the background area to 0, the sample image can be masked to obtain a mask image containing complete entity information while partially covering up other irrelevant entity information, so as to be better used for subsequent model training.

[0039] Of course, in order to better implement the masking of the sample image to obtain the mask image for training the model, as another feasible implementation scheme of this application, please refer to Figure 2 , a schematic flow chart of steps for performing mask processing on a sample image is also provided, specifically, comprising steps S210 to S220:

[0040] S210 , obtaining a heat map of a sample image, wherein the pixel value of each pixel in the heat map is used to represent the probability that the corresponding pixel in the sample image is the entity information.

[0041] In the embodiments of the present application, the heatmaps (CAMs) of the sample images, also known as class activation maps, class heatmaps, and saliency maps, are similar to heatmaps generated by thermal imagers. The heatmaps of the sample images can show key areas in the image that indicate entities. Specifically, the different color information of the pixels in the heatmap can be used to indicate the possibility that the corresponding pixel in the image indicates a certain category of entity. That is, the pixel value of each pixel in the heatmap is used to represent the probability that the corresponding pixel in the sample image is entity information.

[0042] S220 , masking the area of ​​the sample image excluding the entity information based on the pixel value of each pixel point in the heat map to obtain a plurality of different mask images containing the entity information.

[0043] In order to achieve image masking and better train the model, unlike randomly masking the background area that does not contain entities in the sample image, the technical solution provided in the embodiment of the present application will take into account the pixel value of each pixel in the heat map, that is, the probability that the corresponding pixel in the sample image is entity information, to mask the background area in the sample image other than the entity information, thereby obtaining multiple different mask images containing entity information. The masked parts of these mask images in the background area are different, but on the whole, for pixels with lower probabilities, the probability of being masked during the masking process is higher, that is, the pixel will be obscured in more mask images.

[0044] By training the text mapping network using the multiple different mask images obtained above, the ability of the text mapping network to focus on information related to the retrieval task can be further improved, that is, information useful for retrieval can be more accurately extracted, thereby improving the effect of subsequent combined image retrieval.

[0045] On the basis of the above, since the sample image and the sample text correspond to the same content, that is, there is also a field corresponding to the entity information in the sample text, in order to achieve the complementarity of the image and text to ensure the training effect of the model, the field corresponding to the entity information in the sample text can be masked to obtain a masked text that covers up the entity information, that is, the masked text and the masked image are complementary in terms of the included content, and after the masked text and the masked image are complementary, they can correspond to the original image before masking processing.

[0046] Specifically, to facilitate understanding of the above content, take the sample text "A pony running on the grassland" and the sample image describing the scene of a pony running on the grassland as an example. At this time, by masking the areas other than "pony" in the sample image, a masked image can be obtained, and by masking the field "pony" corresponding to the entity information in the sample text, the masked text can be obtained, that is, "A [DELETE] running on the grassland", where the [DELETE] character represents the field removed by the mask.

[0047] S130: Perform text mapping processing on the first image feature of the mask image to obtain a first mapped text corresponding to the mask image.

[0048] On the basis of the mask image and mask text obtained above, in order to realize the training of the text mapping network, in the solution provided in the embodiment of the present application, the first image feature of the mask image will be subjected to text mapping processing to obtain the first mapping text corresponding to the mask image. For example, as a feasible implementation scheme, it can usually be through an initial text mapping network that has the ability to map images to pseudo-words, that is, the first model that needs to be trained to perform text mapping processing on the first image feature of the mask image to obtain the first mapping text. Among them, the initial text mapping network usually refers to the parameters in the network being randomly initialized.

[0049] Furthermore, the first image feature of the mask image can usually be obtained by processing the mask image through the image encoder in the pre-trained visual language model. The specific network structure can be set based on actual needs. For example, a deep convolutional neural network based on the residual network (ResNet) can be adopted. Of course, ViT (Vision Transformer, visual compilation module) can also be selected to implement it. Since the image encoder used in the embodiment of the present application has been pre-trained, no additional training parameters are required. Therefore, this application does not limit the choice of image encoder. Any coding structure that can encode the image to extract image features is within the scope of protection required by this application.

[0050] S140: Combine the first mapping text and the mask text to obtain a combined text.

[0051] In an embodiment of the present application, the first mapping text and the mask text obtained by processing the mask image through the first model are combined. For example, the [DELETE] character in the mask text can be replaced by the first mapping text to obtain the combined text, that is, the combined text for describing the original image obtained by fusing the mask image and the mask text. It can be understood that the more ideal the effect of the text mapping network is, the more the combined text can reflect the characteristics of the original image before masking.

[0052] S150: Train the first model according to the sample image and the combined text to obtain a second model for image retrieval.

[0053] In the embodiment of the present application, combined with the above-mentioned related descriptions, it can be seen that since the effect of the text mapping network will affect the effect of the obtained combined text, it is possible to consider using the difference between the combined text and the sample image as the loss value of the model, and use the reverse update algorithm to train the initial text mapping network, that is, the first model that needs to be trained, until the final trained second model that can be used for image retrieval is obtained.

[0054] For example, as a possible implementation, see Figure 3 , Figure 3 A flowchart of the steps for training the first model provided in an embodiment of the present application is described in detail as follows.

[0055] In the embodiment of the present application, the sample image and the combined text are processed respectively by the image encoder and the text encoder in the pre-trained visual language model to obtain corresponding image features and text features. It can be understood that the more ideal the effect of the text mapping network is, the higher the similarity between the image features and the text features should be. Therefore, the similarity between the image features and the text features can be used to construct a model loss value to train the first model. Specifically, the process includes steps S310 to S340:

[0056] S310: Perform image encoding on the sample image to obtain a second image feature of the sample image.

[0057] S320: Perform text encoding on the combined text to obtain a first text feature of the combined text.

[0058] In an embodiment of the present application, regarding the image encoding processing of the sample image, the second encoding feature of the sample image obtained can be similar to the first image feature obtained by the encoding processing of the mask image in the aforementioned step S130. Specifically, it can be obtained by processing the sample image through an image encoder trained in a pre-trained visual language model. For example, it can be implemented by using a deep convolutional neural network based on a residual network or a VIT model. The embodiments of the present application will not be repeated here.

[0059] In addition, for the text encoding of the combined text, the combined text can be processed by a text encoder trained in a pre-trained visual language model to obtain the first text feature of the combined text. Among them, the text encoder can be a variant based on the BERT (Bidirectional Encoder Representations from Transformers) model, or any other model that can effectively convert text into a feature vector. Through such an encoder, the semantic information in the text can be converted into a numerical form, which is convenient for subsequent model training and processing.

[0060] S330: Determine a first model loss value of the first model based on the similarity between the second image feature and the first text feature.

[0061] In an embodiment of the present application, after obtaining the second image feature of the sample image and the first text feature of the combined text, the similarity between the two features will be further calculated to determine the first model loss value of the first model. Among them, the similarity can be calculated using a variety of methods, such as cosine similarity, Euclidean distance, or a similarity calculation method based on an attention mechanism. By calculating the similarity, the degree of matching between the image features and the text features can be evaluated, thereby reflecting the effect of the first model mapping the image to text. Specifically, if the similarity is high, it means that the combined text obtained by the model's image mapping processing and the mask text can well describe the content of the original sample image, that is, the image retrieval can be completed subsequently, and the model loss value is small. On the contrary, if the similarity is low, it means that the model's loss value is large, and the model needs to be further trained and optimized.

[0062] Of course, it is understandable that the calculation of the model loss value will affect the training effect of the model. Therefore, as another feasible embodiment of this application, another implementation scheme for determining the model loss value based on the two dimensions of text and image and image and text is also provided. Figure 4 , Figure 4 A flowchart of a step for calculating a model loss value based on the similarity between an image and text provided in an embodiment of the present application, specifically including steps S410 to S430:

[0063] S410: Determine a first similarity between the transposed second image feature and the first text feature.

[0064] S420: Determine a second similarity between the transposed first text feature and the second image feature.

[0065] Unlike the conventional method of directly calculating the similarity between image features and text features through Euclidean distance, cosine similarity, etc. to determine the model loss value, in the embodiment of the present application, the similarity in two directions, from text to image and from image to text, is fully considered. Specifically, considering that image features and text features are usually in the form of vectors or matrices, in the embodiment of the present application, the image features or text features can be transposed to obtain the similarity with the other feature, that is, the first similarity in the direction from image to text is determined by the transposition of the second image feature and the similarity between the first text feature, and the second similarity in the direction from text to image is determined by the transposition of the first text feature and the similarity between the second image feature. Such a two-way similarity evaluation can more comprehensively reflect the performance of the model, more deeply explore the intrinsic connection between image and text, and then more accurately calculate the model loss value, effectively guiding the optimization process of the model. In addition, through this bidirectional similarity evaluation method, we can not only improve the performance of the model in a single direction, but also promote the flexibility of mutual conversion between images and text, and enhance the generalization ability of the model in multimodal tasks such as multimodal image retrieval tasks.

[0066] S430: Determine a first model loss value of the first model according to the first similarity and the second similarity.

[0067] In an embodiment of the present application, after obtaining the first similarity from the image to the text and the second similarity from the text to the image, the first model loss value of the first model can be obtained by weighted fusion of the first similarity and the second similarity. As a feasible implementation scheme, weighted fusion can refer to calculating the average similarity of the first similarity and the second similarity. Of course, in specific application scenarios, different weight coefficients can be assigned to the first similarity and the second similarity in combination with the representation content of the image and text, so as to more accurately determine the model loss value.

[0068] Specifically, in order to facilitate the understanding of the implementation scheme of the above-mentioned calculation model loss value, the following will combine the specific calculation formula to give the specific calculation process of the calculation model loss value. Specifically, the first model loss value L QT The calculation formula is as follows:

[0069]

[0070] in, That is, it represents the similarity from image to text (i.e., the abbreviation of image to text in the superscript i2t). It represents the similarity from text to image (i.e. t2i in the superscript, abbreviation of text to image). and The calculation formula is as follows:

[0071]

[0072] Among them, QT is query-target, from query to target, that is, L QT It refers to the loss value between the query input and the target output. In this embodiment, it refers to the loss value between the sample image and the combined text. N represents the number of image-text pairs processed in a batch or a round, img refers to the sample image, and C refers to the combined text. i and j represent the i-th group of images / texts and the j-th group of images / texts, respectively. For example, and Represent the second image feature of the i-th sample image and the first text feature of the i-th combined text, respectively. and They represent the second image feature of the jth sample image and the first text feature of the jth combined text, log is the logarithm symbol, exp is the natural exponential function, and T represents the transposition symbol, for example Represents the transpose of the second image feature of the i-th sample image.

[0073] S340: Train the first model according to the first model loss value to obtain a second model for image retrieval.

[0074] After determining the initial model loss, the model is optimized using the backpropagation algorithm. Specifically, the model parameters are adjusted based on the model loss, so that the trained model can more accurately map the mask image to a pseudo-word text. When combined with the mask text, the text describing the original sample image can be obtained.

[0075] Of course, it should be noted that the above process usually relies on multiple rounds of training of the model, that is, in each round of training, it is necessary to iteratively update the parameters in the model and gradually reduce the calculated model loss value until the model reaches a relatively ideal training state. For example, after several consecutive rounds of detecting that the calculated model loss value is less than the preset threshold, it can be considered that the model has converged to a relatively satisfactory performance level and the training can be terminated. At this time, the currently trained model is the second model, which can be used for subsequent image retrieval.

[0076] Of course, on the basis of the aforementioned scheme for training the model, in order to better realize the calculation of the model loss value, as another feasible embodiment of the present application, the model loss value used for model training, in addition to the first model loss value calculated by the image features of the original image and the text features of the combined text mentioned above, can also generally include a second model loss value between the second text features of the second mapping text obtained after the original sample image is processed by the first model and the second image features of the original sample image. Specifically, the second model loss value can reflect the consistency between the text after the image is processed by the model and the original image, and can be further used to guide the optimization of the model. Specifically, to facilitate understanding of the above content, as shown in FIG. Figure 5 As shown, Figure 5 Another flowchart of the steps for training a model based on a model loss value provided in an embodiment of the present application includes steps S510 to S540:

[0077] S510: Process the second image feature of the sample image to obtain a second mapping text corresponding to the sample image.

[0078] In addition to the aforementioned mapping of the mask image to pseudo-word text and combining it with the mask text to obtain a combined text, and determining the first loss value of the model by comparing the similarity between the combined text and the original image, similar to the aforementioned processing of the mask image, in an embodiment of the present application, the sample image will also be directly processed by the aforementioned first model to be trained, that is, the text mapping network with randomly initialized parameters, to obtain the pseudo-word text corresponding to the sample image, that is, the second mapping text.

[0079] S520: Perform text encoding on the second mapping text to obtain a second text feature of the second mapping text.

[0080] In the embodiment of the present application, similar to the aforementioned step S320 in which the combined text is processed using the text encoder trained in the pre-trained visual language model, the second mapped text is subjected to corresponding text encoding processing by the text encoder, and the second text features of the second mapped text can also be obtained. It can be understood that the second text features processed by the first model should describe the image information in the original sample image. Therefore, the obtained second text features can also be used to describe another loss value of the first model in the text mapping process. Details are as follows.

[0081] S530: Determine a second model loss value of the first model according to the similarity between the second image feature and the second text feature.

[0082] Similar to determining the first loss value of the model by the similarity between the first text feature of the combined text and the second image feature of the original sample image, in the embodiment of the present application, the second model loss value of the first model is also determined according to the similarity between the second text feature of the second mapped text and the second image feature of the original sample image, wherein, for the description of the similarity, please refer to the aforementioned step S330 and Figure 4 For example, it can be obtained by calculating cosine similarity or Euclidean distance. Of course, in the specific calculation process, it can usually be obtained by considering the similarity from the image to the text and the similarity from the text to the image. Specifically, the step of determining the second model loss value of the first model based on the similarity between the second image feature and the second text feature generally includes:

[0083] determining a third similarity between the transpose of the second image feature and the second text feature;

[0084] determining a fourth similarity between the transpose of the second text feature and the second image feature;

[0085] A second model loss value of the first model is determined according to the third similarity and the fourth similarity.

[0086] Specifically, for ease of understanding, the second model loss value L CIR , which is the loss value of combined retrieval used in combined image retrieval CIR technology, is calculated as follows:

[0087]

[0088] in, That is, it represents the similarity from image to text (i.e., superscript i2t). It represents the similarity from text to image (i.e. superscript t2i), and The calculation formula is as follows:

[0089]

[0090] Where N represents the number of image-text pairs processed in a batch or a round, img describes the image, and txt refers to the mapped text obtained by text mapping the sample image, i and j represent the i-th group of image / text and the j-th group of image / text, respectively. For example, and They represent the second image feature of the i-th sample image and the second text feature of the i-th second mapping text, respectively. and They represent the second image feature of the jth sample image and the second text feature of the jth second mapping text, log is the logarithm symbol, exp is the natural exponential function, and T represents the transposition symbol, for example Represents the transpose of the second image feature of the i-th sample image.

[0091] S540: Train the first model according to the first model loss value and the second model loss value to obtain a second model for image retrieval.

[0092] Different from the aforementioned method of training the model only through the first model loss value, in the embodiment of the present application, a second model loss value is further introduced so that the model can more comprehensively capture the mapping relationship between the image and the text during the training process. Through the optimization of this dual loss value, the model can deeply explore the intrinsic connection between the image and the text, and improve the accuracy and efficiency of image retrieval. Among them, regarding the process of training the model, you can refer to the relevant instructions of the aforementioned step S340. For example, the back propagation algorithm is used to adjust the parameters in the model. By iteratively updating the parameters in the model in each round of training, the calculated model loss value is gradually reduced until the model reaches a relatively ideal training state, and the currently trained model is determined as the second model for subsequent image retrieval.

[0093] Compared to using a ternary training group that is more difficult to collect to train the model, in this application, the model training will be completed by selecting a binary training group that is easier to collect, that is, an image-text pair, and by further using the entity information in the sample image to perform corresponding masking on the sample image and the sample text, a mask image containing entity information and a mask text that masks the field of the corresponding entity information are obtained, so as to form a mask image-text pair that is complementary in content. At the same time, the original sample image before masking can also be used as a complementary target image to correspond to the mask image-text pair that is complementary in content, so that in the subsequent training process, by combining the mask text and the mapping text corresponding to the masked image, the model can be trained using the combined text and the original sample image before masking, so that the model can focus on information related to the retrieval task, thereby extracting information useful for retrieval, and effectively completing the training of the model through binary training samples that are easier to collect. Through the above method, the difficulty of collecting training samples can be reduced on the basis of effectively ensuring the training effect of the model, thereby improving the training effect of the model.

[0094] Of course, it should be noted that after the model training is completed and the second model for image retrieval is obtained, the image to be queried is usually mapped into text by the second model and then combined with the query text to obtain the combined query text, thereby completing the image retrieval. For details, please refer to Figure 6 , Figure 6 A flowchart of the steps of performing image retrieval using a trained model provided in an embodiment of the present application, specifically including steps S610 to S630:

[0095] S610: Input the query image into the second model to obtain a third mapping text corresponding to the query image.

[0096] S620: Combine the third mapping text and the text to be queried to obtain a combined query text.

[0097] S630: Determine, from the candidate images, retrieval results corresponding to the image to be queried and the text to be queried based on the third text feature of the combined query text and the third image feature of the candidate images in a preset database.

[0098] Of course, in order to more clearly understand the complete solution provided by the embodiment of the present application, the following will be combined with the above Figures 1 to 6 The content provided provides the complete implementation steps of the model training method provided in the embodiment of this application and the subsequent model application process, specifically including the following steps.

[0099] (1) Collect sample image-sample text pairs, where the sample text can be understood as the descriptive text of the sample image. For example, if the sample image is a scene of a pony running on the grassland, the sample text can be "a pony running on the grassland"; if the sample image is a scene of a kitten basking in the sun on the beach, the sample text can be "a kitten basking in the sun on the beach", etc. Specifically, the sample image-sample text pairs can be directly collected from the network platform through network technology means, or collected from a public database dedicated to model training. Of course, an image can also be input into a trained large language model for recognition to generate a corresponding descriptive text as the sample text corresponding to the image. For the convenience of subsequent description, I n Represents the original sample image, with T n Represents the original sample text;

[0100] (2) Through the original sample image I n The category heat map can determine the areas in the image that are related to and unrelated to a certain category entity. For example, taking the sample image of a pony running on the grassland as an example, you can choose "pony" as the entity object. nRandomly mask the areas unrelated to the “pony” to obtain the corresponding mask image At the same time in T n Mask the characters of "xiaoma" in the text, for example, replace them with [DELETE], and get the masked text with "xiaoma" removed. Such as "a horse [DELETE] ran on the grassland";

[0101] (3) Using the mask image obtained above and mask text Constructing composite text The details are as follows:

[0102] First, use the image encoder trained in the pre-trained visual language model to mask the image Processing to obtain the first image feature of the mask image That is to say Among them, E image That is, it represents the image encoding process of the image encoder;

[0103] Then, the text mapping network is used, which is the first model to be trained to transform the first image feature of the mask image Mapping into a pseudo-word text (only has the format of a word, has no meaning itself, and can be regarded as a fixed-length one-dimensional feature vector composed of characters) to obtain the first mapping text The calculation formula for the first mapping text is: in, This is the text mapping process of the text mapping network. Specifically, unlike the aforementioned image encoder, the text mapping network is a lightweight n-layer MLP (multi-layer perceptron) that needs to be trained. That is, in the embodiment of the present application, it is necessary to perform multiple rounds of iterative optimization training on the parameters in the first model until a trained text mapping network is obtained, which is the second model that can be used for image retrieval.

[0104] Finally, the first mapping text and mask text Combination, for example, the first mapping text Replace the mask text The [DELETE] character in the text box can be used to get the final combined text.

[0105] (4) In addition to combined text In addition, the original sample image I can be encoded using the image encoder. n Processing to obtain the second image feature of the sample image Right now Then the second image feature After processing by the text mapping network, the second mapping text corresponding to the original sample image can also be obtained Right now The second mapping text and combined text The difference is that the second mapping text is the direct text mapping result of the original sample image, and the combined text It is the text mapping result after the mask information is integrated. In addition, it is necessary to further combine the text and the second mapping text After processing by the text encoder trained in the pre-trained visual language model, the corresponding combined text is obtained The first text feature and the second mapping text The second text feature Right now Among them E text That is, it represents the text encoding process of the text encoder;

[0106] (5) Constructing a loss function. In order to better train the text mapping network, that is, the first model, a loss function consisting of two parts can be used. Specifically, it can include the first model loss value L mentioned above. QT , used to indicate the loss value between the constructed query information (the combined text after mask combination) and the query result (the original sample image), as well as the second model loss value L CIR , which is used to indicate the loss of the text mapping network itself, that is, the specific calculation formula of the model loss value L is as follows:

[0107] L=αL QT +βL CIR

[0108] Among them, α and β are two hyperparameters used to control the weights of the two losses. Their specific settings can be set based on actual needs. For example, in the early stage of model training, by setting a relatively high value of α and a relatively low value of β, the model can quickly learn the ability of image retrieval, that is, the model can focus more on the image retrieval task in the early stage of training. In the later stage of model training, by gradually reducing the value of α and appropriately increasing the value of β, the model's image retrieval ability can be further guaranteed while improving its own text mapping ability.

[0109] In addition, L QT and L CIR The calculation formula is as follows:

[0110]

[0111] For detailed descriptions of the parameters in the above formula, please refer to the related explanations of the above steps S430 and S530. For example, and That corresponds to the second image feature and the first text feature The encoding features corresponding to the i-th sample image and combined text;

[0112] (6) Using the loss function calculated above, the model parameters in the first model are optimized using the idea of ​​back propagation, and then the image-sample pairs are re-input into the model for processing to obtain an updated model loss value. After a certain round of updates, the updated model loss value is less than a preset threshold for several consecutive rounds. The model can be considered to have been trained and optimized. At this point, the currently updated model can be regarded as a trained model and used as the second model for image retrieval, so as to be used in subsequent image retrieval processing.

[0113] (7) After the model is trained, the model is deployed in the corresponding image retrieval platform. At this time, when the user inputs the query image and query text, the query image will be processed by the trained text mapping network, that is, the second model, to obtain the third mapping text corresponding to the query image. The third mapping text and the query text are then combined to obtain the combined query text, and the text encoding features of the combined query text are obtained through the text encoder. By comparing the text encoding features with the image encoding features of the images in the database to be queried, the target images with higher similarity between the image encoding features and the text encoding features of the combined query text can be screened out from the database to be queried as the final query results and output.

[0114] This application also provides a model training device for executing the aforementioned model training method. For details, please refer to Figure 7 , Figure 7 A schematic diagram of the structure of a model training device provided in an embodiment of the present application, specifically comprising:

[0115] A masking module 710 is configured to perform masking on the sample image according to entity information in the sample image to obtain a mask image containing the entity information;

[0116] The masking module 710 is further configured to mask the fields corresponding to the entity information in the sample text to obtain masked text; the sample text is used to describe the sample image;

[0117] A processing module 720 is configured to perform text mapping processing on the first image feature of the mask image to obtain a first mapped text corresponding to the mask image;

[0118] A combining module 730, configured to combine the first mapped text and the masked text to obtain a combined text;

[0119] The training module 740 is used to train the first model according to the sample image and the combined text to obtain a second model for image retrieval.

[0120] As a feasible embodiment of the present application, the training module 740 is further configured to: perform image encoding on the sample image to obtain a second image feature of the sample image;

[0121] Performing text encoding on the combined text to obtain a first text feature of the combined text;

[0122] determining a first model loss value of the first model based on a similarity between the second image feature and the first text feature;

[0123] The first model is trained according to the first model loss value to obtain a second model for image retrieval.

[0124] As a feasible embodiment of the present application, the training module 740 is further configured to: determine a first similarity between the transposed second image feature and the first text feature;

[0125] determining a second similarity between the transpose of the first text feature and the second image feature;

[0126] A first model loss value of the first model is determined according to the first similarity and the second similarity.

[0127] As a feasible embodiment of the present application, the training module 740 is further configured to: process the second image feature of the sample image using the first model to obtain a second mapping text corresponding to the sample image;

[0128] Performing text encoding on the second mapped text to obtain a second text feature of the second mapped text;

[0129] determining a second model loss value of the first model according to a similarity between the second image feature and the second text feature;

[0130] The first model is trained according to the first model loss value and the second model loss value to obtain a second model for image retrieval.

[0131] As a feasible embodiment of the present application, the training module 740 is further configured to: determine a third similarity between the transposed second image feature and the second text feature;

[0132] determining a fourth similarity between the transpose of the second text feature and the second image feature;

[0133] A second model loss value of the first model is determined according to the third similarity and the fourth similarity.

[0134] As a feasible embodiment of the present application, the mask module 710 is further used to obtain a heat map of the sample image, and the pixel value of each pixel point in the heat map is used to represent the probability that the corresponding pixel point in the sample image is the entity information;

[0135] Based on the pixel value of each pixel point in the heat map, the area of ​​the sample image except the entity information is masked to obtain a plurality of different mask images containing the entity information.

[0136] As a feasible embodiment of the present application, the training module 740 is further configured to: input the query image into the second model to obtain a third mapping text corresponding to the query image;

[0137] Combining the third mapping text and the text to be queried to obtain a combined query text;

[0138] According to the third text feature of the combined query text and the third image feature of the candidate images in the preset database, a retrieval result corresponding to the image to be queried and the text to be queried is determined from the candidate images.

[0139] In one embodiment, the present application further provides a computer device, wherein the computer device is taken as a terminal device as an example, and its internal structure diagram can be as follows: Figure 8As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface, the display unit and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a model training method is implemented. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse, etc.

[0140] Those skilled in the art will understand that Figure 8 The structure shown is only a block diagram of a portion of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0141] Based on the same inventive concept, an embodiment of the present application also provides a computer-readable storage medium, which may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0142] Since the computer program stored in the computer-readable storage medium can execute any of the model training methods provided in the embodiments of the present application, the beneficial effects that can be achieved by any of the model training methods provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here. For example, the computer program stored in the computer-readable storage medium can execute the following steps:

[0143] performing mask processing on the sample image according to entity information in the sample image to obtain a mask image containing the entity information;

[0144] Masking the fields corresponding to the entity information in the sample text to obtain masked text; the sample text is used to describe the sample image;

[0145] Performing text mapping processing on the first image feature of the mask image to obtain a first mapping text corresponding to the mask image;

[0146] Combining the first mapping text and the mask text to obtain a combined text;

[0147] The first model is trained according to the sample image and the combined text to obtain a second model for image retrieval.

[0148] Based on the same inventive concept, embodiments of the present application further provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in various optional implementations of the above embodiments.

[0149] It should be noted that the object data (including but not limited to user device information, user personal information, etc.) and conversation data involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions. Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods.

[0150] Any reference to the memory, database or other media used in the various embodiments provided herein may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0151] The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may include, but are not limited to, general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic units, data processing logic units based on quantum computing, and the like.

[0152] In the above-mentioned model training device, computer equipment, computer-readable storage medium, and computer program product embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments. Those skilled in the art will clearly understand that for the convenience and brevity of description, the specific working processes and beneficial effects of the above-described model training device, computer-readable storage medium, computer program product, computer equipment, and corresponding units can refer to the description of the model training method in the above embodiments, and will not be repeated here.

[0153] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0154] The above is a detailed introduction to a model training method, device, computer equipment and storage medium provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, based on the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A model training method, characterized in that: include: performing mask processing on the sample image according to entity information in the sample image to obtain a mask image containing the entity information; Masking the fields corresponding to the entity information in the sample text to obtain masked text, wherein the sample text is used to describe the sample image; Performing text mapping processing on the first image feature of the mask image to obtain a first mapping text corresponding to the mask image; Combining the first mapping text and the mask text to obtain a combined text; The first model is trained according to the sample image and the combined text to obtain a second model for image retrieval.

2. The method according to claim 1, characterized in that The step of training the first model according to the sample image and the combined text to obtain a second model for image retrieval includes: performing image encoding on the sample image to obtain a second image feature of the sample image; Performing text encoding on the combined text to obtain a first text feature of the combined text; determining a first model loss value of the first model based on a similarity between the second image feature and the first text feature; The first model is trained according to the first model loss value to obtain a second model for image retrieval.

3. The method according to claim 2, characterized in that Determining a first model loss value of the first model according to the similarity between the second image feature and the text feature includes: determining a first similarity between the transpose of the second image feature and the first text feature; determining a second similarity between the transpose of the first text feature and the second image feature; A first model loss value of the first model is determined according to the first similarity and the second similarity.

4. The method according to claim 2, characterized in that The method further comprises: Processing the second image feature of the sample image to obtain a second mapping text corresponding to the sample image; Performing text encoding on the second mapped text to obtain a second text feature of the second mapped text; determining a second model loss value of the first model according to a similarity between the second image feature and the second text feature; The step of training the first model according to the first model loss value to obtain a second model for image retrieval includes: The first model is trained according to the first model loss value and the second model loss value to obtain a second model for image retrieval.

5. The method according to claim 4, characterized in that Determining a second model loss value of the first model according to the similarity between the second image feature and the second text feature includes: determining a third similarity between the transpose of the second image feature and the second text feature; determining a fourth similarity between the transpose of the second text feature and the second image feature; A second model loss value of the first model is determined according to the third similarity and the fourth similarity.

6. The method according to claim 1, characterized in that The processing of the sample image according to the entity information in the sample image to obtain a mask image containing the entity information includes: Obtaining a heat map of the sample image, where the pixel value of each pixel in the heat map is used to represent the probability that the corresponding pixel in the sample image is the entity information; Based on the pixel value of each pixel point in the heat map, the area of ​​the sample image except the entity information is masked to obtain a plurality of different mask images containing the entity information.

7. The method according to any one of claims 1 to 6, characterized in that After the first model is trained according to the sample image and the combined text to obtain the second model for image retrieval, the method further includes: Inputting the query image into the second model to obtain a third mapping text corresponding to the query image; Combining the third mapping text and the text to be queried to obtain a combined query text; According to the third text feature of the combined query text and the third image feature of the candidate images in the preset database, a retrieval result corresponding to the image to be queried and the text to be queried is determined from the candidate images.

8. A model training device, characterized in that: include: a mask module, configured to perform mask processing on the sample image according to entity information in the sample image to obtain a mask image containing the entity information; The masking module is further configured to mask the fields corresponding to the entity information in the sample text to obtain masked text, wherein the sample text is used to describe the sample image; a processing module, configured to perform text mapping processing on the first image feature of the mask image to obtain a first mapped text corresponding to the mask image; a combining module, configured to combine the first mapping text and the mask text to obtain a combined text; The training module is used to train the first model according to the sample image and the combined text to obtain a second model for image retrieval.

9. A computer device, characterized in that: It includes a processor and a memory, wherein the memory stores multiple instructions; the processor loads instructions from the memory to execute the steps of the model training method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a plurality of instructions, which are suitable for loading by a processor to execute the steps of the model training method according to any one of claims 1 to 7.

11. A computer program product, characterized in that The computer program product includes a computer program, and the computer program is used by a processor to execute the steps of the model training method according to any one of claims 1 to 7.

Images