Model training method, image retrieval method, device and electronic equipment

Abstract

Embodiments of the present application disclose a model training method, an image retrieval method, a device, an electronic device and a storage medium. The method comprises: obtaining a training data set, the training data set comprising a plurality of training data, each training data comprising text data and image data; inputting the training data set into a to-be-trained model, obtaining a text vector and an image vector corresponding to each training data output by the to-be-trained model; inputting the training data set into a reference model, obtaining a reference text vector and a reference image vector corresponding to each training data output by the reference model; adjusting parameters of the to-be-trained model based on the text vector, the image vector, the reference text vector and the reference image vector until a training end condition is met, and obtaining a text-image retrieval model. Through the above method, the prediction accuracy of a text-image retrieval model with a small scale is improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of data processing, and particularly relates to a model training method, an image retrieval method, a device, electronic equipment and a storage medium. BACKGROUND

[0002] Cross-modal retrieval is a challenging task in the fields of language and vision. Cross-modal retrieval is to match a given image with text respectively, and select the most similar text as the matching result; or to match a given text with images respectively, and select the most similar image as the matching result.

[0003] Most current methods independently embed images and sentences into a joint space to compare their similarities, and rarely explore the interaction between images and text before calculating the joint space similarity. Therefore, the model trained by such methods is difficult to fully understand the semantic information contained in the image-text pair, thereby affecting the retrieval accuracy of the model. SUMMARY

[0004] In view of the above problems, the application provides a model training method, an image retrieval method, a device, electronic equipment and a storage medium to improve the above problems.

[0005] In a first aspect, an embodiment of the application provides a model training method, which comprises: obtaining a training data set, the training data set comprising a plurality of training data, each training data comprising text data and image data; inputting the training data set into a to-be-trained model to obtain a text vector and an image vector corresponding to each training data output by the to-be-trained model; inputting the training data set into a reference model to obtain a reference text vector and a reference image vector corresponding to each training data output by the reference model, wherein the to-be-trained model and the reference model are used to constitute a knowledge distillation network, the to-be-trained model serving as a student model of the knowledge distillation network, and the reference model serving as a teacher model of the knowledge distillation network; adjusting parameters of the to-be-trained model based on the text vector, the image vector, the reference text vector and the reference image vector until a training end condition is met, to obtain an image-text retrieval model.

[0006] In a second aspect, an embodiment of the application provides an image retrieval method applied to electronic equipment, which comprises: obtaining a query text; inputting the query text into the image-text retrieval model according to any one of claims 1-5 to obtain a text vector corresponding to the query text output by the image-text retrieval model; and obtaining a target image corresponding to the query text from an image library based on a respective image vector of each image in the image library and the text vector, wherein the respective image vector of each image in the image library is output by the image-text retrieval model.

[0007] In a third aspect, an embodiment of the present application provides a model training apparatus, the apparatus comprising: a data acquisition unit configured to acquire a training data set, the training data set comprising a plurality of training data, each training data comprising text data and image data; a vector acquisition unit configured to input the training data set into a to-be-trained model, and acquire a text vector and an image vector corresponding to each training data output by the to-be-trained model; a reference vector acquisition unit configured to input the training data set into a reference model, and acquire a reference text vector and a reference image vector corresponding to each training data output by the reference model, wherein the to-be-trained model and the reference model are configured to constitute a knowledge distillation network, the to-be-trained model serving as a student model of the knowledge distillation network, and the reference model serving as a teacher model of the knowledge distillation network; and a parameter adjustment unit configured to adjust parameters of the to-be-trained model based on the text vector, the image vector, the reference text vector, and the reference image vector until a training end condition is met, and obtain an image-text retrieval model.

[0008] In a fourth aspect, an embodiment of the present application provides an image retrieval apparatus running on an electronic device, the apparatus comprising: a text acquisition unit configured to acquire a query text; a vector acquisition unit configured to input the query text into the image-text retrieval model of any one of claims 1-5, and acquire a text vector corresponding to the query text output by the image-text retrieval model; and an image acquisition unit configured to acquire a target image corresponding to the query text from an image library based on a respective image vector of each image in the image library and the text vector, the respective image vector of each image in the image library being output by the image-text retrieval model.

[0009] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising one or more processors and a memory; one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to perform the method described above.

[0010] In a sixth aspect, an embodiment of the present application provides a computer readable storage medium, the computer readable storage medium storing program code, wherein the program code performs the method described above when the program code is running.

[0011] The embodiment of the present application provides a model training method, an image retrieval method, a device, an electronic device and a storage medium. First, a training data set is acquired, the training data set comprises a plurality of training data, and each training data comprises text data and image data. Then, the training data set is input into a to-be-trained model, text vectors and image vectors corresponding to each training data output by the to-be-trained model are acquired, the training data set is input into a reference model, reference text vectors and reference image vectors corresponding to each training data output by the reference model are acquired, wherein the to-be-trained model and the reference model are used to constitute a knowledge distillation network, the to-be-trained model is used as a student model of the knowledge distillation network, and the reference model is used as a teacher model of the knowledge distillation network. Finally, parameters of the to-be-trained model are adjusted based on the text vectors, the image vectors, the reference text vectors and the reference image vectors until a training end condition is met, and a text-image retrieval model is obtained. Through the above method, a large-scale multi-modal double-flow model is distilled into a small-scale multi-modal double-flow model through a multi-modal knowledge distillation technology, so that the obtained small-scale text-image retrieval model learns the generalization ability of the large-scale reference model, retains the performance close to the reference model, and thus the prediction accuracy of the small-scale text-image retrieval model is improved. BRIEF DESCRIPTION OF DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0013] Figure 1 A flowchart of a model training method according to an embodiment of the present application is shown;

[0014] Figure 2 A flowchart of a model training method according to another embodiment of the present application is shown;

[0015] Figure 3 A schematic diagram of calculating a similarity matrix through text vectors and image vectors in another embodiment of the present application is shown;

[0016] Figure 4 A schematic diagram of calculating a target loss value based on a similarity matrix obtained through text vectors and image vectors output by the same model in another embodiment of the present application is shown;

[0017] Figure 5 A schematic diagram of calculating a target loss value based on a similarity matrix obtained through text vectors and image vectors output by different models in another embodiment of the present application is shown;

[0018] Figure 6 A flow chart of an image retrieval method according to another embodiment of the present application is shown;

[0019] Figure 7 A process diagram of steps S310-S330 according to another embodiment of the present application is shown;

[0020] Figure 8 A structural block diagram of a model training device according to an embodiment of the present application is shown;

[0021] Figure 9 A structural block diagram of an image retrieval device according to an embodiment of the present application is shown;

[0022] Figure 10 A structural block diagram of an electronic device for executing a model training method or an image retrieval method according to an embodiment of the present application is shown;

[0023] Figure 11 A storage unit for storing or carrying program codes for implementing a model training method or an image retrieval method according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0024] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0025] Cross-modal retrieval is a challenging task in the fields of language and vision. Cross-modal retrieval is to match a given image with text respectively, and select the most similar text as the matching result; or to match a given text with images respectively, and select the most similar image as the matching result.

[0026] The inventors found in the research on related model training methods that most current methods independently embed images and sentences into a joint space to compare their similarity, and rarely explore the interaction between images and text before calculating the similarity in the joint space. Therefore, the model trained by such methods is difficult to fully understand the semantic information contained in the image-text pair, thereby affecting the retrieval accuracy of the model.

[0027] Therefore, the inventors have proposed the model training method, image retrieval method, apparatus, electronic device, and storage medium of this application. First, a training dataset is acquired, comprising multiple training data sets, each including text and image data. Then, the training dataset is input into the model to be trained, obtaining the text and image vectors corresponding to each training data set output by the model. The training dataset is then input into a reference model, obtaining the reference text and reference image vectors corresponding to each training data set output by the reference model. The model to be trained and the reference model constitute a knowledge distillation network, with the model to be trained serving as the student model and the reference model serving as the teacher model. Finally, based on the text vectors, image vectors, reference text vectors, and reference image vectors, the parameters of the model to be trained are adjusted until the training termination condition is met, resulting in an image-text retrieval model. Through this method, a large-scale multimodal two-stream model is distilled into a smaller-scale multimodal two-stream model using multimodal knowledge distillation technology. This allows the smaller image-text retrieval model to learn the generalization ability of the larger reference model, retaining performance close to that of the reference model, thereby improving the prediction accuracy of the smaller image-text retrieval model.

[0028] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0029] Please see Figure 1 This application provides a model training method, the method comprising:

[0030] Step S110: Obtain the training dataset, which includes multiple training data sets, each containing text data and image data.

[0031] In the embodiments of this application, image data can be understood as sample images, and text data can be understood as textual description information of sample images.

[0032] One approach is to first acquire multiple sample images during the training of the model, then perform text information mining on these multiple sample images to obtain the text description information of each sample image, thus obtaining the training dataset.

[0033] Alternatively, before training the model, multiple sample images can be collected in advance, and text information mining can be performed on these sample images to obtain textual descriptions of each sample image. Finally, each sample image and its textual description are stored in a designated storage location. When it is time to train the model, these sample images and their textual descriptions are retrieved from this storage location to obtain the training dataset.

[0034] In the embodiments of the present application, text information mining can be performed on each sample image, such as collecting the surrounding text of each sample image through various ways, so as to obtain the text description information of each sample image. For example, the text description information of the image can include one or more of the following: manually annotated image text, machine-automated annotated image text, image text obtained according to the source of the image, image text obtained by performing optical character recognition (OCR) on the image itself, and the like.

[0035] For example, an annotator can perform image description fitting the business scenario for each image. This method is the most accurate, but consumes the most manpower.

[0036] For another example, for a sample image, the existing text description information of the sample image can be mined through big data, such as for an image from Baidu Encyclopedia, the title of the encyclopedia can be used as the surrounding text of the image. This method is relatively good and saves manpower.

[0037] For another example, optical character recognition (OCR) is performed to obtain the text information on the sample image, such as many images contain text information, such as the word "cola" on a bottle of Pepsi. This method can obtain the scene text on the image, and enrich the semantic information of the image.

[0038] For another example, for a sample image, machine-automated image description based on a deep learning model can be performed, such as the image can be automatically described as "a bottle of Maotai liquor under a magnesium light". This method can obtain the entity, its attributes and relationships, and other information on the image, thereby further enriching the semantic information of the image.

[0039] As a way, obtaining the text description information of the sample image can include at least one of the following.

[0040] The annotation information of the sample image is obtained, and the obtained annotation information is taken as the text description information of the sample image. It should be understood that the annotation information of the image can include manually annotated or machine-automated annotated based on a deep learning model, which is not limited here.

[0041] The text information associated with the sample image is obtained, and the obtained text information is taken as the text description information of the sample image. It should also be understood that the text information associated with a specific image can include all other types of text information associated with the specific image, in addition to the information annotated for the image, the text information on the image.

[0042] The text information on the sample image is obtained, and the obtained text information is taken as the text description information of the sample image.

[0043] In the embodiments of the present application, text mining can be performed on each sample image, such as collecting the surrounding text of each sample image through various ways, so as to obtain the text description information of each sample image.

[0044] It should be understood that in the embodiments of the present application, the richer the image text obtained, the more accurate the corresponding image retrieval result can be.

[0045] Step S120: inputting the training data set into the to-be-trained model to obtain the text vector and the image vector corresponding to each training data output by the to-be-trained model.

[0046] In the embodiments of the present application, the text vector is a text feature vector corresponding to the text data in each training data output by the to-be-trained model, and the image vector is an image feature vector corresponding to the image data in each training data output by the to-be-trained model. The dimensions of the text vector and the image vector can be the same or different, which is not limited here.

[0047] As a way, when the training data set is input into the to-be-trained model, the sample image and the text description information thereof in each training data can be input into the to-be-trained model as one-way input information to obtain the text vector and the image vector corresponding to the training data. Alternatively, the sample image and the text description information thereof in each training data can be input into the to-be-trained model as two-way input information to obtain the text vector and the image vector corresponding to the training data.

[0048] In the embodiments of the present application, the to-be-trained model can include a two-way model (also known as a double-flow model) or a one-way model (also known as a single-flow model). The two-way model refers to a model in which the image and the text are input in two ways, for example, the to-be-trained model can be a clip (Contrastive Language-Image Pre-training) model. The one-way model refers to a model in which the image and the text are input in one way. That is, in an embodiment of the present application, when training the to-be-trained model using the sample image and the text description information thereof in the training data set, the sample image and the text description information thereof can be input into the to-be-trained model as one-way input information for training.

[0049] Alternatively, in an embodiment of the present application, when training the to-be-trained model using the sample image and the text description information thereof in the training data set, the sample image and the text description information thereof can also be input into the to-be-trained model as two-way input information for training.

[0050] Step S130: inputting the training data set into a reference model to obtain a reference text vector and a reference image vector corresponding to each training data output by the reference model, wherein the to-be-trained model and the reference model are used to constitute a knowledge distillation network, the to-be-trained model is used as a student model of the knowledge distillation network, and the reference model is used as a teacher model of the knowledge distillation network.

[0051] It can be understood that large-scale machine learning is generally divided into two stages: a training stage and a deployment stage. In the training stage, in order to extract features from a large amount of redundant data set, a large amount of computing resources is usually required, and there is no requirement for real-time performance. The simplest and most effective method to improve performance is ensemble learning, that is, training multiple models in parallel, and then using multiple models to extract data features. When deploying, it is quite costly in terms of computing and memory resources to deploy one or more large models into a device. Knowledge distillation provides a training method that can migrate the knowledge of one or more large models to a small model, facilitating model deployment and speeding up inference. Distillation enables a small model to quickly learn the generalization ability of a large model, that is, knowledge. Knowledge is a very abstract concept, and can be considered as a mapping from an input vector to an output vector.

[0052] In the distillation process, the small model learns the generalization ability of the large model and retains a performance close to that of the large model. The large model can be a single model or an ensemble of multiple models. The distillation method generally first trains a large model (teacher model), then warms up the large model, uses the output of the large model as a soft target, and uses the real label of the data as a hard target, and the two are combined to train a small model (student model).

[0053] In the embodiments of the present application, the to-be-trained model can be used as a student model in the knowledge distillation network, and the reference model can be used as a teacher model in the knowledge distillation network, so as to realize the learning of the to-be-trained model by migrating useful knowledge from the reference network based on the knowledge distillation algorithm.

[0054] As a way, the reference model can also be a double-path model (also known as a double-flow model) or a single-path model (also known as a single-flow model), which is not specifically limited here.

[0055] When the reference model is a single-path model, when the training data set is input into the reference model, the sample image and the text description information in each training data can be input into the reference model as one-way input information to obtain the reference text vector and the reference image vector corresponding to the training data.

[0056] Or, when the reference model is a dual-path model, the sample image and its text description information in each training data can be input into the to-be-trained model as two-path input information respectively, to obtain the reference text vector and the reference image vector corresponding to the training data.

[0057] The reference text vector is a text feature vector corresponding to the text data in each training data output by the reference model, and the reference image vector is an image feature vector corresponding to the image data in each training data output by the reference model. Optionally, the dimensions of the reference text vector and the reference image vector can be the same as the dimensions of the text vector and the image vector.

[0058] Optionally, in order to improve the capability of the lightweight small model (to-be-trained model), a large-scale multi-modal dual-flow model (reference model) can be used as a teacher model. The reference model can be a clip large model.

[0059] Step S140: adjusting the parameters of the to-be-trained model based on the text vector, the image vector, the reference text vector and the reference image vector until a training end condition is met, to obtain a text-image retrieval model.

[0060] In the embodiments of the present application, after obtaining the text vector, the image vector, the reference text vector and the reference image vector of each training data through the to-be-trained model and the reference model, the text vector and the image vector of each training data can be mapped to a common space, and then the similarity between the text vector and the image vector is calculated to obtain a similarity matrix. Then, based on the similarity matrix and the real label corresponding to each training data, a corresponding loss function value can be calculated through contrast learning, and then the parameters of the to-be-trained model can be adjusted in reverse through the loss function value until a training end condition is met, to obtain a text-image retrieval model. The training end condition can be set as that the number of iterations reaches a preset number or the loss function value reaches a target loss value, etc., which is not limited here.

[0061] In the embodiments of the present application, when the parameters of the to-be-trained model are adjusted based on the loss function value, the parameters of the reference model can also be adjusted based on the loss function value. Of course, in order to save the calculation cost, the reference model in the embodiments of the present application can be a large model that has been pre-trained, and then when the parameters of the model are adjusted based on the loss function, only the parameters of the to-be-trained model can be adjusted.

[0062] The model training method provided in the application first acquires a training data set, the training data set includes multiple training data, each training data includes text data and image data, then inputs the training data set into a to-be-trained model, acquires the text vector and the image vector corresponding to each training data output by the to-be-trained model, inputs the training data set into a reference model, acquires the reference text vector and the reference image vector corresponding to each training data output by the reference model, wherein the to-be-trained model and the reference model are used to constitute a knowledge distillation network, the to-be-trained model is used as a student model of the knowledge distillation network, and the reference model is used as a teacher model of the knowledge distillation network, finally, based on the text vector, the image vector, the reference text vector and the reference image vector, the parameters of the to-be-trained model are adjusted until a training end condition is met, and a picture-text retrieval model is obtained. Through the above method, the multi-modal double-flow model with a larger scale is distilled into a multi-modal double-flow model with a smaller scale through the multi-modal knowledge distillation technology, so that the picture-text retrieval model with a smaller scale learns the generalization ability of the reference model with a larger scale, retains the performance close to the reference model, and thus the prediction accuracy of the picture-text retrieval model with a smaller scale is improved.

[0063] Please refer to Figure 2 The model training method provided in the application embodiment includes:

[0064] Step S210: acquire a training data set, the training data set includes multiple training data, each training data includes text data and image data.

[0065] Step S220: input the training data set into a to-be-trained model, and acquire the text vector and the image vector corresponding to each training data output by the to-be-trained model.

[0066] The to-be-trained model includes a text encoder and an image encoder, and the inputting of the training data set into the to-be-trained model and the acquiring of the text vector and the image vector corresponding to each training data output by the to-be-trained model include: inputting the text data in each training data into the text encoder, acquiring the text vector corresponding to each training data output by the text encoder; and inputting the image data in each training data into the image encoder, acquiring the image vector corresponding to each training data output by the image encoder.

[0067] In the embodiments of the present application, the text encoder can be, for example, ERNIE (Ernie Model), TBERT (Bidirectional Encoder Representations from Transformers), etc.; and the image encoder can be, for example, ViT (Vision Transformer), DeiT (Data-efficient image Transformers), etc.

[0068] Specifically, the text data in the training data is input into the text encoder to obtain a text vector corresponding to the training data; and the image data of the training data is input into the image encoder to obtain an image vector corresponding to the training data.

[0069] Step S230: inputting the training data set into the reference model to obtain a reference text vector and a reference image vector corresponding to each training data output by the reference model.

[0070] The reference model includes a reference text encoder and a reference image encoder, and the inputting of the training data set into the reference model to obtain the reference text vector and the reference image vector corresponding to each training data output by the reference model includes: inputting the text data in each training data into the reference text encoder to obtain the reference text vector corresponding to each training data output by the reference text encoder; and inputting the image data in each training data into the reference image encoder to obtain the reference image vector corresponding to each training data output by the reference image encoder.

[0071] In the embodiments of the present application, the reference text encoder can also be, for example, ERNIE (Ernie Model), TBERT (Bidirectional Encoder Representations from Transformers), etc.; and the reference image encoder can also be, for example, ViT (Vision Transformer), DeiT (Data-efficient image Transformers), etc.

[0072] Specifically, the text data in the training data is input into the reference text encoder to obtain a reference text vector corresponding to the training data; and the image data of the training data is input into the reference image encoder to obtain a reference image vector corresponding to the training data.

[0073] Step S240: determining a first image-text similarity matrix based on the text vector and the image vector.

[0074] In the embodiments of the present application, the first text-image similarity matrix is used to represent whether the text data and the image data are a pair.

[0075] As one way, the first text-image similarity matrix is determined based on the text vector and the reference image vector.

[0076] As another way, the first text-image similarity matrix is determined based on the image vector and the reference text vector.

[0077] Step S250: The second text-image similarity matrix is determined based on the reference text vector and the reference image vector.

[0078] In the embodiments of the present application, the second text-image similarity matrix is also used to represent whether the text data and the image data are a pair.

[0079] As one way, the second text-image similarity matrix is determined based on the reference text vector and the image vector.

[0080] As another way, the second text-image similarity matrix is determined based on the reference image vector and the text vector.

[0081] When the text vector, the image vector, the reference text vector and the reference image vector of each training data are obtained, two kinds of similarity matrices can be obtained based on the text vector, the image vector, the reference text vector and the reference image vector of each training data, and each kind of similarity matrix includes two similarity matrices. One kind of similarity matrix can be obtained based on the text vector and the image vector output by the same model, for example, the first text-image similarity matrix can be calculated based on the text vector and the image vector output by the to-be-trained model, and the second text-image similarity matrix can be calculated based on the reference text vector and the reference image vector output by the reference model; and the other kind of similarity matrix can be obtained based on the text vector and the image vector output by different models, for example, the first text-image similarity matrix can be calculated based on the text vector output by the to-be-trained model and the reference image vector output by the reference model, and the second text-image similarity matrix can be calculated based on the image vector output by the to-be-trained model and the reference text vector output by the reference model, or the first text-image similarity matrix can be calculated based on the image vector output by the to-be-trained model and the reference text vector output by the reference model, and the second text-image similarity matrix can be calculated based on the text vector output by the to-be-trained model and the reference image vector output by the reference model.

[0082] For example, the schematic diagram for calculating the similarity matrix based on the text vector and the image vector is shown in FIG. 2. Figure 3 As shown in FIG. 2, the text vector is taken as the row, the image vector is taken as the column, and then the similarity between the elements of each row and the elements in each column is calculated to obtain the corresponding similarity matrix.

[0083] Step S260: determining a target loss value based on the first graph-text similarity matrix, the second graph-text similarity matrix, and a preset graph-text similarity matrix.

[0084] In the embodiments of the present application, the preset graph-text similarity matrix is a true label corresponding to the training data. The target loss value is a weighted sum of multiple loss function values.

[0085] Among them, the loss function values that need to be calculated can include the contrast loss function values of the first graph-text similarity matrix and the second graph-text similarity matrix with the preset graph-text similarity matrix, and the KL divergence loss function values and cross entropy between the first graph-text similarity matrix and the second graph-text similarity matrix.

[0086] For example, the processes shown in steps S240-S260 can be as shown in Figure 4 or Figure 5 . Figure 4 The schematic diagram for calculating the target loss value based on the similarity matrix of the text vector and the image vector output by the same model; Figure 5 The schematic diagram for calculating the target loss value based on the similarity matrix of the text vector and the image vector output by different models. Among them, Figure 4 and Figure 5 The student model in and the teacher model in the embodiments of the present application are the to-be-trained model in the embodiments of the present application, the teacher model is the reference model in the embodiments of the present application, and the ground truth is the preset similarity matrix in the embodiments of the present application. The CE represents the contrast loss function value, and the KL represents the KL divergence loss function value.

[0087] Further, as shown in Figure 4 or Figure 5 In the similarity matrix, the diagonal part in a batch can be used as a positive sample, and the other part can be used as a negative sample. To consider the influence of increasing the amount of negative samples in the batch on the model training result, a queue data structure can be used to store negative samples, so that the same batch can use as many negative samples as possible. In the storage process of the queue, in addition to the image negative features and text negative features obtained by using a small model in the queue, the negative features of the image and text obtained by using a large model can also be stored in the queue, and the same model and cross-modal distillation can be used. Among them, once an image is input into the to-be-trained model or the reference model, the negative features can be stored once.

[0088] Step S270: adjusting the parameters of the to-be-trained model based on the target loss value until a training end condition is met, to obtain the graph-text retrieval model.

[0089] The model training method provided in the application can distill a large-scale multi-modal double-flow model into a small-scale multi-modal double-flow model through a multi-modal knowledge distillation technology, so that the obtained small-scale image-text retrieval model learns the generalization ability of the large-scale reference model, retains the performance close to the reference model, and thus improves the prediction accuracy of the small-scale image-text retrieval model.

[0090] Referring to Figure 6 The image retrieval method provided in the embodiment of the application is applied to an electronic device, and the method comprises:

[0091] Step S310: obtaining a query text.

[0092] In the embodiment of the application, the query text can be a text input by a user in real time in an information input area of the electronic device, or a text received by the electronic device and sent by another electronic device, which is not limited here.

[0093] Step S320: inputting the query text into an image-text retrieval model to obtain a text vector corresponding to the query text output by the image-text retrieval model.

[0094] In the embodiment of the application, the image-text retrieval model is a lightweight small model. The image-text retrieval model can comprise a picture encoder and a text encoder. After the electronic device obtains the query text, the electronic device can input the query text into the text encoder of the image-text retrieval model to obtain a text vector corresponding to the query text.

[0095] Step S330: obtaining a target image corresponding to the query text from an image library based on a respective image vector of each image in the image library and the text vector, wherein the respective image vector of each image in the image library is output by the image-text retrieval model.

[0096] In the embodiment of the application, the target image can be one or more candidate objects with the largest similarity in the image library. Alternatively, the target image can also be one or more candidate objects with a similarity greater than a threshold in the image library.

[0097] The respective image vector of each image in the image library can be obtained through the picture encoder in the image-text retrieval model in advance when the image is stored. When a picture is stored in the image library, the picture is input into the picture encoder of the image-text retrieval model to obtain an image vector corresponding to the picture, and the image vector of the picture is stored. When the image vector of the picture is stored, a corresponding relationship between the picture and the image vector can be established, and then when the target image is obtained, the corresponding target image can be found based on the corresponding relationship.

[0098] When the text vector corresponding to the query text is obtained, the text vector of the query text can be compared with the image vectors respectively corresponding to the images included in the image library to obtain a comparison result, so that the target image corresponding to the query text can be obtained from the image library according to the comparison result.

[0099] As a manner, the target image corresponding to the query text is obtained from the image library based on the text vector and the image vectors respectively corresponding to the images in the image library, comprising: obtaining the similarity between the text vector and the image vectors respectively corresponding to the images in the image library; based on the similarity, the target image corresponding to the query text is obtained from the image library.

[0100] The similarity between the text vector and the image vectors respectively corresponding to the images included in the image library is calculated respectively to obtain a plurality of similarities, and then the images included in the image library can be sorted in order from high to low or from low to high according to the plurality of similarities to obtain a sorted image library.

[0101] When the sorted image library is obtained, the target image corresponding to the query text can be obtained from the sorted image library through the topK function. The similarity between the text vector and the image vectors respectively corresponding to the images in the image library can be cosine similarity.

[0102] When the target image corresponding to the query text is obtained, the image can be displayed. When displaying, if there are multiple target images, the multiple images can be displayed overlapped or one image is displayed every interval of a preset time. Optionally, when the multiple images are displayed overlapped, the images can be displayed overlapped in order from high to low according to the similarity; similarly, when one image is displayed every interval of a preset time, the images can also be displayed in interval in order from high to low according to the similarity, which is not limited here.

[0103] Exemplarily, the method shown in steps S310-S330 can be applied in the photo album product of the electronic device. Specifically, as shown in Figure 7 In the deployment stage, it can be divided into two modules of offline process and real-time process. The offline process performs inference and storage of the image representation vector, and in the real-time process, the representation vector of the query text is inferred, and the similarity between the query text vector and the image library vector is calculated, K similar images are obtained through the topK function, and displayed in the electronic device.

[0104] The image retrieval method provided in the application first acquires a query text, inputs the query text into a picture-text retrieval model, acquires a text vector corresponding to a query file output by the picture-text retrieval model, and then acquires a target image corresponding to the query text from an image library based on the text vector and image vectors corresponding to images in the image library respectively.

[0105] Please refer to Figure 8 The model training device 400 provided in the embodiments of the application comprises:

[0106] The data acquisition unit 410 is configured to acquire a training data set, wherein the training data set comprises a plurality of training data, and each training data comprises text data and image data.

[0107] The vector acquisition unit 420 is configured to input the training data set into a to-be-trained model, and acquire a text vector and an image vector corresponding to each training data output by the to-be-trained model.

[0108] The to-be-trained model comprises a text encoder and an image encoder, and the vector acquisition unit 420 is specifically configured to input the text data in each training data into the text encoder, acquire a text vector corresponding to each training data output by the text encoder, and input the image data in each training data into the image encoder, acquire an image vector corresponding to each training data output by the image encoder.

[0109] The reference vector acquisition unit 430 is configured to input the training data set into a reference model, and acquire a reference text vector and a reference image vector corresponding to each training data output by the reference model, wherein the to-be-trained model and the reference model are used to constitute a knowledge distillation network, the to-be-trained model is used as a student model of the knowledge distillation network, and the reference model is used as a teacher model of the knowledge distillation network.

[0110] The reference model comprises a reference text encoder and a reference image encoder, and the reference vector acquisition unit 430 is specifically configured to input the text data in each training data into the reference text encoder, acquire a reference text vector corresponding to each training data output by the reference text encoder, and input the image data in each training data into the reference image encoder, acquire a reference image vector corresponding to each training data output by the reference image encoder.

[0111] The parameter adjusting unit 440 is configured to adjust the parameters of the to-be-trained model based on the text vector, the image vector, the reference text vector, and the reference image vector until a training end condition is met, to obtain the image-text retrieval model.

[0112] As one way, the parameter adjusting unit 440 is specifically configured to determine a first image-text similarity matrix based on the text vector and the image vector, determine a second image-text similarity matrix based on the reference text vector and the reference image vector, determine a target loss value based on the first image-text similarity matrix, the second image-text similarity matrix, and a preset image-text similarity matrix, and adjust the parameters of the to-be-trained model based on the target loss value until the training end condition is met, to obtain the image-text retrieval model.

[0113] Further, the parameter adjusting unit 440 is specifically configured to determine a first image-text similarity matrix based on the text vector and the reference image vector, and determine a second image-text similarity matrix based on the reference text vector and the image vector.

[0114] Please refer to Figure 9 The embodiment of the present application provides an image-text retrieval device 500, and the device 500 comprises:

[0115] The text acquisition unit 510 is configured to acquire a query text.

[0116] The vector acquisition unit 520 is configured to input the query text into the image-text retrieval model in any one of claims 1-5, and acquire a text vector corresponding to the query text output by the image-text retrieval model.

[0117] The image acquisition unit 530 is configured to acquire a target image corresponding to the query text from an image library based on a respective image vector of each image in the image library and the text vector, wherein the respective image vector of each image in the image library is output by the image-text retrieval model.

[0118] As one way, the image acquisition unit 530 is specifically configured to acquire a similarity between the text vector and a respective image vector of each image in the image library, and acquire the target image corresponding to the query text from the image library based on the similarity.

[0119] It should be noted that the device embodiment in the present application corresponds to the foregoing method embodiment, and the specific principles in the device embodiment can be referred to the content in the foregoing method embodiment, which will not be described here again.

[0120] An electronic device provided by the present application will be described below in combination with 10.

[0121] Please refer toFigure 10 Based on the model training method, the image retrieval method and the device, another electronic device 800 capable of executing the model training method or the image retrieval method is further provided. The electronic device 800 includes one or more (only one is shown in the figure) processors 802, a memory 804 and a network module 806 which are coupled with each other. The memory 804 stores programs capable of executing the content of the foregoing embodiments, and the processor 802 can execute the programs stored in the memory 804.

[0122] The processor 802 can include one or more processing cores. The processor 802 connects various parts in the electronic device 800 through various interfaces and lines, executes various functions of the electronic device 800 and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 804 and calling data stored in the memory 804. Optionally, the processor 802 can be implemented in at least one of a digital signal processing (DSP), a field-programmable gate array (FPGA) and a programmable logic array (PLA). The processor 802 can be integrated with a combination of one or more of a central processing unit (CPU), a graphics processing unit (GPU) and a modem. The CPU is mainly used to process operating systems, user interfaces and application programs; the GPU is used to render and draw display content; and the modem is used to process wireless communication. It can be understood that the modem can also not be integrated into the processor 802, but can be implemented by a separate communication chip.

[0123] The memory 804 can include a random access memory (RAM) and a read-only memory (ROM). The memory 804 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 804 can include a program storage area and a data storage area. The program storage area can store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playing function, an image playing function, etc.), instructions for implementing the various method embodiments described below, etc. The data storage area can also store data created by the electronic device 800 in use (such as a phonebook, audio and video data, chat record data, etc.).

[0124] The network module 806 is configured to receive and send electromagnetic waves, and to convert the electromagnetic waves and electrical signals to each other, so as to communicate with a communication network or other devices, for example, an audio playing device. The network module 806 can include various existing circuit elements for performing these functions, for example, an antenna, a radio frequency transceiver, a digital signal processor, an encryption / decryption chip, a subscriber identity module (SIM) card, a memory, and the like. The network module 806 can communicate with various networks such as the Internet, an intranet, a wireless network, or other devices through the wireless network. The wireless network can include a cellular telephone network, a wireless local area network or metropolitan area network. For example, the network module 806 can interact with a base station.

[0125] Reference is made to Figure 11 which shows a structural block diagram of a computer readable storage medium provided by an embodiment of the present application. The computer readable storage medium 900 stores program codes therein, which can be invoked by a processor to execute the methods described in the above method embodiments.

[0126] The computer readable storage medium 900 can be an electronic storage such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk or a ROM. Alternatively, the computer readable storage medium 900 includes a non-transitory computer readable storage medium. The computer readable storage medium 900 has a storage space for program codes 910 to execute any of the above methods. These program codes can be read from or written to one or more computer program products. The program codes 910 can be compressed in a suitable form, for example.

[0127] The model training method, the image retrieval method, the device, the electronic equipment and the storage medium provided by the present application first acquire a training data set, the training data set includes multiple training data, each training data includes text data and image data, then input the training data set into a to-be-trained model, acquire the text vector and the image vector corresponding to each training data output by the to-be-trained model, input the training data set into a reference model, acquire the reference text vector and the reference image vector corresponding to each training data output by the reference model, wherein the to-be-trained model and the reference model are used to constitute a knowledge distillation network, the to-be-trained model is used as a student model of the knowledge distillation network, and the reference model is used as a teacher model of the knowledge distillation network, finally, based on the text vector, the image vector, the reference text vector and the reference image vector, the parameters of the to-be-trained model are adjusted until a training end condition is met, and a picture-text retrieval model is obtained. Through the above method, the multi-modal double-flow model with a larger scale is distilled into a multi-modal double-flow model with a smaller scale through the multi-modal knowledge distillation technology, so that the picture-text retrieval model with a smaller scale learns the generalization ability of the reference model with a larger scale, retains the performance close to the reference model, and thus the prediction accuracy of the picture-text retrieval model with a smaller scale is improved.

[0128] The embodiments of the present application are described above in combination with the drawings, but the present application is not limited to the above specific embodiments, and the above specific embodiments are only illustrative but not restrictive. Those skilled in the art can make many forms under the inspiration of the present application without departing from the purpose of the present application and the scope protected by the claims, and all of them belong to the protection of the present application.

Claims

1. A model training method, characterized in that, The method comprises: obtaining a training data set, the training data set comprising a plurality of training data, each training data comprising text data and image data; inputting the training data set into a to-be-trained model to obtain a text vector and an image vector corresponding to each training data output by the to-be-trained model; inputting the training data set into a reference model to obtain a reference text vector and a reference image vector corresponding to each training data output by the reference model, wherein the to-be-trained model and the reference model are used to constitute a knowledge distillation network, the to-be-trained model serving as a student model of the knowledge distillation network, and the reference model serving as a teacher model of the knowledge distillation network; determining a first text-image similarity matrix based on the text vector and the image vector; determining a second text-image similarity matrix based on the reference text vector and the reference image vector; determining a target loss value based on the first text-image similarity matrix, the second text-image similarity matrix, and a preset text-image similarity matrix, the target loss value comprising a contrast loss function value of the first text-image similarity matrix and the second text-image similarity matrix with the preset text-image similarity matrix, and a KL divergence loss function value and a cross entropy between the first text-image similarity matrix and the second text-image similarity matrix; adjusting parameters of the to-be-trained model based on the target loss value until a training end condition is met to obtain a text-image retrieval model.

2. The method of claim 1, wherein, The to-be-trained model comprises a text encoder and an image encoder, and the inputting the training data set into the to-be-trained model to obtain a text vector and an image vector corresponding to each training data output by the to-be-trained model comprises: inputting the text data in each training data into the text encoder to obtain a text vector corresponding to each training data output by the text encoder; inputting the image data in each training data into the image encoder to obtain an image vector corresponding to each training data output by the image encoder.

3. The method of claim 1, wherein, The reference model comprises a reference text encoder and a reference image encoder, and the inputting the training data set into the reference model to obtain a reference text vector and a reference image vector corresponding to each training data output by the reference model comprises: inputting the text data in each training data into the reference text encoder to obtain a reference text vector corresponding to each training data output by the reference text encoder; inputting the image data in each training data into the reference image encoder to obtain a reference image vector corresponding to each training data output by the reference image encoder.

4. The method of claim 1, wherein, The method further comprises: determining a first text-image similarity matrix based on the text vector and the reference image vector; determining a second text-image similarity matrix based on the reference text vector and the image vector.

5. An image retrieval method characterized by, An application electronic device comprises: obtaining a query text; inputting the query text into the text-image retrieval model of any one of claims 1-4 to obtain a text vector corresponding to the query text output by the text-image retrieval model; obtain, based on the text vector and image vectors respectively corresponding to images in an image library, a target image corresponding to the query text from the image library, the image vectors respectively corresponding to the images in the image library being output by the image-text retrieval model.

6. The method of claim 5, wherein, The obtaining, based on the text vector and image vectors respectively corresponding to images in an image library, a target image corresponding to the query text from the image library includes: obtaining a similarity between the text vector and the image vectors respectively corresponding to the images in the image library; obtaining, based on the similarity, the target image corresponding to the query text from the image library.

7. A model training apparatus characterized by comprising: The apparatus includes: a data obtaining unit configured to obtain a training data set, the training data set including a plurality of training data, each training data including text data and image data; a vector obtaining unit configured to input the training data set into a to-be-trained model to obtain a text vector and an image vector corresponding to each training data output by the to-be-trained model; a reference vector obtaining unit configured to input the training data set into a reference model to obtain a reference text vector and a reference image vector corresponding to each training data output by the reference model, wherein the to-be-trained model and the reference model are used to constitute a knowledge distillation network, the to-be-trained model serving as a student model of the knowledge distillation network, and the reference model serving as a teacher model of the knowledge distillation network; a parameter adjusting unit configured to determine a first image-text similarity matrix based on the text vector and the image vector, determine a second image-text similarity matrix based on the reference text vector and the reference image vector, determine a target loss value based on the first image-text similarity matrix, the second image-text similarity matrix, and a preset image-text similarity matrix, the target loss value including a contrast loss function value of the first image-text similarity matrix and the second image-text similarity matrix with the preset image-text similarity matrix, and a KL divergence loss function value and a cross entropy between the first image-text similarity matrix and the second image-text similarity matrix, and adjust parameters of the to-be-trained model based on the target loss value until a training end condition is met to obtain an image-text retrieval model.

8. An electronic device, comprising: one or more processors; and one or more programs stored in the memory and configured to be executed by the one or more processors to perform the method of any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program code, wherein when the program code is executed by a processor, the method of any one of claims 1-6 is performed.

Images