Method and device for training student image-text comparison model, medium and electronic equipment

Abstract

The embodiment of the invention discloses a method and device for training a student image-text comparison model, a medium and electronic equipment, and the method comprises the steps: inputting first image data without labels into a trained teacher image-text comparison model, and obtaining a first output result corresponding to the first image data; inputting the first image data and a corresponding trainable first image prompt word into an image encoder in a student image-text comparison model corresponding to the teacher image-text comparison model to obtain an output target image feature; multiplying the target image features by text features pre-stored in the training process of the trained teacher image-text comparison model to obtain a second output result corresponding to the first image data; and performing knowledge distillation on the student image-text comparison model according to the first output result and the second output result to obtain a trained student image-text comparison model.

Description

Technical Field

[0001] The present invention relates to computer technology, and in particular to a method, device, medium and electronic equipment for training a picture-text comparison model for students. Background Art

[0002] With the continuous development of artificial intelligence technology and the Internet, the performance of machine learning models has been continuously improved thanks to the massive Internet image and text data. The image-text comparison model (image-text comparison learning base model) based on the pre-training of massive image-text pairs can be applied to downstream tasks when there is no downstream task data or a small amount of downstream task data involved in fine-tuning. Among them, the more representative image-text comparison model is the CLIP (Contrastive Language-Image Pre-training) model, which is based on the image-text pairs (the image and the corresponding text describe the same content) data on the Internet, and compares the image representation model with the text representation model. The distance between the image and text representation of the same description content is close in the latent space, and the distance between the image and text representation of different description content is far in the latent space, so that the image representation and the text representation can be aligned in the same latent space. After the pre-training is completed, it can be directly applied to various downstream image classification tasks. It only needs to input the image into the image representation model, the category into the text representation model and compare them in the latent space.

[0003] Although CLIP's pre-trained image and text representation models can be applied to downstream tasks, their generalization ability may be limited for certain specific, highly specialized tasks or domains, especially when the data distribution is significantly different from the data distribution used for pre-training. CLIP may perform poorly in understanding complex contexts or text descriptions with implicit meanings because its training focuses more on the association between visual content and direct descriptive text. For tasks that require fine-grained distinctions, CLIP may not be able to distinguish very similar categories without further training, such as different types of dogs with similar appearance. A simple solution is to post-train a linear classifier based on the CLIP image representation model and fine-tune it on downstream tasks, but it may overfit on small-scale annotated datasets, resulting in insufficient generalization. Summary of the invention

[0004] The purpose of the embodiments of this specification is to provide a method, device, storage medium and electronic device for training students' picture-text comparison model.

[0005] The embodiments of this specification provide a method for training a student image-text comparison model, and propose a distillation scheme for an image-text comparison model based on a trainable prompt. The scheme includes a high-parameter and high-precision image-text comparison model (called a teacher model) and a low-parameter image-text comparison model to be trained (called a student model). The scheme aims to distill the knowledge of the teacher model into the student model by training the trainable prompt words embedded in the student model. The student model after distillation has higher accuracy than the retrained model and a small number of parameters, and can replace the heavy teacher model for business deployment. In the distillation process, this scheme further proposes a method for reusing teacher text features. In particular, after the pre-training of the teacher model is completed, the text features output by its text encoder are stored, and the image features output by the student model are multiplied by the text features of the teacher model to obtain the final prediction result. This calculation method significantly saves the calculation amount of the text encoder part of the student model, and the high-quality features output by the teacher model also ensure the prediction performance of the student. Due to the existence of a pre-trained teacher model, this solution can be further extended to unlabeled data, using the teacher model to provide soft labels for unlabeled data to train the student model, which can save a lot of labeling costs. At the same time, extensive unlabeled data training can make the student model have higher performance. The advantage of this solution is that compared with the traditional model full parameter distillation method, the distillation method based on the student model prompt words proposed in this solution requires fewer parameters to be trained and has high training efficiency. This solution reduces the dependence on training data and can achieve good training results with a small amount of data training. The method includes:

[0006] Inputting the unlabeled first image data and the corresponding first text data into the trained teacher image-text comparison model to obtain a first output result corresponding to the first image data;

[0007] Inputting the first image data and the corresponding trainable first image prompt word into an image encoder in a student image-text contrast model corresponding to the teacher image-text contrast model to obtain the target image features outputted by the encoder;

[0008] Multiplying the target image feature with the text feature pre-stored in the training process of the trained teacher image-text comparison model to obtain a second output result corresponding to the first image data;

[0009] The student image-text comparison model is enabled to perform knowledge distillation according to the first output result and the second output result to obtain a trained student image-text comparison model, wherein the trained student image-text comparison model is used to output a corresponding image prediction result according to the input target image data to be predicted.

[0010] Furthermore, the method further comprises:

[0011] Inputting the labeled second image data and the corresponding second text data into the teacher image-text comparison model to obtain a third output result corresponding to the second image data;

[0012] The teacher image-text comparison model is trained based on the third output result and the true value label corresponding to the second image data to obtain a trained teacher image-text comparison model.

[0013] Furthermore, the step of inputting the labeled second image data and the corresponding second text data into the teacher image-text comparison model to obtain a third output result corresponding to the second image data includes:

[0014] Input the labeled second image data into the image encoder in the teacher image-text comparison model to obtain the corresponding second image features;

[0015] The second text data corresponding to the second image data is input into the text encoder in the teacher image-text comparison model to obtain the corresponding second text feature, and the second image feature is multiplied by the second text feature to obtain a third output result corresponding to the second image data.

[0016] Furthermore, the method further comprises:

[0017] After the teacher image-text comparison model is trained, well-trained second text features are pre-stored.

[0018] Furthermore, the step of inputting the first image data and the corresponding trainable first image prompt word into the image encoder in the student image-text comparison model corresponding to the teacher image-text comparison model to obtain the output target image features includes:

[0019] Obtaining a plurality of first vector representations corresponding to the first image data and a plurality of second vector representations corresponding to the trainable first image prompt words corresponding to the first image data;

[0020] The multiple first vector representations and the multiple second vector representations are cascaded and input into the image encoder in the student image-text contrast model corresponding to the teacher image-text contrast model to obtain the target image features of its output.

[0021] Furthermore, the cascading of the multiple first vector representations and the multiple second vector representations into an image encoder in a student image-text contrast model corresponding to the teacher image-text contrast model to obtain the output target image features includes:

[0022] Cascading the multiple first vector representations and the multiple second vector representations, and inputting them into an image encoder in a student image-text contrast model corresponding to the teacher image-text contrast model to obtain a first image feature outputted by the encoder;

[0023] The first image feature is input into a feature mapper to obtain a target image feature output by the feature mapper, wherein the target image feature matches the text feature pre-stored in the training process of the trained teacher image-text comparison model in terms of feature dimension.

[0024] Furthermore, the step of performing knowledge distillation on the student image-text comparison model according to the first output result and the second output result to obtain a trained student image-text comparison model includes:

[0025] Using the first output result as a soft label corresponding to the first image data;

[0026] The soft label is used as a training target for the second output result, so that the student image-text comparison model performs knowledge distillation according to the second output result and the training target to obtain a trained student image-text comparison model.

[0027] Furthermore, the method further comprises:

[0028] Inputting the target image data to be predicted and the trained first image prompt words into the image encoder in the trained student image-text comparison model to obtain the second image features outputted by the encoder;

[0029] The second image feature is multiplied by the text feature pre-stored in the training process of the trained teacher image-text comparison model to obtain an image prediction result corresponding to the target image data.

[0030] The embodiment of this specification also provides a method for predicting an image, the method comprising:

[0031] Input the target image data to be predicted and the trained first image prompt words into the image encoder in the trained student image-text comparison model to obtain the output second image features;

[0032] The second image feature is multiplied by the text feature pre-stored in the training process of the trained teacher image-text contrast model corresponding to the student image-text contrast model to obtain an image prediction result corresponding to the target image data.

[0033] The embodiment of this specification also provides a device for training a student image-text comparison model, including:

[0034] A first output result obtaining module, used for inputting the unlabeled first image data and the corresponding first text data into the trained teacher image-text comparison model to obtain a first output result corresponding to the first image data;

[0035] A target image feature acquisition module, used for inputting the first image data and the corresponding trainable first image prompt word into an image encoder in a student image-text comparison model corresponding to the teacher image-text comparison model to obtain the target image feature outputted by the encoder;

[0036] A second output result obtaining module, used for multiplying the target image feature with the text feature pre-stored in the training process of the trained teacher image-text comparison model to obtain a second output result corresponding to the first image data;

[0037] A model training module is used to enable the student image-text comparison model to perform knowledge distillation according to the first output result and the second output result to obtain a trained student image-text comparison model, wherein the trained student image-text comparison model is used to output a corresponding image prediction result according to the input target image data to be predicted.

[0038] The embodiment of this specification also provides a device for predicting an image, including:

[0039] A second image feature acquisition module is used to input the target image data to be predicted and the trained first image prompt words into the image encoder in the trained student image-text comparison model to obtain the second image feature outputted by the encoder;

[0040] An image prediction module is used to multiply the second image feature with the text feature pre-stored in the training process of the trained teacher image-text comparison model corresponding to the student image-text comparison model to obtain an image prediction result corresponding to the target image data.

[0041] The embodiments of the present specification also provide a storage medium, wherein the storage medium stores a computer program, and the computer program is suitable for being loaded by a processor and executing the steps of the above method.

[0042] An embodiment of the present specification also provides an electronic device, comprising: a processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the steps of the above method.

[0043] In the embodiments of this specification, a distillation scheme for a picture-text comparison model based on a trainable prompt is proposed. The scheme includes a picture-text comparison model with a high parameter count and high precision (called a teacher model) and a picture-text comparison model with a low parameter count to be trained (called a student model). The scheme aims to distill the knowledge of the teacher model into the student model by training the trainable prompt words embedded in the student model. The student model after distillation has higher accuracy than the retrained model and a small number of parameters, and can replace the heavy teacher model for business deployment. In the distillation process, this scheme further proposes a method for reusing teacher text features. In particular, after the pre-training of the teacher model is completed, the text features output by its text encoder are stored, and the image features output by the student model are multiplied by the text features of the teacher model to obtain the final prediction result. This calculation method significantly saves the amount of calculation of the text encoder part of the student model. At the same time, the high-quality features output by the teacher model also ensure the student's prediction performance. Due to the existence of the pre-trained teacher model, this solution can be further extended to unlabeled data, using the teacher model to provide soft labels for unlabeled data to train the student model, which can save a lot of labeling costs. At the same time, extensive unlabeled data training can make the student model have higher performance. The advantage of this solution is that compared with the traditional model full parameter distillation method, the distillation method based on the student model prompt words proposed in this solution requires fewer parameters to be trained and has high training efficiency. This solution reduces the dependence on training data and can achieve good training results with a small amount of data training. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A flowchart of a method for training a student image-text comparison model provided in an embodiment of this specification.

[0045] Figure 2 A flowchart of a method for predicting an image provided in an embodiment of this specification.

[0046] Figure 3 A flowchart of a method for output prediction using a teacher picture-text comparison model is provided as an example in an embodiment of this specification.

[0047] Figure 4 A flowchart of a method for performing knowledge distillation using a student picture-text comparison model is provided as an example in an embodiment of this specification.

[0048] Figure 5 A flowchart of a method for output prediction using a student picture-text comparison model is provided as an example in an embodiment of this specification.

[0049] Figure 6A schematic diagram of the structure of a device for training students' picture-text comparison model provided in an embodiment of this specification.

[0050] Figure 7 A schematic diagram of the structure of a device for predicting an image provided in an embodiment of this specification.

[0051] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of this specification. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solutions and advantages of this specification more clear, the technical solutions of this specification will be clearly and completely described below in combination with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this specification.

[0053] See also Figure 1 , is a flow chart of a method for training a student picture-text comparison model provided in an embodiment of this specification. In an embodiment of this specification, the method for training a student picture-text comparison model is applied to a device for training a student picture-text comparison model (hereinafter referred to as a "student picture-text comparison model training device") or an electronic device equipped with a student picture-text comparison model training device. Figure 1 The process shown in FIG. 1 is described in detail, and the method for training a student picture-text comparison model may specifically include the following steps:

[0054] S102, inputting the unlabeled first image data and the corresponding first text data into the trained teacher image-text comparison model to obtain a first output result corresponding to the first image data.

[0055] In some embodiments, the unlabeled first image data is input into a trained teacher model, which is a picture-text comparison model (picture-text comparison learning base model). The picture-text comparison model is pre-trained based on massive picture-text pairs, and can be applied to downstream tasks when there is no downstream task data or a small amount of downstream task data involved in fine-tuning. Among them, a more representative picture-text pre-training model is the CLIP (Contrastive Language-Image Pre-training) model. The CLIP model obtains image representation and text representation models by contrast learning and pre-training of massive image and text pairs on the Internet, which can effectively extract effective features in images and texts, and its output image / text representation can be directly used for downstream tasks. For example, the teacher picture-text comparison model can be a ViT-L / 14CLIP model. This example embodiment does not specifically limit the specific type of the teacher picture-text comparison model. It should be noted that the above CLIP model is only an example, not a limitation. Those skilled in the art should understand that any type of picture-text comparison model can be included in the protection scope of this specification.

[0056] In some embodiments, the first image data is an unlabeled image. By inputting the unlabeled first image data and the first text data corresponding to the first image data into a trained teacher image-text comparison model, a first output result corresponding to the first image data output by the model can be obtained. The first output result includes multiple prediction results of the teacher image-text comparison model regarding the first image data and the probability corresponding to each prediction result, wherein the first text data is a text description of the entity corresponding to the first image data, such as "a photo of a{classname}". The teacher image-text comparison model is trained based on multiple labeled second image data and their corresponding second text data. The specific training method can be unsupervised training or supervised training. This example embodiment does not specifically limit the specific training method of the teacher image-text comparison model.

[0057] In some embodiments, the teacher image-text comparison model includes an image encoder and a text encoder. The image data input into the teacher image-text comparison model will be input into the image encoder, and the image encoder will output the image features corresponding to the image data. The text data corresponding to the image data input into the teacher image-text comparison model will be input into the text encoder, and the text encoder will output the text features corresponding to the text data. The image features are then multiplied by the text features to obtain the output result of the teacher image-text comparison model regarding the image data.

[0058] In some embodiments, a picture-text comparison model can be pre-trained to obtain a trained teacher picture-text comparison model, which can enable the teacher model to have a better performance and thus better student distillation results. Alternatively, a trained picture-text comparison model can be directly used as the teacher model.

[0059] S104, inputting the first image data and the corresponding trainable first image prompt word into the image encoder in the student image-text contrast model corresponding to the teacher image-text contrast model to obtain the target image features outputted therefrom.

[0060] In some embodiments, prompt learning is a fine-tuning method based on a pre-trained large model. Different from the wire-type layer after the traditional large model representation, prompt learning is to add a pre-set trainable prompt word (Prompt) at the input end or inside the network and input it into the pre-trained large model together with the original features of the input image. The pre-trained model is directly trained by training the learnable prompt word to fit the downstream task data distribution to obtain a high-performance fine-tuning model. Compared with the full-parameter fine-tuning large model, this method is more efficient and requires fewer computing resources. In the fields of computer science and natural language processing, prompt words refer to input information or instructions provided to a computer program or model. In the model, the role of prompt words is to prompt the AI ​​model with contextual information of the input information (i.e., the first image data). During the model training process, prompt words can help the model better understand the intention of the input and make corresponding responses. The prompt word can be a word, a sentence, a picture or a feature. In this example embodiment, the first image prompt word includes one or more image type prompt words. For example, the first image prompt word is a group of image type prompt words. In some embodiments, the first image prompt word input into the student picture-text comparison model in this example embodiment is in a feature-based form, which is trainable (i.e., trainable parameters), that is, the first image prompt word will be trained during the training process of the student picture-text comparison model.

[0061] In some embodiments, the "Teacher-Student Training Paradigm" generally refers to a knowledge distillation technique in deep learning, in which a fully trained and well-performing model (teacher model) guides the learning process of another to-be-trained or simpler model (student model). In some embodiments, the student image-text comparison model (student model) corresponding to the trained teacher image-text comparison model (teacher model) described above only includes an image encoder, but not a text encoder. The image encoder is used to output corresponding image features based on the image data of the input model. For example, the student image-text comparison model can be a ViT-B / 16CLIP model corresponding to the teacher image-text comparison model, i.e., the ViT-L / 14CLIP model. Here, this example embodiment does not specifically limit the specific type of the student image-text comparison model. In some embodiments, the first image data and the trainable first image prompt word corresponding to the first image data are input into the image encoder in the student image-text comparison model corresponding to the trained teacher image-text comparison model described above. For example, the first image data and the trainable first image prompt word corresponding to the first image data are cascaded and then input into the image encoder in the student image-text comparison model to obtain the target image features corresponding to the two (the first image data and the first image prompt word) output by the image encoder.

[0062] S106, multiplying the target image feature with the text feature pre-stored in the training process of the trained teacher image-text comparison model to obtain a second output result corresponding to the first image data.

[0063] In some embodiments, during the training process of the teacher image-text comparison model, the text encoder in the teacher image-text comparison model will output text features corresponding to the text data based on the text data corresponding to the training image data of the input model, and during the training process or after the training is completed, all or part of all text features obtained during the training process will be pre-stored.

[0064] In some embodiments, by multiplying the target image features output by the image encoder of the student image-text comparison model with the text features pre-stored in the teacher image-text comparison model corresponding to the student image-text comparison model during its training process, a second output result corresponding to the first image data (i.e., the output result of the student image-text comparison model for the first image data) is obtained, wherein the second output result includes multiple prediction results of the student image-text comparison model regarding the first image data and the probability corresponding to each prediction result.

[0065] S108, enabling the student image-text comparison model to perform knowledge distillation according to the first output result and the second output result to obtain a trained student image-text comparison model, wherein the trained student image-text comparison model is used to output a corresponding image prediction result according to the input target image data to be predicted.

[0066] In some embodiments, knowledge distillation is a compression method for a neural network model, which includes two models, namely a large teacher model with a large number of parameters and a lightweight small student model. In the process of training the student model, the output of the teacher model is used to provide additional supervision information, and the knowledge contained in the teacher model is transferred to the student model. The student model after distillation has a higher accuracy rate than the original trained model. In some embodiments, the student image-text comparison model provides additional supervision information to the second output result of the student image-text comparison model for the first image data through the first output result of the corresponding trained teacher image-text comparison model for the first image data, so that the second output result output by the student image-text comparison model is aligned with the first output result output by the teacher image-text comparison model, and the knowledge contained in the teacher image-text comparison model is transferred to the student image-text comparison model, thereby completing knowledge distillation and obtaining a trained student image-text comparison model. The student image-text comparison model after distillation has a higher accuracy rate than the previous model while maintaining a small weight and a small amount of calculation required, and can meet the accuracy requirements required by the business. In some embodiments, during the training process of the student image-text contrast model, the first image prompt word is trainable, and the parameters of the image encoder in the student image-text contrast model are not trainable (i.e., frozen parameters). Therefore, training the student image-text contrast model is essentially training the first image prompt word. In some embodiments, the input of the student image-text contrast model is the target image data to be predicted, and the output is the image prediction result corresponding to the target image data. The image prediction result includes multiple prediction outputs of the student image-text contrast model about the target image data and the probability corresponding to each prediction output, that is, the trained student image-text contrast model has the function of image prediction.

[0067] In the embodiments of this specification, a distillation scheme for a picture-text comparison model based on a trainable prompt is proposed. The scheme includes a picture-text comparison model with a high parameter count and high precision (called a teacher model) and a picture-text comparison model with a low parameter count to be trained (called a student model). The scheme aims to distill the knowledge of the teacher model into the student model by training the trainable prompt words embedded in the student model. The student model after distillation has higher accuracy than the retrained model and a small number of parameters, and can replace the heavy teacher model for business deployment. In the distillation process, this scheme further proposes a method for reusing teacher text features. In particular, after the pre-training of the teacher model is completed, the text features output by its text encoder are stored, and the image features output by the student model are multiplied by the text features of the teacher model to obtain the final prediction result. This calculation method significantly saves the amount of calculation of the text encoder part of the student model. At the same time, the high-quality features output by the teacher model also ensure the student's prediction performance. Due to the existence of the pre-trained teacher model, this solution can be further extended to unlabeled data, using the teacher model to provide soft labels for unlabeled data to train the student model, which can save a lot of labeling costs. At the same time, extensive unlabeled data training can make the student model have higher performance. The advantage of this solution is that compared with the traditional model full parameter distillation method, the distillation method based on the student model prompt words proposed in this solution requires fewer parameters to be trained and has high training efficiency. This solution reduces the dependence on training data and can achieve good training results with a small amount of data training.

[0068] In some embodiments, the method further includes: inputting the labeled second image data and the corresponding second text data into the teacher image-text comparison model to obtain the third output result corresponding to the second image data; training the teacher image-text comparison model based on the third output result and the true value label corresponding to the second image data to obtain the trained teacher image-text comparison model. In some embodiments, a plurality of labeled second image data and the second text data corresponding to each second image data are input into the teacher image-text comparison model to obtain the third output result corresponding to the second image data output by the model, wherein the third output result includes a plurality of prediction results of the teacher image-text comparison model on the second image data and the probability corresponding to each prediction result, the second text data is the text description of the entity corresponding to the second image data, for example, "a photo of a{classname}", and the teacher image-text comparison model is trained based on the third output result corresponding to each second image data and the label corresponding to the second image data to obtain the trained teacher image-text comparison model. In some embodiments, the third output result corresponding to the second image data is supervised by the true value label of each second image data, and the teacher image-text comparison model is trained to obtain the trained teacher image-text comparison model.

[0069] In some embodiments, the step of inputting the labeled second image data and the corresponding second text data into the teacher's picture-text comparison model to obtain the third output result corresponding to the second image data includes: inputting the labeled second image data into the image encoder in the teacher's picture-text comparison model to obtain the corresponding second image feature; inputting the second text data corresponding to the second image data into the text encoder in the teacher's picture-text comparison model to obtain the corresponding second text feature, and multiplying the second image feature by the second text feature to obtain the third output result corresponding to the second image data. In some embodiments, the teacher's picture-text comparison model includes an image encoder and a text encoder, inputting the second image data into the image encoder in the teacher's picture-text comparison model to obtain the second image feature output by the image encoder, inputting the second text data corresponding to the second image data into the text encoder in the teacher's picture-text comparison model to obtain the second text feature output by the text encoder, and multiplying the second image feature by the second text feature to obtain the third output result corresponding to the second image data.

[0070] In some embodiments, the method further comprises: after the training of the teacher image-text comparison model is completed, pre-storing well-trained second text features. In some embodiments, during the training of the teacher image-text comparison model, the text encoder in the teacher image-text comparison model will output the second text features corresponding to the second text data according to the second text data corresponding to the training image data (i.e., the second image data) of the input model. After the training is completed, not all the second text features obtained during the training process are pre-stored, but only some of the well-trained second text features are pre-stored.

[0071] In some embodiments, the inputting of the first image data and the corresponding trainable first image prompt words into the image encoder in the student image-text comparison model corresponding to the teacher image-text comparison model to obtain the target image features outputted by the image encoder includes: obtaining a plurality of first vector representations corresponding to the first image data and a plurality of second vector representations corresponding to the trainable first image prompt words corresponding to the first image data; cascading the plurality of first vector representations and the plurality of second vector representations and inputting them into the image encoder in the student image-text comparison model corresponding to the teacher image-text comparison model to obtain the target image features outputted by the image encoder. In some embodiments, the first image data is cut into image blocks (patches), each patch is digitized (mapped into vector representation), and a plurality of first vector representations corresponding to the first image data are obtained, for example, by using ViT (Vision Transformer, a deep learning model based on Transformer architecture) to obtain a plurality of first vector representations corresponding to the first image data. In some embodiments, the trainable first image prompt words corresponding to the first image data are cut into minimum semantic units (tokens), each token is digitized (mapped into vector representation), and a plurality of second vector representations corresponding to the first image prompt words are obtained. In some embodiments, the multiple first vector representations and the multiple second vector representations are cascaded and then input into an image encoder in a student image-text comparison model to obtain target image features corresponding to the two (first image data and first image prompt words) output by the image encoder.

[0072] In some embodiments, the cascading of the multiple first vector representations and the multiple second vector representations into an image encoder in a student image-text comparison model corresponding to the teacher image-text comparison model to obtain a target image feature outputted by the image encoder includes: cascading the multiple first vector representations and the multiple second vector representations into an image encoder in a student image-text comparison model corresponding to the teacher image-text comparison model to obtain a first image feature outputted by the image encoder; inputting the first image feature into a feature mapper to obtain a target image feature outputted by the feature mapper, wherein the target image feature matches the text feature pre-stored by the trained teacher image-text comparison model during its training process in terms of feature dimension. In some embodiments, the multiple first vector representations and the multiple second vector representations are cascaded and then input into the image encoder in the student image-text comparison model to obtain the first image feature corresponding to the two (first image data and first image prompt word) output by the image encoder. If the first image feature does not match the text feature pre-stored by the teacher image-text comparison model during its training process in terms of feature dimension, the first image feature can be input into a feature mapper (Projector) to obtain the target image feature output by the feature mapper, so that the target image feature does not match the text feature pre-stored by the teacher image-text comparison model during its training process in terms of feature dimension. In some embodiments, the feature mapper can be a two-layer MLP (Multilayer Perceptron). It should be noted that the above-mentioned MLP is only for example and not for limitation. Those skilled in the art should understand that any type of feature mapper can be included in the protection scope of this specification. In some embodiments, during the training process of the student image-text contrast model, the first image prompt word is trainable, the parameters of the image encoder in the student image-text contrast model are non-trainable (i.e., frozen parameters), and the parameters of the feature mapper are trainable parameters. Therefore, training the student image-text contrast model is essentially training the parameters of the first image prompt word and the feature mapper.

[0073] In some embodiments, the step of making the student image-text comparison model perform knowledge distillation according to the first output result and the second output result to obtain a trained student image-text comparison model includes: taking the first output result as a soft label corresponding to the first image data; taking the soft label as a training target for the second output result, so that the student image-text comparison model performs knowledge distillation according to the second output result and the training target to obtain a trained student image-text comparison model. In some embodiments, the first output result of the teacher image-text comparison model with respect to the first image data inputted thereto is taken as a soft label for the first image data, and the soft label is taken as a training target for the second output result of the student image-text comparison model with respect to the first image data inputted thereto, so that the student image-text comparison model performs knowledge distillation based on the second output result and its corresponding training target, and transfers the knowledge contained in the teacher image-text comparison model to the student image-text comparison model to obtain a trained student image-text comparison model, that is, the distillation process in the embodiments of this specification does not require labeled images, and it uses the pre-trained teacher model to generate soft labels for unlabeled images, so that the student model can be trained without the need for labeled images. In the embodiments of this specification, due to the existence of the teacher model, the problem of data label limitation in the existing scenario is solved, and a large amount of unlabeled data can be used to train the student model, no longer sticking to the original limited labeled data, which can save a lot of labeling costs. During training, we use a large amount of unlabeled data as the input of the teacher model to obtain their corresponding soft labels. During distillation, these soft labels are used as the target of training the student model. In this way, the trainable first image prompt word can learn a wider range of knowledge. At the same time, the high-performance teacher model also ensures the accuracy of the soft labels used for distillation.

[0074] In some embodiments, the method further includes: inputting the target image data to be predicted and the trained first image prompt word into the image encoder in the trained student image-text contrast model to obtain the second image feature outputted by the image encoder; multiplying the second image feature with the text feature pre-stored by the trained teacher image-text contrast model during its training process to obtain the image prediction result corresponding to the target image data. In some embodiments, the target image data to be predicted and the trained first image prompt word (the first image prompt word inputted into the student image-text contrast model will be trained during the training process of the student image-text contrast model) are inputted into the image encoder in the trained student image-text contrast model, for example, the target image data to be predicted and the trained first image prompt word are cascaded and then inputted into the image encoder in the student image-text contrast model to obtain the second image feature corresponding to the two (target image data and first image prompt word) outputted by the image encoder. In some embodiments, by multiplying the second image feature output by the image encoder of the student image-text comparison model with the text feature pre-stored in the teacher image-text comparison model corresponding to the student image-text comparison model during its training process, an image prediction result corresponding to the target image data (i.e., the output result of the student image-text comparison model for the target image data) is obtained. The image prediction result includes multiple prediction outputs of the student image-text comparison model regarding the target image data and the probability corresponding to each prediction output.

[0075] Figure 2 A flowchart of a method for predicting an image provided in an embodiment of the present specification is provided. In an embodiment of the present specification, the method for predicting an image is applied to a device for predicting an image (hereinafter referred to as an "image prediction device") or an electronic device equipped with an image prediction device. Figure 1 The process shown in FIG. 1 is described in detail, and the method for predicting an image may specifically include the following steps:

[0076] S202, input the target image data to be predicted and the trained first image prompt word into the image encoder in the trained student image-text comparison model to obtain the second image feature outputted by it. The relevant operations have been described in detail above and will not be repeated here. In some embodiments, the training method of the student image-text comparison model has been described in detail above and will not be repeated here.

[0077] S204, multiplying the second image feature with the text feature pre-stored in the training process of the trained teacher image-text comparison model corresponding to the student image-text comparison model to obtain the image prediction result corresponding to the target image data. The relevant operations have been described in detail above and will not be repeated here.

[0078] Figure 3A flowchart of a method for output prediction using a teacher picture-text comparison model is provided as an example in an embodiment of this specification.

[0079] like Figure 3 As shown, the input image is input into the image encoder in the teacher image-text comparison model to obtain the image features output by the image encoder, the input text corresponding to the input image (for example, "aphoto of a{classname}") is input into the text encoder in the teacher image-text comparison model to obtain the text features output by the text encoder, the image features are multiplied by the text features to obtain the output prediction of the teacher image-text comparison model for the input image.

[0080] Figure 4 A flowchart of a method for performing knowledge distillation using a student picture-text comparison model is provided as an example in an embodiment of this specification.

[0081] like Figure 4 As shown, the input image and the corresponding input text are respectively input to the image encoder and text encoder of the CLIP teacher model (teacher image-text comparison model), the image features output by the image encoder are multiplied by the text features output by the text encoder, and the output probability of the CLIP teacher model for the input image is obtained. At the same time, the text features output by the text encoder are pre-stored, and the input image and the corresponding prompt word (that is, the prompt word (prompt) is divided into the smallest semantic units (token)) are input to the image encoder of the CLIP learning model (student image-text comparison model) corresponding to the CLIP teacher model, and the image is encoded. The first image feature output by the encoder is input into the feature mapper of the CLIP learning model to obtain the target image feature output by the feature mapper, the target image feature is multiplied by the text feature pre-stored in the CLIP teacher model to obtain the output probability of the CLIP student model for the input image, and the output probability of the CLIP student model is subjected to knowledge distillation based on the output probability of the CLIP teacher model, wherein the parameters of the image encoder in the CLIP teacher model, the parameters of the text encoder and the parameters of the image encoder in the CLIP student model are frozen parameters (i.e., trainable), and the parameters of the feature mapper in the prompt word and the CLIP student model are trainable parameters.

[0082] Figure 5 A flowchart of a method for output prediction using a student picture-text comparison model is provided as an example in an embodiment of this specification.

[0083] like Figure 5As shown, the test image to be predicted and its corresponding prompt word are cascaded and input into the image encoder in the trained student image-text contrast model to obtain the first image feature output by the image encoder, the first image feature is input into the feature mapper in the student image-text contrast model to obtain the second feature output by the feature mapper, the second feature is multiplied with the well-trained text feature pre-stored in the teacher image-text contrast model corresponding to the student image-text contrast model during its training process to obtain the output prediction of the student image-text contrast model for the test image.

[0084] Figure 6 A schematic diagram of the structure of a device for training a student image-text comparison model provided in an embodiment of the present specification, the device for training a student image-text comparison model (hereinafter referred to as "student image-text comparison model training device 1") can be implemented as all or part of an electronic device through software, hardware, or a combination of both. According to some embodiments, the student image-text comparison model training device 1 includes a first output result acquisition module 11, a target image feature acquisition module 12, a second output result acquisition module 13, and a model training module 14.

[0085] A first output result obtaining module, used for inputting the unlabeled first image data and the corresponding first text data into the trained teacher image-text comparison model to obtain a first output result corresponding to the first image data;

[0086] A target image feature acquisition module, used for inputting the first image data and the corresponding trainable first image prompt word into an image encoder in a student image-text comparison model corresponding to the teacher image-text comparison model to obtain the target image feature outputted by the encoder;

[0087] A second output result obtaining module, used for multiplying the target image feature with the text feature pre-stored in the training process of the trained teacher image-text comparison model to obtain a second output result corresponding to the first image data;

[0088] A model training module is used to enable the student image-text comparison model to perform knowledge distillation according to the first output result and the second output result to obtain a trained student image-text comparison model, wherein the trained student image-text comparison model is used to output a corresponding image prediction result according to the input target image data to be predicted.

[0089] In some embodiments, the student picture-text comparison model training device 1 is also used for:

[0090] Inputting the second image data and the corresponding second text data into the teacher image-text comparison model to obtain a third output result corresponding to the second image data;

[0091] The teacher image-text comparison model is trained based on the third output result and the true value label corresponding to the second image data to obtain a trained teacher image-text comparison model.

[0092] In some embodiments, inputting the second image data and the corresponding second text data into the teacher image-text comparison model to obtain a third output result corresponding to the second image data includes:

[0093] Inputting the second image data into the image encoder in the teacher image-text comparison model to obtain the corresponding second image features;

[0094] The second text data corresponding to the second image data is input into the text encoder in the teacher image-text comparison model to obtain the corresponding second text feature, and the second image feature is multiplied by the second text feature to obtain a third output result corresponding to the second image data.

[0095] In some embodiments, the student picture-text comparison model training device 1 is also used for:

[0096] After the teacher image-text comparison model is trained, well-trained second text features are pre-stored.

[0097] In some embodiments, the step of inputting the first image data and the corresponding trainable first image prompt word into an image encoder in a student image-text comparison model corresponding to the teacher image-text comparison model to obtain the target image features outputted by the encoder includes:

[0098] Obtaining a plurality of first vector representations corresponding to the first image data and a plurality of second vector representations corresponding to the trainable first image prompt words corresponding to the first image data;

[0099] The multiple first vector representations and the multiple second vector representations are cascaded and input into the image encoder in the student image-text contrast model corresponding to the teacher image-text contrast model to obtain the target image features of its output.

[0100] In some embodiments, the step of concatenating the plurality of first vector representations and the plurality of second vector representations and inputting them into an image encoder in a student image-text contrast model corresponding to the teacher image-text contrast model to obtain the target image features outputted by the encoder comprises:

[0101] Cascading the multiple first vector representations and the multiple second vector representations, and inputting them into an image encoder in a student image-text contrast model corresponding to the teacher image-text contrast model to obtain a first image feature outputted by the encoder;

[0102] The first image feature is input into a feature mapper to obtain a target image feature output by the feature mapper, wherein the target image feature matches the text feature pre-stored in the training process of the trained teacher image-text comparison model in terms of feature dimension.

[0103] In some embodiments, the step of performing knowledge distillation on the student image-text comparison model according to the first output result and the second output result to obtain a trained student image-text comparison model includes:

[0104] Using the first output result as a soft label corresponding to the first image data;

[0105] The soft label is used as a training target for the second output result, so that the student image-text comparison model performs knowledge distillation according to the second output result and the training target to obtain a trained student image-text comparison model.

[0106] In some embodiments, the student picture-text comparison model training device 1 is also used for:

[0107] Inputting the target image data to be predicted and the trained first image prompt words into the image encoder in the trained student image-text comparison model to obtain the second image features outputted by the encoder;

[0108] The second image feature is multiplied by the text feature pre-stored in the training process of the trained teacher image-text comparison model to obtain an image prediction result corresponding to the target image data.

[0109] Figure 7 A schematic diagram of a device for predicting an image provided in an embodiment of the present specification, the device for predicting an image (hereinafter referred to as "image prediction device 2") can be implemented as all or part of an electronic device through software, hardware, or a combination of both. According to some embodiments, the image prediction device 2 includes a second image feature acquisition module 21 and an image prediction module 22.

[0110] The second image feature acquisition module is used to input the target image data to be predicted and the trained first image prompt words into the image encoder in the trained student image-text comparison model to obtain the second image feature outputted by it.

[0111] An image prediction module is used to multiply the second image feature with the text feature pre-stored in the training process of the trained teacher image-text comparison model corresponding to the student image-text comparison model to obtain an image prediction result corresponding to the target image data.

[0112] The above device embodiments correspond to the method embodiments. For specific descriptions, please refer to the description of the method embodiments, which will not be repeated here. The device embodiments are obtained based on the corresponding method embodiments and have the same technical effects as the corresponding method embodiments. For specific descriptions, please refer to the corresponding method embodiments.

[0113] The embodiment of the present specification also provides a computer storage medium, which can store multiple instructions, and the instructions are suitable for being loaded by a processor to execute the method of the embodiment of the present specification.

[0114] The embodiments of the present specification also provide a computer program product, which stores at least one instruction, and the at least one instruction is loaded by the processor to execute the method of the embodiments of the present specification.

[0115] The embodiments of this specification also provide Figure 8 The structural diagram of the electronic device shown in FIG. Figure 8 At the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include hardware required for other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the method of the embodiment of this specification.

[0116] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0117] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0118] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0119] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0120] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0121] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0122] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0123] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0124] The above description is only an embodiment of the present specification and is not intended to limit the present specification. For those skilled in the art, the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification shall be included in the scope of the claims of the present specification.

Claims

1. A method for training a student image-text comparison model, comprising: Inputting the unlabeled first image data and the corresponding first text data into the trained teacher image-text comparison model to obtain a first output result corresponding to the first image data; Inputting the first image data and the corresponding trainable first image prompt word into an image encoder in a student image-text contrast model corresponding to the teacher image-text contrast model to obtain the target image features outputted by the encoder; Multiplying the target image feature with the text feature pre-stored in the training process of the trained teacher image-text comparison model to obtain a second output result corresponding to the first image data; The student image-text comparison model performs knowledge distillation according to the first output result and the second output result to obtain a trained student image-text comparison model, wherein the trained student image-text comparison model is used to output a corresponding image prediction result according to the input target image data to be predicted.

2. The method according to claim 1, further comprising: Inputting the labeled second image data and the corresponding second text data into the teacher image-text comparison model to obtain a third output result corresponding to the second image data; The teacher image-text comparison model is trained based on the third output result and the true value label corresponding to the second image data to obtain a trained teacher image-text comparison model.

3. The method according to claim 2, wherein the step of inputting the labeled second image data and the corresponding second text data into the teacher image-text comparison model to obtain a third output result corresponding to the second image data comprises: Input the labeled second image data into the image encoder in the teacher image-text comparison model to obtain the corresponding second image features; The second text data corresponding to the second image data is input into the text encoder in the teacher image-text comparison model to obtain the corresponding second text feature, and the second image feature is multiplied by the second text feature to obtain a third output result corresponding to the second image data.

4. The method according to claim 3, further comprising: After the teacher image-text comparison model is trained, well-trained second text features are pre-stored.

5. The method according to claim 1, wherein the step of inputting the first image data and the corresponding trainable first image prompt word into an image encoder in a student image-text contrast model corresponding to the teacher image-text contrast model to obtain the target image features outputted by the encoder comprises: Obtaining a plurality of first vector representations corresponding to the first image data and a plurality of second vector representations corresponding to the trainable first image prompt words corresponding to the first image data; The multiple first vector representations and the multiple second vector representations are cascaded and input into the image encoder in the student image-text contrast model corresponding to the teacher image-text contrast model to obtain the target image features of its output.

6. The method according to claim 5, wherein the cascading of the plurality of first vector representations and the plurality of second vector representations is input into an image encoder in a student image-text contrast model corresponding to the teacher image-text contrast model to obtain the target image features outputted by the encoder, comprises: Cascading the multiple first vector representations and the multiple second vector representations, and inputting them into an image encoder in a student image-text contrast model corresponding to the teacher image-text contrast model to obtain a first image feature outputted by the encoder; The first image feature is input into a feature mapper to obtain a target image feature output by the feature mapper, wherein the target image feature matches the text feature pre-stored in the training process of the trained teacher image-text comparison model in terms of feature dimension.

7. According to the method of claim 1, the step of performing knowledge distillation on the student image-text comparison model according to the first output result and the second output result to obtain a trained student image-text comparison model comprises: Using the first output result as a soft label corresponding to the first image data; The soft label is used as a training target for the second output result, so that the student image-text comparison model performs knowledge distillation according to the second output result and the training target to obtain a trained student image-text comparison model.

8. The method according to any one of claims 1 to 7, further comprising: Inputting the target image data to be predicted and the trained first image prompt words into the image encoder in the trained student image-text comparison model to obtain the second image features outputted by the encoder; The second image feature is multiplied by the text feature pre-stored in the training process of the trained teacher image-text comparison model to obtain an image prediction result corresponding to the target image data.

9. A method for predicting an image, comprising: Input the target image data to be predicted and the trained first image prompt words into the image encoder in the trained student image-text comparison model to obtain the output second image features; The second image feature is multiplied by the text feature pre-stored in the training process of the trained teacher image-text contrast model corresponding to the student image-text contrast model to obtain the image prediction result corresponding to the target image data, wherein the trained student image-text contrast model is trained by the method described in any one of claims 1 to 7.

10. A device for training a student image-text comparison model, comprising: A first output result obtaining module, used for inputting the unlabeled first image data and the corresponding first text data into the trained teacher image-text comparison model to obtain a first output result corresponding to the first image data; A target image feature acquisition module, used for inputting the first image data and the corresponding trainable first image prompt word into an image encoder in a student image-text comparison model corresponding to the teacher image-text comparison model to obtain the target image feature outputted by the encoder; A second output result obtaining module, used for multiplying the target image feature with the text feature pre-stored in the training process of the trained teacher image-text comparison model to obtain a second output result corresponding to the first image data; A model training module is used to enable the student image-text comparison model to perform knowledge distillation according to the first output result and the second output result to obtain a trained student image-text comparison model, wherein the trained student image-text comparison model is used to output a corresponding image prediction result according to the input target image data to be predicted.

11. A device for predicting an image, comprising: A second image feature acquisition module is used to input the target image data to be predicted and the trained first image prompt words into the image encoder in the trained student image-text comparison model to obtain the second image feature outputted by the encoder; An image prediction module is used to multiply the second image feature with the text feature pre-stored in the training process of the trained teacher image-text contrast model corresponding to the student image-text contrast model to obtain the image prediction result corresponding to the target image data, wherein the trained student image-text contrast model is trained by the method described in any one of claims 1 to 7.

12. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

13. An electronic device, characterized in that: include: A processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the steps of the method as claimed in any one of claims 1 to 9.

14. A computer program product having at least one instruction stored thereon, characterized in that: When the at least one instruction is executed by the processor, the steps of the method described in any one of claims 1 to 9 are implemented.

Images