Model training method and device, image recognition method and device and electronic equipment

Abstract

The embodiment of the invention provides a model training method and device, an image recognition method and device and electronic equipment, and relates to the technical field of machine learning. The model training method comprises the following steps: acquiring a sample image; inputting the sample image into a to-be-trained image recognition model to obtain an output result corresponding to each sample cue word; determining a loss value corresponding to the image recognition model based on the output result corresponding to each sample prompt word and the label of the sample image; and when it is judged that the image recognition model does not meet a model convergence condition based on the loss value, performing parameter adjustment on the current weight parameter of the full connection layer and the image encoder based on the loss value, and returning to the step of acquiring the sample image. Through the scheme, the training efficiency of the image recognition model can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of machine learning, and in particular to a model training method, an image recognition method, a device and an electronic device. Background Art

[0002] In order to maintain the network environment of the network platform, it is necessary to review the images or videos uploaded to the platform to avoid the dissemination of images or videos that may cause bad guidance through the platform, for example, the dissemination of images including tattoos, smoke and other objects.

[0003] Typically, an image is identified using an image recognition model for image recognition to obtain a recognition result, which characterizes the presentation state of an object in an image, such as whether an object exists in the image. In the prior art, a pre-trained contrastive learning model is used as an image recognition model, for example, a pre-trained CLIP (Contrastive Language-Image Pre-Training) model. Among them, the above-mentioned CLIP model includes a text encoder for processing prompt words and an image encoder for processing images. During the recognition process, the CLIP model calls the text encoder to process the preset prompt words respectively, and at the same time, calls the image encoder to process the image, thereby realizing image recognition based on the processing results of the two encoders. Therefore, in the process of training to obtain the above-mentioned image recognition model for image recognition, it is necessary to train the two encoders at the same time, during which the number of training samples and the training time required are greatly improved, resulting in a reduction in the training efficiency of the image recognition model.

[0004] Based on this, how to improve the training efficiency of image recognition models used to recognize images has become a technical problem that urgently needs to be solved. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a model training method, an image recognition method, a device and an electronic device, so as to improve the training efficiency of the model used for image recognition by optimizing the image recognition model. The specific technical solution is as follows:

[0006] In a first aspect provided by an embodiment of the present application, a model training method is first provided, the method comprising:

[0007] Acquire a sample image; wherein the sample image is provided with a label for characterizing a presentation state of the sample object;

[0008] The sample image is input into the image recognition model to be trained to obtain an output result corresponding to each sample prompt word; wherein the image recognition model includes a fully connected layer with a predetermined initial weight parameter and an image encoder in a pre-trained contrastive learning model; each sample prompt word is an image description text with the presentation state of the sample object as the description content, and the initial weight parameter is predetermined based on the vector features of each sample prompt word; the output result corresponding to each sample prompt word is used to characterize the similarity between the vector features of the sample prompt word and the vector features of the sample image;

[0009] Determine a loss value corresponding to the image recognition model based on the output result corresponding to each sample prompt word and the label of the sample image;

[0010] When it is determined based on the loss value that the image recognition model does not meet the model convergence condition, the current weight parameters of the fully connected layer and the image encoder are adjusted based on the loss value, and the process returns to the step of obtaining the sample image.

[0011] In a second aspect provided by an embodiment of the present application, an image recognition method is also provided, the method comprising:

[0012] Acquire a target image; wherein the target image is an image used to identify a target object;

[0013] The target image is input into a pre-trained image recognition model to obtain an output result corresponding to each predetermined prompt word; wherein the image recognition model is trained based on the model training method described in the first aspect; the predetermined prompt words are sample prompt words used to train the image recognition model; the output result corresponding to each predetermined prompt word is used to characterize the similarity between the vector feature of the predetermined prompt word and the vector feature of the target image;

[0014] Based on the output result corresponding to each predetermined prompt word, a presentation state of the target image for the target object is determined.

[0015] In a third aspect provided by an embodiment of the present application, a model training device is provided, the device comprising:

[0016] An image acquisition module, used to acquire a sample image; wherein the sample image is provided with a label for characterizing a presentation state of the sample object;

[0017] A model training module, used for inputting the sample image into the image recognition model to be trained, and obtaining an output result corresponding to each sample prompt word; wherein the image recognition model includes a fully connected layer with a predetermined initial weight parameter and an image encoder in a pre-trained contrastive learning model; each sample prompt word is an image description text with the presentation state of the sample object as the description content, and the similarity of each sample prompt word to the sample image is used to characterize the similarity between the vector feature of the sample prompt word and the vector feature of the sample image, and the initial weight parameter is predetermined based on the vector feature of each sample prompt word; the output result corresponding to each sample prompt word is used to characterize the similarity between the vector feature of the sample prompt word and the vector feature of the sample image;

[0018] A loss value determination module, used to determine the loss value corresponding to the image recognition model based on the output result corresponding to each sample prompt word and the label of the sample image;

[0019] A parameter adjustment module is used to adjust the weight parameters of the fully connected layer and the image encoder based on the loss value when it is judged that the image recognition model does not meet the model convergence condition based on the loss value, and return to the step of obtaining the sample image.

[0020] In a fourth aspect provided by an embodiment of the present application, an image recognition device is provided, the device comprising:

[0021] An image acquisition module, used to acquire a target image; wherein the target image is an image used to identify a target object;

[0022] A similarity determination module is used to input the target image into a pre-trained image recognition model to obtain an output result corresponding to each predetermined prompt word; wherein the image recognition model is trained based on a model training method provided in an embodiment of the present application; the predetermined prompt word is a sample prompt word used to train the image recognition model; the output result corresponding to each predetermined prompt word is used to characterize the similarity between the vector feature of the predetermined prompt word and the vector feature of the target image;

[0023] The presentation state determination module is used to determine the presentation state of the target image for the target object based on the output result corresponding to each predetermined prompt word.

[0024] In the fifth aspect provided by the embodiment of the present application, there is also provided an electronic device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; the memory is used to store computer programs; the processor is used to implement any model training method provided in the first aspect and / or any image recognition method provided in the second aspect when executing the program stored in the memory.

[0025] In another aspect provided by the embodiments of the present application, a computer-readable storage medium is also provided, in which a computer program is stored. When the computer program is executed by a processor, it implements any model training method provided in the first aspect above, and / or any image recognition method provided in the second aspect above.

[0026] In another aspect provided by the embodiments of the present application, a computer program product comprising instructions is also provided, which, when executed on a computer, enables the computer to execute any model training method provided in the first aspect above, and / or any image recognition method provided in the second aspect above.

[0027] It can be seen from the above that, by applying the solution provided in the embodiment of the present application, before model training, an image recognition model to be trained can be pre-constructed, and the image recognition model includes a fully connected layer with predetermined initial weight parameters and an image encoder in a pre-trained contrastive learning model. The image recognition model can be used to determine the similarity between the vector features of the prompt word and the vector features of the input image. Therefore, before model training, each sample prompt word can be obtained, wherein each sample prompt word is an image description text with the presentation state of the sample object as the description content. Then, based on the vector features of each sample prompt word, the initial weight parameters of the fully connected layer are predetermined to provide the image recognition process with vector features about the prompt word through the fully connected layer with the initial weight parameters.

[0028] In this way, when training the model, it is only necessary to input the acquired sample image into the image recognition model to be trained to obtain the output result corresponding to each sample prompt word, and the output result is used to characterize the similarity between the vector feature of the sample prompt word and the vector feature of the sample image. Then, based on the output result corresponding to each sample prompt word and the pre-set label of the sample image itself, the loss value of the current weight parameter of the fully connected layer of the image recognition model and the loss value of the image encoder when acquiring the vector feature of the sample image are determined, and then the current weight parameter of the fully connected layer of the image recognition model and the image encoder are adjusted based on the determined loss value. In this way, by reconstructing the model structure of the image recognition model, the training effect of training two encoders can be achieved by training the number of training samples and training time required for one model, that is, by reducing the number of encoders required to be trained during model training, the number of training samples and training time required during training are reduced, thereby improving the training efficiency of the image recognition model.

[0029] In addition, in the image recognition method provided in the embodiment of the present application, since the image recognition model used is trained based on a model training method provided in the embodiment of the present application, that is to say, the vector features of the predetermined prompt words used for image recognition in the image recognition model used are fixed, when using the image recognition model for image recognition, there is no need to input prompt words, only the target image needs to be input to achieve automatic recognition of the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0031] Figure 1 A schematic diagram of the classification process of the existing CLIP model;

[0032] Figure 2 A flowchart of a model training method provided in an embodiment of the present application;

[0033] Figure 3 A flowchart of an existing training method for a contrastive learning model is provided;

[0034] Figure 4 This is a schematic diagram of the structure of a conventional deep learning recognition model;

[0035] Figure 5 A flowchart of an image recognition method provided in an embodiment of the present application;

[0036] Figure 6 A schematic diagram of the structure of a model training device provided in an embodiment of the present application;

[0037] Figure 7 A schematic diagram of the structure of an image recognition device provided in an embodiment of the present application;

[0038] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0039] The technical solutions in the embodiments of the present invention will be described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0040] In order to better understand the present application, the professional terms involved in the embodiments of the present application are first introduced below.

[0041] The text-image pre-training model based on contrastive learning (also referred to as the contrastive learning model) is a model pre-trained on 400 million pairs of image-text matching pairs, which has a strong generalization ability. By giving the text prompt words corresponding to the contrastive learning model, the model can obtain the similarity between the features of the text prompt words and the features of the corresponding image as the output result of the model. Specifically, the contrastive learning model identifies the uploaded image, that is, identifies the presentation state of the image for the object, such as whether there is an object in the image (whether there is a cat in the image), or whether there is an object in a certain state in the image (whether there is a white cat in the image), etc. In the recognition process, the contrastive learning model can classify and identify the image based on the pre-set prompt words. The above-mentioned text-image pre-training model based on contrastive learning can be a CLIP model, a BLIP (Bootstrapping Language-Image Pre-training, bootstrap language image pre-training) model or a SigLip (Sigmoid Loss for Language-Image Pre-training, language-image pre-training) model, etc., which is not specifically limited in the embodiments of the present application.

[0042] Taking the above contrastive learning model as the CLIP model as an example, when the CLIP model is used to classify an image, the image can be classified by giving the CLIP model two prompt words, "people with tattoos" and "people without tattoos", and letting the CLIP model determine which prompt word has a closer similarity.

[0043] Combine the following Figure 1 This paper introduces the classification process of the CLIP model. Figure 1As shown in FIG. 1 , plane, car, dog, and bird are all target objects (or sample objects) in the embodiment of the present application, and “A photo of a {object}” is the prompt word in the embodiment of the present application. The byte position of object is the byte position occupied by the target object (or sample object) in the prompt word.

[0044] After receiving the prompt word, the Text Encoder of the CLIP model can convert the text multimodal features corresponding to the prompt word, including T1, T2, T3...T N That is to say, after receiving the prompt word, the text encoder of the CLIP model extracts the text features of the prompt word and calls its own text multimodal network to convert the extracted text features into text multimodal features.

[0045] After receiving the image, the Image Encoder of the CLIP model can convert and obtain the image multimodal features I1 corresponding to the above image. In other words, after receiving the image, the image encoder of the CLIP model extracts the image features of the image, and calls its own image multimodal network to convert the extracted image features into image multimodal features.

[0046] The CLIP model can then associate the obtained image multimodal feature I1 with the text multimodal features T1, T2, T3, ... T corresponding to the prompt word. N Perform feature similarity matching to obtain the similarities I1·T1, I1·T2, I1·T3, I1·T N Among them, I1·T3 has the highest similarity, and the object in the prompt word corresponding to I1·T3 is dog. Therefore, it can be determined that the object in the image input to the Image Encoder is dog, and the output result of the model is "A photo of a dog".

[0047] Among them, the above-mentioned image multimodal features are the vector features of the image in the embodiment of the present application; the above-mentioned text multimodal features (text multimodal features) are the vector features of the prompt words in the embodiment of the present application.

[0048] In order to maintain the network environment of the network platform, images or videos uploaded to the platform need to be reviewed to avoid spreading images or videos that may cause bad guidance through the platform, such as spreading images of objects such as tattoos and smoke. Therefore, each image or each frame of video uploaded to the platform can be manually reviewed. However, for online review services, the cost of manual review is huge.

[0049] In the related art, a pre-trained image recognition model for image recognition is used to identify images or videos that need to be reviewed, and the image recognition model is used to automatically filter out images or videos that do not contain the above-mentioned objects.

[0050] Exemplarily, a pre-trained CLIP model is used as an image recognition model for image recognition to recognize an image. The image recognition model includes a text encoder for processing prompt words and an image encoder for processing images. During the recognition process, the image recognition model can call the text encoder to process the preset prompt words respectively, and at the same time, call the image encoder to process the image, so as to realize the recognition of the image based on the processing results of the two encoders.

[0051] Therefore, in the process of training the above-mentioned image recognition model, the two encoders need to be trained at the same time, during which the number of training samples and the training time required are greatly increased, resulting in a decrease in the training efficiency of the image recognition model.

[0052] Based on this, how to improve the training efficiency of image recognition models used to recognize images has become a technical problem that urgently needs to be solved.

[0053] In order to solve the above technical problems, an embodiment of the present application provides a model training method.

[0054] Among them, a model training method provided in an embodiment of the present application can be applied to an electronic device, and the electronic device is specifically used for training the model. In a specific application, the electronic device can be a smart phone, a tablet computer, etc., which is all reasonable.

[0055] A model training method provided in an embodiment of the present application may include the following steps:

[0056] Acquire a sample image; wherein the sample image is provided with a label for characterizing a presentation state of the sample object;

[0057] The sample image is input into the image recognition model to be trained to obtain an output result corresponding to each sample prompt word; wherein the image recognition model includes a fully connected layer with a predetermined initial weight parameter and an image encoder in a pre-trained contrastive learning model; each sample prompt word is an image description text with the presentation state of the sample object as the description content, and the initial weight parameter is predetermined based on the vector features of each sample prompt word; the output result corresponding to each sample prompt word is used to characterize the similarity between the vector features of the sample prompt word and the vector features of the sample image;

[0058] Determine a loss value corresponding to the image recognition model based on the output result corresponding to each sample prompt word and the label of the sample image;

[0059] When it is determined based on the loss value that the image recognition model does not meet the model convergence condition, the current weight parameters of the fully connected layer and the image encoder are adjusted based on the loss value, and the process returns to the step of obtaining the sample image.

[0060] It can be seen from the above that, by applying the solution provided in the embodiment of the present application, before model training, an image recognition model to be trained can be pre-constructed, and the image recognition model includes a fully connected layer with predetermined initial weight parameters and an image encoder in a pre-trained contrastive learning model. The image recognition model can be used to determine the similarity between the vector features of the prompt word and the vector features of the input image. Therefore, before model training, each sample prompt word can be obtained, wherein each sample prompt word is an image description text with the presentation state of the sample object as the description content. Then, based on the vector features of each sample prompt word, the initial weight parameters of the fully connected layer are predetermined to provide the image recognition process with vector features about the prompt word through the fully connected layer with the initial weight parameters.

[0061] In this way, when training the model, it is only necessary to input the acquired sample image into the image recognition model to be trained to obtain the output result corresponding to each sample prompt word, and the output result is used to characterize the similarity between the vector feature of the sample prompt and the vector feature of the sample image. Then, based on the output result corresponding to each sample prompt word and the pre-set label of the sample image itself, the loss value of the weight parameter of the fully connected layer of the image recognition model and the loss value of the image encoder when acquiring the vector feature of the sample image are determined, and the weight parameter of the fully connected layer of the image recognition model and the image encoder are adjusted based on the determined loss value. In this way, by reconstructing the model structure of the image recognition model, the training effect of training two encoders can be achieved by training the number of training samples and training time required for one model, that is, by reducing the number of encoders required to be trained during model training, the number of training samples and training time required during training are reduced, thereby improving the training efficiency of the image recognition model.

[0062] The following is a detailed description of a model training method provided in an embodiment of the present application in conjunction with the accompanying drawings.

[0063] Figure 2 A flow chart of a model training method provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the method may include the following steps:

[0064] S201: Acquire a sample image.

[0065] The sample image is provided with a label for characterizing the presentation status of the sample object.

[0066] In this application, since the public data set may not contain training data related to actual business scenarios, for example, the public data set does not contain training data related to tattoos. Therefore, the image recognition model trained using the public data set has a low recognition accuracy for highly targeted objects. Taking the object as an example, if there is no training data related to tattoos in the public data set, the image recognition model trained using the public data set has a low recognition accuracy for whether there are tattoos in the image.

[0067] Therefore, an image provided with a label characterizing the presentation state of the sample object can be acquired as a sample image.

[0068] It is understandable that the number of sample images acquired may be one or more, and this is not specifically limited in the embodiments of the present application.

[0069] Among them, the presentation state of the above-mentioned sample object can be understood as whether the sample object exists in the sample image, for example, whether there is smoke in the image, or whether there is a cat in the image, etc.; it can also be understood as whether there is a sample object in a certain state or belonging to a certain attribute in the sample image, for example, whether there is smoke in a burning state in the image, or whether there is a white cat in the image, etc., which are all reasonable.

[0070] It should be noted that the label used to characterize the presentation status of the sample object may be a digital label or a text label, and the embodiment of the present invention does not specifically limit this. For example, if the sample object exists in the sample image, the label of the sample image may be set to 1; if the sample object does not exist in the sample image, the label of the sample image may be set to 0.

[0071] Optionally, in one implementation, the sample object is a tattoo.

[0072] In this implementation, if the sample object is a tattoo, the acquired sample image is an image of the tattoo. Accordingly, the presentation status of the sample object may include: the presence or absence of a tattoo, the presence or absence of a dark tattoo, etc.

[0073] S202: Input the sample image into the image recognition model to be trained to obtain the output result corresponding to each sample prompt word;

[0074] Among them, the image recognition model includes a fully connected layer with a predetermined initial weight parameter and an image encoder in a pre-trained contrastive learning model; each sample prompt word is an image description text with the presentation state of the sample object as the description content, and the initial weight parameter is predetermined based on the vector features of multiple sample prompt words; the output result corresponding to each sample prompt word is used to characterize the similarity between the vector feature of the sample prompt word and the vector feature of the sample image.

[0075] In the present application, it is considered that in the prior art, a contrastive learning model is usually used as an image recognition model for image recognition. When performing image recognition, this type of model calls a text encoder and an image encoder to process the input prompt word and image respectively, that is, calls the text encoder to process the input prompt word to obtain the vector features of the prompt word, and calls the image encoder to process the input image to obtain the vector features of the image. In this way, this type of model can recognize the image by determining the similarity between the vector features of the prompt word and the vector features of the image.

[0076] That is to say, the key to the image recognition model's ability to recognize an image is to obtain the vector features of the prompt word and the vector features of the image, and to determine the similarity of the two vector features obtained. At this time, if the image recognition model is trained for a certain type of object or a certain type of practical application scenario, then it can be understood that the prompt word related to a certain type of object or a certain type of practical application scenario can be fixed, and then, the vector features of the predetermined dimension of the generated prompt word are also fixed within a certain range.

[0077] Furthermore, considering that the role of the fully connected layer in the entire image recognition model is to process multimodal data, and the process of processing multimodal data in the embodiment of the present application is: using the vector features of the sample image and the vector features of each sample prompt word, calculate the similarity of each sample prompt word to the sample image. Therefore, when the vector features of the generated sample prompt words are fixed within a certain range, the parameters determined based on the vector features of multiple sample prompt words can be used as the current weight parameters of the fully connected layer itself, so as to provide the vector features of the sample prompt words for the image recognition process through the weight parameters of the fully connected layer.

[0078] Therefore, the fully connected layer can be spliced ​​after the image encoder through data association to achieve the reconstruction of the internal structure of the image recognition model and obtain the image recognition model to be trained. Before the model training starts, the image description texts with the presentation status of the sample object as the description content are used as the sample prompt words, and then the vector features of each sample prompt word are obtained, and the matrix determined based on the vector features of multiple sample prompt words is used as the initial weight parameter of the fully connected layer to provide the vector features of the sample prompt words for subsequent data processing.

[0079] That is, the image recognition model to be trained includes a fully connected layer with predetermined initial weight parameters and an image encoder in a pre-trained contrastive learning model.

[0080] In this way, after obtaining the sample image, the sample image can be input into the image recognition model to be trained to obtain the similarity between the vector feature of each sample prompt word and the vector feature of the sample image as the output result corresponding to the sample prompt word.

[0081] Among them, the feature score can be used to represent the output result corresponding to each sample prompt word, that is, the feature score is used to represent the similarity between the vector feature of the sample prompt word and the vector feature of the sample image. This embodiment of the application does not make any specific limitation on this.

[0082] Optionally, in one implementation, the above step S202, inputting the sample image into the image recognition model to be trained to obtain the output result corresponding to each sample prompt word, may include the following steps:

[0083] Step A: Input the sample image into the image encoder in the image recognition model to be trained so that the image encoder generates vector features of the sample image, and transmits the vector features of the sample image to the fully connected layer to process the vector features of the sample image based on the current weight parameters of the fully connected layer itself to obtain the output result corresponding to each sample prompt word.

[0084] In this implementation, after obtaining the sample image, the sample image can be input into the above-mentioned image recognition model to be trained, so that the image encoder in the image recognition model generates vector features of the sample image, and the generated vector features of the sample image are transmitted to the fully connected layer in the image recognition model, so as to process the vector features of the sample image based on the current weight parameters of the fully connected layer itself, that is, calculate the similarity between the vector features of each sample prompt word and the vector features of the sample image, thereby obtaining the output result corresponding to each sample prompt word.

[0085] Optionally, cosine similarity is used to calculate the distance between the vector features of each sample prompt word and the vector features of the sample image; specifically, the closer the distance between the vector features of the sample prompt word and the vector features of the sample image, the higher the similarity between the two vector features, and conversely, the farther the distance between the vector features of the sample prompt word and the vector features of the sample image, the lower the similarity between the two vector features.

[0086] Optionally, considering that different data categories, such as text data, image data, etc., have corresponding vector features in different dimensions, in order to facilitate the subsequent determination of the similarity between the vector features of each sample prompt word and the vector features of the sample image, the vector features corresponding to different data types can be unified into the same predetermined dimension. Therefore, after obtaining the sample image, the image encoder of the image recognition model extracts the image features of the sample image, and generates vector features of a predetermined dimension of the sample image based on the extracted image features. Among them, the vector features of the image features can be represented by a feature matrix, and this is not specifically limited in the embodiments of the present application.

[0087] In addition, considering that the image sizes of the acquired sample images are different, in order to ensure that each acquired sample image can be converted into image features of the same dimension, so as to be further converted into vector features of a predetermined dimension, the sample image can be scaled to a predetermined size before generating the vector features of the sample image, so as to be used in the process of generating the vector features. Based on this, optionally, after acquiring the sample image, the sample image is scaled to a predetermined size, and after image features are extracted from the sample image of the predetermined size, the extracted image features are converted into vector features of a predetermined dimension.

[0088] Exemplarily, after acquiring a sample image, the image size of the sample image is scaled to a fixed size, for example, 224×224, or 336×336, and then the scaled sample image is input into a module for generating image vector features to generate vector features of the sample image.

[0089] In this implementation, the dimensions of the image features of the sample image are converted to obtain vector features of the sample image at a predetermined dimension, thereby facilitating subsequent determination of the similarity between the vector features of each sample prompt word and the vector features of the sample image.

[0090] It should be noted that the embodiments of the present application involve a text-image pre-training model based on contrastive learning, and the main ability of this type of model is to align image features and text features in the same feature space (dimension). Therefore, in theory, only the trained model layer in the pre-training model can be used to convert image or text features to vector features, that is, the image features are converted to vector features through the image encoder in the pre-training model, and the text features are converted to vector features through the text encoder in the pre-training model. Other untrained methods cannot be used, otherwise, the vector features of the image and the vector features of the text obtained will not be in the same feature space from a mathematical perspective.

[0091] Based on this, optionally, the sample image is input into an image encoder of a pre-constructed image recognition model to be trained, so that the image encoder extracts image features of the sample image and converts the extracted image features to obtain vector features of the sample image; each sample prompt word is input into a text encoder of a pre-trained contrastive learning model corresponding to the image recognition model before construction, so that the text encoder extracts text features of the sample prompt word and converts the extracted text features to obtain vector features of the sample prompt word, and then, the obtained vector features of the sample prompt word are used as matrix elements contained in the initial weight parameters of the fully connected layer.

[0092] In this way, since the generation method of the vector features of the sample image and the vector features of the sample prompt words is pre-trained, the obtained vector features of the sample image and the vector features of the sample prompt words are in the same feature space from a mathematical perspective, that is, the vector features obtained using the image encoder of the above-mentioned image recognition model and the text encoder of the contrastive learning model are in the same predetermined dimension.

[0093] S203: Based on the output result corresponding to each sample prompt word and the label of the sample image, determine the loss value corresponding to the image recognition model.

[0094] In this application, since the sample image is provided with a label for representing the presentation state of the sample object, that is, the presentation state of the sample object represented by the label of the sample image is the actual state presented by the sample object in the sample image. Therefore, after obtaining the output result corresponding to each sample prompt word, the loss value corresponding to the image recognition model can be determined through the above output result and the label of the sample image.

[0095] S204: When it is determined based on the loss value that the image recognition model does not meet the model convergence condition, the current weight parameters of the fully connected layer and the image encoder are adjusted based on the loss value, and the process returns to step S201.

[0096] In the present application, after determining the loss value corresponding to the image recognition model, it is possible to further determine whether the image recognition model meets the model convergence conditions, that is, to determine whether the loss value reaches a predetermined loss value.

[0097] When it is determined based on the loss value that the image recognition model does not meet the model convergence conditions, the current weight parameters of the fully connected layer and the image encoder can be adjusted based on the loss value, and the process returns to step S201 to continue training the image recognition model to be trained based on the next sample image using a model training method provided in an embodiment of the present application.

[0098] When it is determined based on the loss value that the image recognition model meets the model convergence conditions, the model training is terminated.

[0099] Optionally, in order to improve the training efficiency of the model training, the model training end condition can be set to the model training times reaching a predetermined number of times, that is, when the model training times reaches the predetermined number of times, the model training is ended.

[0100] Optionally, a back propagation algorithm is used to calculate the gradient value of the feature vector corresponding to the current weight parameter of the fully connected layer and the encoder parameter of the image encoder based on the loss value, and the calculated gradient value is used to adjust the current weight parameter of the fully connected layer and the encoder parameter of the image encoder. Thus, in the process of model training, the model parameters are adjusted according to the determined loss value to balance the above loss value and optimize the model performance.

[0101] It can be seen from the above that, by applying the solution provided in the embodiment of the present application, before model training, an image recognition model to be trained can be pre-constructed, and the image recognition model includes a fully connected layer with predetermined initial weight parameters and an image encoder in a pre-trained contrastive learning model. The image recognition model can be used to determine the similarity between the vector features of the prompt word and the vector features of the input image. Therefore, before model training, each sample prompt word can be obtained, wherein each sample prompt word is an image description text with the presentation state of the sample object as the description content. Then, based on the vector features of each sample prompt word, the initial weight parameters of the fully connected layer are predetermined to provide the image recognition process with vector features about the prompt word through the fully connected layer with the initial weight parameters.

[0102] In this way, when training the model, it is only necessary to input the acquired sample image into the image recognition model to be trained to obtain the output result corresponding to each sample prompt word, and the output result is used to characterize the similarity between the vector feature of the sample prompt word and the vector feature of the sample image. Then, based on the output result corresponding to each sample prompt word and the pre-set label of the sample image itself, the loss value of the current weight parameter of the fully connected layer of the image recognition model and the loss value of the image encoder when acquiring the vector feature of the sample image are determined, and then the current weight parameter of the fully connected layer of the image recognition model and the image encoder are adjusted based on the determined loss value. In this way, by reconstructing the model structure of the image recognition model, the training effect of training two encoders can be achieved by training the number of training samples and training time required for one model, that is, by reducing the number of encoders required to be trained during model training, the number of training samples and training time required during training are reduced, thereby improving the training efficiency of the image recognition model.

[0103] Optionally, in one embodiment, a method for determining the initial weight parameters of the fully connected layer includes the following steps:

[0104] Step B: Obtain the vector features of each sample prompt word, and concatenate the vector features of each sample prompt word as a row of matrix elements to obtain the initial weight parameters of the fully connected layer.

[0105] In this embodiment, after obtaining each sample prompt word, the vector features of the sample prompt word can continue to be obtained, so that the vector features of each sample prompt word are spliced ​​in a manner of splicing the vector features of each sample prompt word as a row of matrix elements to obtain the initial weight parameters of the fully connected layer.

[0106] Optionally, an image description text with the presentation status of the sample object as the description content is obtained in advance as a sample prompt word; wherein the above-mentioned sample prompt word can be a single word or a long sentence composed of multiple words, and the embodiment of the present application does not make any specific limitation on this.

[0107] In this way, after obtaining each sample prompt word, the text feature of each sample prompt word can be extracted. Considering different data types, such as text data, image data, etc., the corresponding vector features are in different dimensions. In order to facilitate the subsequent determination of the similarity between the vector features of each sample prompt word and the vector features of the sample image, the vector features corresponding to different data types can be unified into the same predetermined dimension. Therefore, after obtaining the text feature of each sample prompt word, the vector feature of the predetermined dimension of the sample prompt word can be generated.

[0108] Since the feature matrix can be used to represent the vector features of the sample prompt words, after obtaining the vector features of multiple sample prompt words, the vector features of the multiple sample prompt words can be spliced ​​into a matrix to obtain the initial weight parameters of the fully connected layer.

[0109] Optionally, a target keyword and other non-keywords that can be combined with the target keyword are set in advance for a presentation state of the sample object. In this way, through the combination of a target keyword and other non-keywords, an image description text with the presentation state of the sample object as the description content can be obtained as a sample prompt word, thereby obtaining multiple sample prompt words.

[0110] The combination of the above target keywords and other non-keywords may include:

[0111] (1) For different presentation states of the same sample object, corresponding target keywords are set, and for each target keyword, other non-keywords are randomly combined with the target keyword to obtain multiple sample prompt words. For example, when the sample object is a tattoo, the presentation state of the sample object may include: the presence of a tattoo and the absence of a tattoo. In this case, the target keywords that can be set are the presence of a tattoo and the absence of a tattoo.

[0112] (2) For a presentation state of the same sample object, a corresponding target keyword is set, and multiple sample prompt words are obtained by randomly combining the target keyword with other non-keywords. For example, when the sample object is a tattoo, the presentation state of the sample object may include: the presence of a tattoo and the absence of a tattoo. In this case, the target keyword that can be set is one of the presence of a tattoo and the absence of a tattoo.

[0113] The embodiment of the present application does not limit the number of target keywords in the multiple sample prompt words, nor the number of other non-keywords.

[0114] In this embodiment, considering that the training process of the fully connected layer requires a large number of sample prompt words, before training the image recognition model to be trained, the initial weight parameters of the fully connected layer are predetermined. While reducing the number of training samples required for the training process, pre-training parameters about the fully connected layer can be provided for the model training process. In this way, in the process of training the image recognition model, only sample images need to be used to enable the trained image recognition model to obtain better training results.

[0115] Moreover, by adjusting the current weight parameters of the fully connected layer and the parameters of the image encoder in a reconstructed image recognition model, the image recognition model can be trained. There is no need to train the text encoder and image encoder of the comparative learning model in the prior art, thereby reducing the number of encoders that need to be trained, thereby improving the efficiency of model training.

[0116] Optionally, in one implementation, the above step S203, based on the output result corresponding to each sample prompt word and the label of the sample image, determines the loss value corresponding to the image recognition model, which may include the following steps:

[0117] Step C1: based on the output results corresponding to each sample prompt word, determine the target confidence corresponding to the target sample prompt word;

[0118] The presentation state described by the target sample prompt word is the same as the presentation state of the sample object represented by the label of the sample image; the target confidence is used to represent the presentation state described in the target sample prompt word, which is the predicted probability of the presentation state of the sample object in the sample image;

[0119] Step C2: Based on the target confidence and the label of the sample image, determine the loss value corresponding to the image recognition model.

[0120] In this implementation, since the sample image is provided with a label for characterizing the presentation state of the sample object, and each sample prompt word is an image description text with the presentation state of the sample object as the description content, that is to say, the sample prompt word can be understood as a label set on the sample image, and in a model training process, each sample image will only be provided with one label, that is, there is only one target sample prompt word corresponding to the sample image among multiple sample prompt words, and the presentation state described by the target sample prompt word is the same as the presentation state of the sample object represented by the label of the sample image.

[0121] Therefore, the loss value determined based on the output result corresponding to each sample prompt word and the label of the sample image can be understood as being caused by the low accuracy of the vector features of the target sample prompt word in the current weight parameters of the fully connected layer of the image recognition model. Therefore, the loss value of the image recognition model during the training process can be determined based on the target confidence corresponding to the target sample prompt word and the label of the sample image.

[0122] Therefore, after determining the output result corresponding to each sample prompt word, the target confidence corresponding to the target sample prompt word is further determined based on each output result to determine the prediction probability that the presentation state described in the target sample prompt word is the presentation state of the sample object in the sample image. The higher the target confidence obtained, the higher the prediction probability that the presentation state described in the target sample prompt word is the presentation state of the sample object in the sample image.

[0123] In this way, based on the target confidence corresponding to the target sample prompt word and the label of the sample image, the loss value corresponding to the image recognition model can be determined. The smaller the determined loss value is, the closer the presentation state described in the target sample prompt word is to the presentation state of the sample object in the sample image.

[0124] Among them, the label set by the sample image for representing the presentation state of the sample object can be understood as a label used to represent the actual state presented by the sample object in the sample image. Therefore, the corresponding confidence can be set directly based on the actual state presented by the sample object. For example, if the sample object is a tattoo, one presentation state represented by the label is the presence of a tattoo, and the corresponding confidence of the presentation state represented by the label can be set to 1; conversely, if one presentation state represented by the label is the absence of a tattoo, the corresponding confidence of the presentation state represented by the label can be set to 0.

[0125] Optionally, in one implementation, the target confidence corresponding to the target sample prompt word and the confidence corresponding to the label of the sample image are input into a predetermined loss function to calculate the loss value corresponding to the image recognition model. The predetermined loss function may include a cross entropy loss function, etc., which is not specifically limited in the embodiments of the present application.

[0126] In this implementation, the confidence level is used to digitize the probability that the presentation state described in the target sample prompt word is the presentation state of the sample object corresponding to the sample image, so as to determine the loss value of the image recognition model.

[0127] Optionally, in one implementation, the above step C1, based on the output result corresponding to each sample prompt word, determines the target confidence corresponding to the target sample prompt word, and may include the following steps:

[0128] Step C11: determining the target confidence corresponding to the target sample prompt word based on the output result corresponding to the target sample prompt word and the ratio of the target and value;

[0129] Among them, the target and value are obtained by summing the output results corresponding to each sample prompt word.

[0130] In this implementation, after obtaining the output result corresponding to each sample prompt word, the sum of the output results corresponding to each sample prompt word can be calculated as the target sum value. Thus, based on the ratio of the output result corresponding to the target sample prompt word to the above target sum value, the target confidence corresponding to the target sample prompt word can be determined.

[0131] Optionally, for each sample prompt word, the output result corresponding to the sample prompt word is normalized to obtain the confidence corresponding to the sample prompt word, and then the confidence corresponding to the target sample prompt word is selected from each confidence as the target confidence corresponding to the target sample prompt word.

[0132] Optionally, for each sample prompt word, a confidence level corresponding to the sample prompt word is calculated using a predetermined formula; wherein the predetermined formula includes:

[0133]

[0134] σ(z i ) represents the confidence corresponding to the i-th sample prompt word of the sample object, K represents the total number of sample prompt words of the sample object, z i Characterize the feature score corresponding to the i-th sample prompt word, z j Characterize the feature score corresponding to the j-th sample prompt word.

[0135] Among them, the feature score corresponding to each of the above-mentioned sample prompt words is used to characterize the similarity between the sample prompt word and the sample image, that is, the similarity between the vector features of each sample prompt word in the embodiment of the present application and the vector features of the sample image, as well as the output result corresponding to each sample prompt word in the embodiment of the present application.

[0136] In this implementation, by normalizing the similarity between the vector features of each sample prompt word and the vector features of the sample image to a specific interval, for example, [0, 1] or [-1, 1], etc., the data distribution is made statistically significant, and the probability distribution of similarity is more statistically intuitive, thereby improving the comparability of the output results corresponding to each sample prompt word.

[0137] The following is a specific example in which a model training method provided in the present application is specifically introduced.

[0138] Since conventional deep learning recognition models need to label large quantities of data, and in actual business, the data that needs to be recognized has a long-tail distribution, that is, the data that needs to be recognized accounts for a small proportion of the total data and is difficult to collect, resulting in high data labeling costs and slow model iteration cycles. Therefore, considering that the text-image pre-training model based on contrastive learning does not need to collect data to complete the image recognition ability, such as contrastive learning models such as the CLIP model, this type of model can be used for zero-sample prediction to achieve automatic recognition of images. However, since the above-mentioned contrastive learning model is trained based on pre-training data obtained from a public data set, and the above-mentioned public data set may not contain business data related to the actual scene, for example, the public data set does not contain the features required for identifying tattoos, resulting in the trained model having a low recognition accuracy for images with tattoos during application in actual scenes.

[0139] Moreover, when training the contrastive learning model, the text encoder and image encoder contained in the contrastive learning model need to be trained simultaneously, during which the number of required training samples and the training time are greatly increased, resulting in reduced training efficiency of the model.

[0140] For example, Figure 3 As shown in FIG. , it is a flowchart of an existing training method of a contrastive learning model. Figure 3As shown, in the training process of the existing contrastive learning model, it is necessary to obtain the key prompt words with airplane, butterfly and pizza as the objects respectively, combine the key prompt words with other non-keywords, and then output the combined prompt words to the Text Encoder of the CLIP model to obtain the Text features corresponding to each prompt word; then, input the image to the Image Encoder of the CLIP model to obtain the Image features corresponding to the image; then, calculate the Similarity scores of the text features and the image features as the similarity between the two, and thus, based on the determined similarity, calculate the loss value of the Image Encoder and the loss value of the Text Encoder. By Figure 3 It can be seen that when training the existing contrastive learning model, the text encoder and the image encoder need to be trained at the same time, which makes the training efficiency of the model low.

[0141] For example, Figure 4 As shown, this is a schematic diagram of the structure of a conventional deep learning recognition model.

[0142] Considering that in the recognition process of a conventional deep learning recognition model, after an image is input into the deep learning recognition model, a feature extraction backbone network (the feature extraction backbone network may include convolution layer 1, pooling layer 1, convolution layer 2 and pooling layer 2) is used to extract the image features of the image, and then one to three fully connected layers (fully connected layer 1, fully connected layer 2 and fully connected layer 3, respectively) are used to calculate the logistic regression score of the image features to the specific label, and then all the label logistic regression scores are summarized through a softmax function to obtain the confidence of each label, thereby obtaining the classification result of the input image. That is to say, in a conventional deep learning recognition model, the fully connected layer of the deep learning recognition model is used to calculate the logistic regression score of the key prompt words represented by the image features and the specific labels. The actual function of the fully connected layer is to determine the similarity between the features of the content described by the specific label and the features of the image, that is, by dot multiplication and addition, and dividing by the vector length of each text multimodal feature and image multimodal feature, the similarity between the features of the content described by the specific label and the features of the image is calculated.

[0143] Therefore, based on the structural characteristics of the conventional deep learning recognition model and the model characteristics of the CLIP model, this specific embodiment provides a model training method, which adjusts the structure of the CLIP model on the basis of the CLIP model, converts the CLIP model into a conventional deep learning recognition model, and performs model training on the converted model to achieve model tuning. In this way, while ensuring the excellent performance of the CLIP model itself, such as the ability to complete image recognition without collecting data, the cost of training and tuning the model is greatly reduced.

[0144] The above specific embodiment is introduced below in combination with specific implementation steps, wherein the sample object in this specific embodiment is a tattoo.

[0145] Step 1: set initial key prompt words, combine the key prompt words, and use a text encoder to convert the combined prompt words into text multimodal features; wherein the combined prompt words are the sample prompt words in the embodiment of the present application.

[0146] Specifically, before model training, multiple initial key prompt words for tattoo classification can be set. For example, if the image has a tattoo, the initial key prompt word is set to "a photo of person with a tattoo"; if the image does not have a tattoo, the initial key prompt word is set to "a photo of person without a tattoo". Then, the multiple initial key prompt words are randomly combined with other non-keywords to obtain combined prompt words. The multiple combined prompt words are input into the text encoder of the CLIP model to obtain the text features of each combined prompt word, and then each text feature is passed through a text multimodal network to convert the text feature into a text multimodal feature (i.e., the vector feature of the prompt word in this application). Here, the text multimodal feature corresponding to each combined prompt word is a feature matrix of size [1, 512].

[0147] Among them, the initial key prompt word is the target keyword in the embodiment of the present application.

[0148] The specific implementation method corresponding to the above step 1 is similar to the method of obtaining the vector features of the sample prompt word in the embodiment of the present application, and will not be repeated here.

[0149] Step 2: Convert the text multimodal features into fully connected layer weights, and splice the fully connected layer with the added weights behind the corresponding image multimodal network to obtain the recognition model to be optimized; wherein the above-mentioned recognition model to be optimized is the image recognition model to be trained in the embodiment of the present application; the above-mentioned fully connected layer weights are the initial weight parameters of the fully connected layer in the embodiment of the present application.

[0150] Specifically, considering that the actual role of the fully connected layer in the deep learning recognition model is to calculate the similarity between the features of the content described by the specific label and the features of the image by adding the dot product and dividing by the vector length of the multimodal features of the content described by each specific label and the multimodal features of the image. In the actual reasoning process of the image recognition model, it is also necessary to calculate the similarity between the text multimodal features and the image multimodal features in the above manner. Therefore, it can be considered that the role of the acquired text multimodal features is consistent with the role of the fully connected layer in the conventional deep learning recognition model. The various text multimodal features predetermined in the above step one can be converted into fully connected layer weights, and the fully connected layer with the weights added can be spliced ​​behind the image multimodal network to obtain the recognition model to be tuned.

[0151] Since the image encoder sets a fixed preset size for the size of the input image, the image is scaled to the fixed preset size before being input into the image encoder, wherein the fixed preset size is usually 224×224 or 336×336, which are both reasonable and are not specifically limited in the embodiments of the present application.

[0152] In this way, after receiving the input image, the image encoder in the recognition model to be tuned can extract the image features of the image, and pass the image features through the image multimodal network in the image encoder to convert the image features into image multimodal features (vector features of the image in the embodiment of the present application). Here, the image multimodal features corresponding to each image are feature matrices of size [1, 512].

[0153] The specific implementation method corresponding to the above step 2 is similar to the method of obtaining the vector features of the sample image in the embodiment of the present application, and will not be repeated here.

[0154] Step 3: Use the back propagation algorithm to tune the spliced ​​model to obtain an image recognition model tuned based on the contrastive learning model.

[0155] Specifically, multiple text multimodal features and image multimodal features are sequentially dot-multiplied and added, and then divided by the vector length of each text multimodal feature and image multimodal feature to calculate the similarity of each text multimodal feature and image multimodal feature, thereby obtaining the feature score between the text feature of each combined prompt word and the feature of the input image. The feature score is used to characterize the similarity between each text feature and the image feature of the image.

[0156] Then, define the feature score as z and perform softmax (normalization) operation on all feature scores:

[0157]

[0158] σ(z i ) represents the confidence of the prompt word corresponding to the i-th combination of tattoos, K represents the total number of prompt words about tattoos, z i Characterizes the feature score corresponding to the prompt word obtained after the i-th combination, z j Characterizes the feature score corresponding to the prompt word obtained after the j-th combination.

[0159] Through the above formula, the feature score between the text feature of the prompt word and the image feature of the image obtained after each combination can be mapped to a confidence level between 0 and 1 as the confidence level of the prediction.

[0160] Then, based on the cross-entropy loss function, the loss value between the confidence predicted by the model and the label of the image itself is calculated, and the back propagation algorithm is used to update the weight of the fully connected layer based on the loss value, that is, the text modal feature weights of the prompt words obtained after multiple combinations, and the image encoder weight features; wherein, the weight of the above-mentioned fully connected layer and the text modal feature weights of the prompt words obtained after multiple combinations are the matrix obtained by splicing the vector features of multiple sample prompt words in the embodiment of the present application, that is, the initial weight parameters of the fully connected layer in the embodiment of the present application, wherein, if this update is not the first update, the updated weight of the fully connected layer is the current weight parameter of the fully connected layer itself in the embodiment of the present application.

[0161] In this way, the above steps 2 and 3 are repeated until all images in the data set are extracted, and gradient descent calculation is performed to optimize the recognition model to be tuned.

[0162] The above steps can be regarded as one round of model training. Repeat multiple rounds of training until the preset number of training rounds or the loss value reaches a preset value.

[0163] Compared with the use of traditional models and traditional training methods, this embodiment only requires a small number of samples to obtain a higher model accuracy, greatly reducing the difficulty of data collection in scenarios with long-tail distribution data characteristics such as audit business. Moreover, compared with the training process of the existing comparative learning model, in this embodiment, only the image encoder and the fully connected layer of the recognition model to be tuned need to be trained, so as to achieve the effect of training two encoders. In other words, by reducing the number of encoders required to be trained during model training, the number of training samples required during training and the training time are reduced, thereby improving the training efficiency of the image recognition model.

[0164] Moreover, in the subsequent reasoning process of the trained image recognition model, the text encoder can be discarded, and only the weight parameters of the image encoder and the fully connected layer can be retained, thereby improving the image recognition efficiency of the subsequent image recognition model.

[0165] The following is a detailed introduction to an image recognition method provided in an embodiment of the present application.

[0166] Among them, an image recognition method provided in the embodiment of the present application can be applied to an electronic device, and the electronic device is specifically used for image recognition. In a specific application, the electronic device can be a smart phone, a tablet computer, etc., which is reasonable. Among them, the electronic devices used for image recognition and model training can be the same electronic device or different electronic devices, and the embodiment of the present application does not make specific limitations.

[0167] An image recognition method provided in an embodiment of the present application may include the following steps:

[0168] Get the target image;

[0169] The target image is input into a pre-trained image recognition model to obtain an output result corresponding to each predetermined prompt word; wherein the image recognition model is trained based on a model training method provided in an embodiment of the present application; the predetermined prompt words are sample prompt words used to train the image recognition model; the output result corresponding to each predetermined prompt word is used to characterize the similarity between the vector feature of the predetermined prompt word and the vector feature of the target image;

[0170] Based on the output result of each predetermined prompt word, a presentation state of the target image with respect to the target object is determined.

[0171] From the above, it can be seen that in the image recognition method provided in the embodiment of the present application, since the image recognition model used is trained based on a model training method provided in the embodiment of the present application, that is to say, the vector features of the predetermined prompt words used for image recognition in the image recognition model used are fixed, when using the image recognition model for image recognition, there is no need to input the prompt words, only the target image needs to be input to realize automatic recognition of the image.

[0172] Below, an image recognition method provided in an embodiment of the present application is specifically introduced in conjunction with the accompanying drawings.

[0173] Figure 5 A flowchart of an image recognition method provided in an embodiment of the present application is shown in FIG. Figure 5 As shown, the method may include the following steps:

[0174] S501: Acquire a target image;

[0175] The target image is an image used to identify the target object.

[0176] In the present application, when performing image recognition, an image used to identify a target object can be obtained as a target image, thereby executing an image recognition method provided by an embodiment of the present application on the target image.

[0177] Optionally, since the image recognition model has a size restriction range for the input image size, after acquiring the target image, the image size of the target image can be scaled to a predetermined size according to the size restriction range of the comparison model to be input. The predetermined size is within the size restriction range of the image recognition model to be input.

[0178] It should be noted that the acquisition, storage, use, processing, transmission, provision and disclosure of images involved in the technical solution of this application are all carried out with the authorization of the user. In addition, the images of the human body involved in this application are not images of the human body of a specific user and do not reflect the personal information of a specific user.

[0179] S502: Input the target image into a pre-trained image recognition model to obtain an output result corresponding to each predetermined prompt word;

[0180] Among them, the image recognition model is trained based on a model training method provided in the above-mentioned embodiment of the present application; the predetermined prompt words are sample prompt words used to train the image recognition model; the output result corresponding to each predetermined prompt word is used to characterize the similarity between the vector feature of the predetermined prompt word and the vector feature of the target image.

[0181] In the present application, the image recognition model used is trained based on a model training method provided in the embodiment of the present application, that is, the vector features of the prompt words used for image recognition in the image recognition model used are fixed, that is, the vector features of the predetermined prompt words used for image recognition are the same vector features of the sample prompt words used for training the image recognition model. Therefore, when using the image recognition model for image recognition, it is not necessary to input the prompt words, only the target image is required.

[0182] Therefore, after the target image is input into the image recognition model, an output result corresponding to each predetermined prompt word can be obtained, and the output result is used to characterize the similarity between the vector feature of the predetermined prompt word and the vector feature of the target image.

[0183] S503: Determine the presentation status of the target image for the target object based on the output result corresponding to each predetermined prompt word.

[0184] In the present application, the output result corresponding to each predetermined prompt word is obtained, that is, the similarity between the vector feature of the predetermined prompt word and the vector feature of the target image is clarified. Therefore, based on the output result corresponding to each predetermined prompt word, the similarity between the presentation state described by the predetermined prompt word and the presentation state of the target image for the target object can be determined. The higher the determined similarity, the higher the possibility that the presentation state described by the prompt word is the presentation state of the target object in the target image. Therefore, after obtaining the output result corresponding to each predetermined prompt word, the presentation state of the target image for the target object can be determined based on the above output result.

[0185] From the above, it can be seen that in the image recognition method provided in the embodiment of the present application, since the image recognition model used is trained based on a model training method provided in the embodiment of the present application, that is to say, the vector features of the predetermined prompt words used for image recognition in the image recognition model used are fixed, when using the image recognition model for image recognition, there is no need to input the prompt words, only the target image needs to be input to realize automatic recognition of the image.

[0186] Optionally, in one implementation, the above step S503, based on the output result corresponding to each predetermined prompt word, determining the presentation state of the target image for the target object, may include the following steps:

[0187] Step D1: based on the output result corresponding to each predetermined prompt word, determine the confidence level corresponding to the predetermined prompt word;

[0188] The confidence level corresponding to the predetermined prompt word is used to characterize the presentation state described in the predetermined prompt word, which is the probability of the presentation state of the target object corresponding to the target image;

[0189] Step D2: Analyze the presentation status of the target image with respect to the target object based on the confidence level corresponding to each predetermined prompt word.

[0190] In this implementation, for each predetermined prompt word, after determining the output result corresponding to the predetermined prompt word, the confidence level corresponding to the predetermined prompt word can be further determined, thereby determining the probability that the presentation state described in the predetermined prompt word is the presentation state corresponding to the target image with respect to the target object. In other words, the higher the confidence level corresponding to the predetermined prompt word, the higher the probability that the presentation state described in the predetermined prompt word is the presentation state corresponding to the target image with respect to the target object. Thus, based on the confidence level corresponding to each predetermined prompt word, the presentation state of the target image with respect to the target object is analyzed.

[0191] In this implementation, the confidence level is used to digitize the probability that the presentation state described in each predetermined prompt word is the presentation state of the target object corresponding to the target image, so as to match the target image with the corresponding predetermined prompt word.

[0192] Optionally, in one implementation, the above step D2, analyzing the presentation status of the target image to the target object based on the confidence level corresponding to each predetermined prompt word, may include the following steps:

[0193] Step D21: Select the highest confidence level from the confidence levels corresponding to each predetermined prompt word as the target confidence level;

[0194] Step D22: If the target confidence is greater than a predetermined threshold, the presentation state of the target image for the target object is determined to be: the presentation state described by the predetermined prompt word corresponding to the target confidence.

[0195] In this implementation, since the higher the confidence level corresponding to the predetermined prompt word is, the higher the probability that the presentation state described in the predetermined prompt word is the presentation state of the target object corresponding to the target image is, after determining the confidence level corresponding to each predetermined prompt word, the highest confidence level can be selected as the target confidence level.

[0196] However, it is possible that although the determined confidence is the highest confidence, the probability that the presentation state described in the predetermined prompt word represented by the confidence is the presentation state of the target object corresponding to the target object is still low; for example, generally speaking, the two can only be considered to be matched when the confidence is above 50%. At this time, if the highest determined confidence is 35%, although this confidence is the highest confidence among the confidences corresponding to the predetermined prompt words, when the confidence is 50% for the two to be considered to be a match, the presentation state described by the predetermined prompt word corresponding to the confidence does not actually match the presentation state of the target object to the target object.

[0197] Therefore, in order to improve the accuracy of the recognition result corresponding to the determined target image, a predetermined threshold can be set in advance for the confidence level, that is, when the determined confidence level is greater than the predetermined threshold, it is characterized that the presentation state described by the predetermined prompt word is the presentation state of the target object corresponding to the target image. Therefore, by comparing the relationship between the determined target confidence level and the predetermined threshold, it can be determined whether the presentation state described by the predetermined prompt word is the presentation state of the target object corresponding to the target image.

[0198] If the target confidence is greater than the above-mentioned predetermined threshold, then the presentation state described by the predetermined prompt word corresponding to the target confidence is the presentation state of the target object corresponding to the target image.

[0199] If the target confidence is not greater than the above-mentioned predetermined threshold, the presentation state described by the predetermined prompt word corresponding to the target confidence is not the presentation state of the target object corresponding to the target image.

[0200] In this implementation, the analysis accuracy of the presentation state of the target object in the determined target image is further improved by presetting a predetermined threshold value regarding the confidence level.

[0201] Optionally, in one implementation, the above step D1, based on the output result corresponding to each predetermined prompt word, determines the confidence level corresponding to the predetermined prompt word, may include the following steps:

[0202] Step D11: normalize the output result corresponding to each predetermined prompt word to obtain the confidence level corresponding to the predetermined prompt word.

[0203] In this implementation, after the output result corresponding to each predetermined prompt word is obtained, the output result corresponding to each predetermined prompt word may be normalized to obtain the confidence level corresponding to the predetermined prompt word.

[0204] Optionally, based on the output result corresponding to each predetermined prompt word, a confidence level corresponding to the predetermined prompt word is calculated using a predetermined formula; wherein the predetermined formula includes:

[0205]

[0206] σ(z i ) represents the confidence corresponding to the i-th predetermined prompt word about the target object, K represents the total number of predetermined prompt words about the target object, z i Characterize the feature score corresponding to the i-th predetermined prompt word, z j Characterize the feature score corresponding to the j-th predetermined prompt word.

[0207] Among them, the feature score corresponding to each of the above-mentioned predetermined prompt words is used to characterize the similarity between the predetermined prompt word and the target image, that is, the similarity between the vector feature of each predetermined prompt word in the embodiment of the present application and the vector feature of the target image, as well as the output result corresponding to each predetermined prompt word in the embodiment of the present application.

[0208] This implementation method, by normalizing the similarity between the vector features of each predetermined prompt word and the vector features of the target image to a specific interval, for example, [0, 1] or [-1, 1], etc., makes the data distribution statistically significant and the probability distribution of similarity more statistically intuitive, thereby improving the comparability of the output results corresponding to each predetermined prompt word.

[0209] Considering that for an image, there may be multiple target objects to be identified in the image area it contains, in order to improve the accuracy of overall image recognition, it is necessary to identify the image area corresponding to each target object in the image. Based on this, optionally, in one embodiment, a region detection algorithm for the carrier area is used to perform region detection on the initial image to obtain a region of interest, and the obtained region of interest is intercepted to obtain a target image; wherein the region of interest is the carrier area where the target object to be identified is located in the initial image when the target object to be identified exists in the initial image.

[0210] In this embodiment, considering that the target object to be identified exists in the initial image, the carrier region where the target object is located in the initial image will not change much. Therefore, after acquiring the initial image, the region detection algorithm for the carrier region can be used to perform region detection on the initial image to obtain the region of interest, and the region of interest in the initial image can be intercepted to obtain the target image. In other words, the image region contained in the obtained target image is the region of interest.

[0211] Optionally, in one embodiment, the target object is a tattoo, and the carrier region is a human body region, that is, when the target object is a tattoo, the carrier region where the tattoo exists is a human body region, and therefore, the captured target image is an image containing a human body.

[0212] Considering that the types of objects to be identified are different, the corresponding carrier areas are different, and accordingly, the region detection algorithms for the carrier areas are also different. For example, if the object to be identified is a tattoo, the corresponding carrier area is the human body area, and accordingly, the region detection algorithm for the carrier area can be a human body detection algorithm; if the object to be identified is a trademark, the corresponding carrier area is the edge area of ​​the target image, and accordingly, the region detection algorithm for the carrier area can be a trademark detection algorithm. Therefore, optionally, based on the type of object to be identified, the corresponding region detection algorithm is selected, and the target image is subjected to region detection to obtain the region of interest.

[0213] Among them, the above-mentioned human body detection algorithm can be an anchor-base (based on anchor frame) human body detection algorithm, and the embodiment of the present application does not make any specific limitation on this.

[0214] It should be noted that in the technical solution of this application, the acquisition, storage, use, processing, transmission, provision and disclosure of user personal information, such as images, are all carried out with the authorization of the user. Moreover, the images of human bodies in this embodiment are not human images of a specific user and cannot reflect the personal information of a specific user. In addition, the two-dimensional human body images in this embodiment come from a public data set.

[0215] In this embodiment, by intercepting the region of interest in the initial image, the interference of other image elements in the image on the recognition of the target image can be reduced, thereby improving the accuracy of automatic recognition of the image.

[0216] Considering that for an initial image, there may be one or more regions of interest contained therein. When there is one region of interest, the number of intercepted target images is one; correspondingly, when there are multiple regions of interest, the number of intercepted target images is multiple. The same method can be used to process each target image to obtain the recognition result corresponding to each target image.

[0217] That is to say, after determining the similarity of the target image to each predetermined prompt word, the similarity between the presentation state of the target object corresponding to the target image and the presentation state described by the predetermined prompt word can be determined. Among them, the higher the similarity, the higher the possibility that the presentation state of the target object in the target image is the presentation state described by the predetermined prompt word. Therefore, after determining the similarity of the target image to each predetermined prompt word, the presentation state of the target object described by the predetermined prompt word with high similarity is determined as the presentation state of the target object corresponding to the target image, thereby determining the recognition result corresponding to the target sub-image. Then, based on the above recognition result, the presentation state of at least one target image for the target object is determined, thereby determining the presentation state of the entire initial image for the target object to obtain the recognition result corresponding to the target object.

[0218] Optionally, in one implementation, the above step of determining the presentation state of the initial image for the target object based on the recognition result corresponding to the target image may include the following steps:

[0219] Step G1: In response to the number of target images being one, the presentation state represented by the recognition result corresponding to the target image is used as the presentation state of the initial image for the target object.

[0220] In this implementation, if the number of captured target images is one, it means that there is only one area in the initial image where the target object exists. At this time, the presentation state represented by the recognition result corresponding to the target image can be directly used as the presentation state of the overall image for the target object to realize automatic recognition of the image.

[0221] In this implementation, when the number of target images is one, the recognition result corresponding to the obtained target image can be directly used as the recognition result corresponding to the entire image, thereby improving the efficiency of automatic image recognition.

[0222] Optionally, in one implementation, the above step of determining the presentation state of the initial image for the target object based on the recognition result corresponding to the target image may include the following steps:

[0223] Step G2: in response to the number of target images being multiple, determining the presentation state of the initial image for the target object based on the recognition results corresponding to each target image in a predetermined analysis manner;

[0224] Among them, the predetermined analysis methods include:

[0225] If the different presentation states of the target object include a first presentation state indicating the existence of the object and a second presentation state indicating the non-existence of the object, then when there is a target image whose recognition result indicates the first presentation state among the target images, the presentation state of the initial image for the target object is determined to be the first presentation state; otherwise, the presentation state of the initial image for the target object is determined to be the second presentation state;

[0226] If the different presentation states of the target object include different presentation states regarding specified attributes, then determine the target recognition result with the highest frequency among the recognition results corresponding to each target image, and determine the presentation state of the initial image for the target object as: the presentation state represented by the target recognition result.

[0227] In this implementation, if the number of captured target images is multiple, it indicates that the initial image contains multiple areas with target objects; at this time, the recognition result corresponding to each captured target image is determined, and thus, based on the recognition results of each target image, the presentation state of the overall initial image for the target object is determined.

[0228] Considering that different presentation states of the target object may include presentation states for whether the target object exists, for example, the presence of a tattoo and the absence of a tattoo; it may also include presentation states for specified attributes of the target object, for example, the presence of a black cat and the presence of a white cat.

[0229] Therefore, a predetermined analysis method is set in advance for different presentation states of the target object, so that the presentation state of the initial image for the target object is determined based on the recognition results corresponding to each target image according to the predetermined analysis method. Specifically:

[0230] If the different presentation states of the target object include a first presentation state representing the existence of the object, and a second presentation state representing the non-existence of the object; when determining that in each target sub-image, there is a target image whose recognition result represents the first presentation state, the presentation state of the initial image for the target object can be determined as the first presentation state; otherwise, the presentation state of the initial image for the target object can be determined as the second presentation state.

[0231] If the different presentation states of the target object include different presentation states regarding specified attributes, it is necessary to determine the target recognition result with the highest frequency in the recognition results corresponding to each target image, thereby determining the presentation state of the initial image for the target object as the presentation state represented by the target recognition result.

[0232] For example, different presentation states of the target object include different presentation states regarding tattoo colors. The target recognition result with the highest frequency indicates that the tattoo color in the image is dark. At this time, the presentation state of the initial image for the target object is determined as the tattoo color in the image is dark, and the initial image is determined to be an image including a dark tattoo.

[0233] In the present implementation, considering that when there are multiple target images, the determination of the presentation state of the target object of the overall initial image needs to be combined with the recognition results of each target image; and considering that the presentation state of the target object includes multiple presentation methods, therefore, when there are multiple target images, the presentation state of the target object of the overall initial image can be determined based on the presentation state of the target object using a corresponding analysis method, thereby improving the accuracy of automatic image recognition.

[0234] Next, in conjunction with a specific embodiment, the step of performing region detection on the initial image using the region detection algorithm for the carrier region to obtain the region of interest is specifically introduced. In this specific embodiment, the target object is a tattoo, and the region of interest is a human body region.

[0235] For an initial image to be identified, determine the size information of the initial image. If the size information is not equal to the first size adapted by the convolutional neural network, scale the initial image to the first size to obtain a first image, and use the convolutional neural network to perform feature extraction on the obtained first image, thereby obtaining a first feature image corresponding to the above-mentioned first image.

[0236] In this way, for each pixel point in the obtained first feature image, multiple first anchor frames about the pixel point can be generated, so that for each first anchor frame, the confidence score of the existence of a human body in the first anchor frame can be determined.

[0237] Among them, the so-called anchor frame can be understood as: when identifying the target existing in the feature image obtained after processing by the convolutional neural network, if it is determined that there is a target belonging to the detected human body in a certain area in the feature image, the area can be identified using a graphic frame, and then, the graphic frame used to identify the area where the human body target exists in the feature image is the anchor frame.

[0238] Since there may be no human body in some areas of the feature image, or only part of the human body exists in the area framed by the first anchor frame, in order to ensure the accuracy of the determined image position information of the human body, a first preset score can be set in advance, so that the number of anchor frames that need to be processed later can be greatly reduced. Therefore, for each pixel point in the first feature image, after determining the confidence score corresponding to each first anchor frame about the pixel point, each anchor frame with a confidence score greater than the first preset score is retained as each first target anchor frame.

[0239] In this way, based on the position information between each first target anchor frame, the IOU (Intersection over Union) value between every two first target anchor frames in all the first target anchor frames can be calculated, where the so-called intersection over union ratio refers to the ratio of the area of ​​the overlapping area of ​​two first target anchor frames divided by the area of ​​the union area of ​​the two first target anchor frames.

[0240] For example, the area of ​​the first target anchor box J1 is S J1 , the area of ​​the first target anchor box J2 is S J2 , the overlapping area of ​​the first target anchor box J1 and the first target anchor box J2 is S J1J2 , then the intersection-and-union ratio between the first target anchor box J1 and the first target anchor box J2 can be calculated as

[0241] Thus, in each group of first target anchor frames whose calculated IOU values ​​are greater than the first preset IOU value, the position information corresponding to the first target anchor frame with the highest confidence score is selected and determined as the region of interest in the initial image.

[0242] For example, for the initial image, if the calculated IOU between any two first target anchor frames is greater than the first preset IOU value, it can be determined that the human bodies framed by the two first target anchor frames belong to the same human body. In this way, the position information corresponding to the first target anchor frame with the highest confidence score among the two first target anchor frames can be determined as the position information of the region of interest in the initial image.

[0243] Optionally, since there is an offset between the position of the preset first anchor frame and the actual position of the human body, a first relative offset can be preset, so that the first offset of the first anchor frame is determined while the confidence score of each first anchor frame is determined. In this way, each first target anchor frame whose first offset is not greater than the first relative offset can be retained, so that the first target anchor frame obtained can be filtered twice using the first relative offset. Then, through two filterings, each first anchor frame whose confidence score is greater than the first preset score and whose first offset is not greater than the first relative offset is determined as each first target anchor frame, further improving the accuracy of the image position information of the determined region of interest in the target image.

[0244] Optionally, in an embodiment, an image recognition method provided in an embodiment of the present application may further include the following steps:

[0245] Step L1: Determine whether the presentation state of the target image for the target object meets the predetermined requirements; if not, execute step L2; otherwise, execute step L3;

[0246] Step L2: performing a predetermined processing operation on the target image;

[0247] The predetermined processing operation includes: using a predetermined image processing method to cover the target object in the target image, or terminating the uploading of the target image;

[0248] Step L3: Upload the target image.

[0249] In this embodiment, in order to maintain the network environment of the network platform, it is necessary to set a predetermined rule for the presentation state of the target object in advance, that is, an image that meets the predetermined rule is a normal image, and an image that does not meet the predetermined rule is an illegal image. For example, a human image without tattoos is a normal image, and a human image with tattoos is an illegal image.

[0250] Therefore, after determining the presentation status of the target image for the target object, it is possible to further determine whether the presentation status complies with the above-mentioned predetermined regulations, thereby determining whether the target image can be uploaded.

[0251] If the presentation state of the target image for the target object meets the predetermined requirements, the target image is characterized as a normal image, and the dissemination of the target image will not cause bad guidance, and the target image can be uploaded.

[0252] If the presentation state of the target image to the target object does not meet the predetermined regulations, the target image is characterized as an illegal image, and the spread of the target image may cause bad guidance. Therefore, in order to maintain the network environment of the network platform, the target image can be subjected to a predetermined processing operation.

[0253] Among them, in order to improve the image upload rate of the platform, the above-mentioned predetermined processing operation may include using a predetermined image processing method to cover the target object in the target image. In this way, when it is determined that the presentation state of the target image for the target object does not meet the predetermined regulations, the target object in the target image can be covered by using a predetermined image processing method, so that after covering the target object in the target image, an image recognition method provided by an embodiment of the present application is performed on the covered target image until it is determined that the presentation state of the target image for the target object meets the predetermined regulations, the target image is a normal image, and the target image is uploaded.

[0254] In addition, considering that the number of images to be identified is too large, the predetermined processing operation may include terminating the uploading of the target image. In this way, when it is determined that the presentation state of the target image for the target object does not meet the predetermined regulations, the uploading of the target image is directly terminated.

[0255] Optionally, after terminating the upload of the target image, a reminder message indicating that the review has not been passed is sent to the corresponding client.

[0256] In this embodiment, target images uploaded to the network platform are classified according to predetermined regulations to ensure that the images uploaded to the network platform will not cause bad guidance and protect the network environment of the network platform.

[0257] Corresponding to the model training method provided in the above embodiment of the present application, the embodiment of the present application also provides a model training device, such as Figure 6 As shown, the device comprises:

[0258] An image acquisition module 610 is used to acquire a sample image; wherein the sample image is provided with a label for characterizing a presentation state of the sample object;

[0259] The model training module 620 is used to input the sample image into the image recognition model to be trained to obtain an output result corresponding to each sample prompt word; wherein the image recognition model includes a fully connected layer with a predetermined initial weight parameter and an image encoder in a pre-trained contrastive learning model; each sample prompt word is an image description text with the presentation state of the sample object as the description content, and the initial weight parameter is predetermined based on the vector features of each sample prompt word; the output result corresponding to each sample prompt word is used to characterize the similarity between the vector features of the sample prompt word and the vector features of the sample image;

[0260] A loss value determination module 630, configured to determine a loss value corresponding to the image recognition model based on an output result corresponding to each sample prompt word and a label of the sample image;

[0261] The parameter adjustment module 640 is used to adjust the weight parameters of the fully connected layer and the image encoder based on the loss value when it is determined that the image recognition model does not meet the model convergence condition based on the loss value, and return to the step of obtaining the sample image.

[0262] It can be seen from the above that, by applying the solution provided in the embodiment of the present application, before model training, an image recognition model to be trained can be pre-constructed, and the image recognition model includes a fully connected layer with predetermined initial weight parameters and an image encoder in a pre-trained contrastive learning model. The image recognition model can be used to determine the similarity between the vector features of the prompt word and the vector features of the input image. Therefore, before model training, each sample prompt word can be obtained, wherein each sample prompt word is an image description text with the presentation state of the sample object as the description content. Then, based on the vector features of each sample prompt word, the initial weight parameters of the fully connected layer are predetermined to provide the image recognition process with vector features about the prompt word through the fully connected layer with the initial weight parameters.

[0263] In this way, when training the model, it is only necessary to input the acquired sample image into the image recognition model to be trained to obtain the output result corresponding to each sample prompt word, and the output result is used to characterize the similarity between the vector feature of the sample prompt word and the vector feature of the sample image. Then, based on the output result corresponding to each sample prompt word and the pre-set label of the sample image itself, the loss value of the current weight parameter of the fully connected layer of the image recognition model and the loss value of the image encoder when acquiring the vector feature of the sample image are determined, and then the current weight parameter of the fully connected layer of the image recognition model and the image encoder are adjusted based on the determined loss value. In this way, by reconstructing the model structure of the image recognition model, the training effect of training two encoders can be achieved by training the number of training samples and training time required for one model, that is, by reducing the number of encoders required to be trained during model training, the number of training samples and training time required during training are reduced, thereby improving the training efficiency of the image recognition model.

[0264] Optionally, in a specific implementation, a method for determining the initial weight parameters of the fully connected layer includes:

[0265] The vector feature of each sample prompt word is obtained, and the vector feature of each sample prompt word is concatenated as a row of matrix elements to obtain the initial weight parameter of the fully connected layer.

[0266] Optionally, in a specific implementation, the loss value determination module 630 includes:

[0267] A confidence determination submodule, used to determine the target confidence corresponding to the target sample prompt word based on the output result corresponding to each sample prompt word; wherein the presentation state described by the target sample prompt word is the same as the presentation state of the sample object represented by the label of the sample image; the target confidence is used to represent the presentation state described in the target sample prompt word, which is the predicted probability of the presentation state of the sample object in the sample image;

[0268] The loss value determination submodule is used to determine the loss value corresponding to the image recognition model based on the target confidence and the label of the sample image.

[0269] Optionally, in a specific implementation, the confidence determination submodule is specifically used to:

[0270] The target confidence corresponding to the target sample prompt word is determined based on the ratio of the output result corresponding to the target sample prompt word to the target sum value; wherein the target sum value is obtained by summing the output results corresponding to each sample prompt word.

[0271] Optionally, in one implementation, the model training module 620 is specifically used to:

[0272] The sample image is input into the image encoder in the image recognition model to be trained so that the image encoder generates vector features of the sample image, and the vector features of the sample image are transmitted to the fully connected layer so as to process the vector features of the sample image based on the current weight parameters of the fully connected layer itself to obtain the output result corresponding to each sample prompt word.

[0273] Corresponding to the image recognition method provided in the above embodiment of the present application, the embodiment of the present application also provides an image recognition device, such as Figure 7 As shown, the device comprises:

[0274] The image acquisition module 710 is used to acquire a target image; wherein the target image is an image used to identify a target object;

[0275] The similarity determination module 720 is used to input the target image into a pre-trained image recognition model to obtain an output result corresponding to each predetermined prompt word; wherein the image recognition model is trained based on a model training method provided in an embodiment of the present application; the predetermined prompt word is a sample prompt word used to train the image recognition model; the output result corresponding to each predetermined prompt word is used to characterize the similarity between the vector feature of the predetermined prompt word and the vector feature of the target image;

[0276] The presentation state determination module 730 is used to determine the presentation state of the target image for the target object based on the output result corresponding to each predetermined prompt word.

[0277] From the above, it can be seen that in the image recognition method provided in the embodiment of the present application, since the image recognition model used is trained based on a model training method provided in the embodiment of the present application, that is to say, the vector features of the predetermined prompt words used for image recognition in the image recognition model used are fixed, when using the image recognition model for image recognition, there is no need to input the prompt words, only the target image needs to be input to realize automatic recognition of the image.

[0278] Optionally, in a specific implementation, the presentation status determination module 730 includes:

[0279] A first determination submodule is used to determine the confidence level corresponding to each predetermined prompt word based on the output result corresponding to the predetermined prompt word; wherein the confidence level corresponding to the predetermined prompt word is used to characterize the probability that the presentation state described in the predetermined prompt word is the presentation state corresponding to the target image with respect to the target object;

[0280] The second determination submodule is used to analyze the presentation status of the target image with respect to the target object based on the confidence level corresponding to each predetermined prompt word.

[0281] Optionally, in a specific implementation, the second determining submodule is specifically configured to:

[0282] From the confidence levels corresponding to each predetermined prompt word, select the highest confidence level as the target confidence level;

[0283] If the target confidence is greater than a predetermined threshold, the presentation state of the target image for the target object is determined to be: the presentation state described by the predetermined prompt word corresponding to the target confidence.

[0284] The present application also provides an electronic device, such as Figure 8 As shown, it includes a processor 801, a communication interface 802, a memory 803 and a communication bus 804, wherein the processor 801, the communication interface 802, and the memory 803 communicate with each other through the communication bus 804.

[0285] Memory 803, used for storing computer programs;

[0286] The processor 801 is used to implement the model training method and / or the image recognition method described in any of the above embodiments when executing the program stored in the memory 803.

[0287] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0288] The communication interface is used for communication between the above terminal and other devices.

[0289] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0290] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0291] In another embodiment provided in the present application, a computer-readable storage medium is also provided, in which a computer program is stored. When the computer program is executed by a processor, the model training method and / or the image recognition method described in any one of the above embodiments are implemented.

[0292] In another embodiment provided in the present application, a computer program product comprising instructions is also provided, which, when executed on a computer, enables the computer to execute the model training method and / or the image recognition method described in any one of the above embodiments.

[0293] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive Solid State Disk (SSD)), etc.

[0294] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0295] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, electronic device embodiment, computer-readable storage medium embodiment, and computer program product embodiment, since they are basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0296] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A model training method, characterized in that: The method comprises: Acquire a sample image; wherein the sample image is provided with a label for characterizing a presentation state of the sample object; The sample image is input into the image recognition model to be trained to obtain an output result corresponding to each sample prompt word; wherein the image recognition model includes a fully connected layer with a predetermined initial weight parameter and an image encoder in a pre-trained contrastive learning model; each sample prompt word is an image description text with the presentation state of the sample object as the description content, and the initial weight parameter is predetermined based on the vector features of each sample prompt word; the output result corresponding to each sample prompt word is used to characterize the similarity between the vector features of the sample prompt word and the vector features of the sample image; Determine a loss value corresponding to the image recognition model based on the output result corresponding to each sample prompt word and the label of the sample image; When it is determined based on the loss value that the image recognition model does not meet the model convergence condition, the current weight parameters of the fully connected layer and the image encoder are adjusted based on the loss value, and the process returns to the step of obtaining the sample image.

2. The method according to claim 1, characterized in that The method for determining the initial weight parameters of the fully connected layer includes: The vector feature of each sample prompt word is obtained, and the vector feature of each sample prompt word is concatenated as a row of matrix elements to obtain the initial weight parameter of the fully connected layer.

3. The method according to claim 1, characterized in that The determining, based on the output result corresponding to each sample prompt word and the label of the sample image, a loss value corresponding to the image recognition model includes: Based on the output results corresponding to each sample prompt word, determine the target confidence corresponding to the target sample prompt word; wherein the presentation state described by the target sample prompt word is the same as the presentation state of the sample object represented by the label of the sample image; the target confidence is used to represent the presentation state described in the target sample prompt word, and the predicted probability of the presentation state of the sample object in the sample image; Based on the target confidence and the label of the sample image, a loss value corresponding to the image recognition model is determined.

4. The method according to claim 3, characterized in that The step of determining the target confidence level corresponding to the target sample prompt word based on the output result corresponding to each sample prompt word includes: The target confidence corresponding to the target sample prompt word is determined based on the ratio of the output result corresponding to the target sample prompt word to the target sum value; wherein the target sum value is obtained by summing the output results corresponding to each sample prompt word.

5. The method according to claim 1, characterized in that The step of inputting the sample image into the image recognition model to be trained to obtain an output result corresponding to each sample prompt word includes: The sample image is input into the image encoder in the image recognition model to be trained so that the image encoder generates vector features of the sample image, and the vector features of the sample image are transmitted to the fully connected layer so as to process the vector features of the sample image based on the current weight parameters of the fully connected layer itself to obtain the output result corresponding to each sample prompt word.

6. An image recognition method, characterized in that: The method comprises: Acquire a target image; wherein the target image is an image used to identify a target object; Input the target image into a pre-trained image recognition model to obtain an output result corresponding to each predetermined prompt word; wherein the image recognition model is trained based on the method described in any one of claims 1 to 5; the predetermined prompt words are sample prompt words used to train the image recognition model; the output result corresponding to each predetermined prompt word is used to characterize the similarity between the vector feature of the predetermined prompt word and the vector feature of the target image; Based on the output result corresponding to each predetermined prompt word, a presentation state of the target image for the target object is determined.

7. The method according to claim 6, characterized in that The determining, based on the output result corresponding to each predetermined prompt word, a presentation state of the target image for the target object, comprises: Based on the output result corresponding to each predetermined prompt word, determine the confidence level corresponding to the predetermined prompt word; wherein the confidence level corresponding to the predetermined prompt word is used to characterize the presentation state described in the predetermined prompt word, which is the probability of the presentation state corresponding to the target image with respect to the target object; Based on the confidence level corresponding to each predetermined prompt word, the presentation status of the target image with respect to the target object is analyzed.

8. The method according to claim 7, characterized in that The analyzing the presentation status of the target image to the target object based on the confidence level corresponding to each predetermined prompt word includes: From the confidence levels corresponding to each predetermined prompt word, select the highest confidence level as the target confidence level; If the target confidence is greater than a predetermined threshold, the presentation state of the target image for the target object is determined to be: the presentation state described by the predetermined prompt word corresponding to the target confidence.

9. A model training device, characterized in that: The device comprises: An image acquisition module, used to acquire a sample image; wherein the sample image is provided with a label for characterizing a presentation state of the sample object; A model training module, used for inputting the sample image into the image recognition model to be trained, and obtaining an output result corresponding to each sample prompt word; wherein the image recognition model includes a fully connected layer with a predetermined initial weight parameter and an image encoder in a pre-trained contrastive learning model; each sample prompt word is an image description text with the presentation state of the sample object as the description content, and the initial weight parameter is predetermined based on the vector features of each sample prompt word; the output result corresponding to each sample prompt word is used to characterize the similarity between the vector features of the sample prompt word and the vector features of the sample image; A loss value determination module, used to determine the loss value corresponding to the image recognition model based on the output result corresponding to each sample prompt word and the label of the sample image; A parameter adjustment module is used to adjust the weight parameters of the fully connected layer and the image encoder based on the loss value when it is judged that the image recognition model does not meet the model convergence condition based on the loss value, and return to the step of obtaining the sample image.

10. An image recognition device, characterized in that: The device comprises: An image acquisition module, used to acquire a target image; wherein the target image is an image used to identify a target object; A similarity determination module, used for inputting the target image into a pre-trained image recognition model to obtain an output result corresponding to each predetermined prompt word; wherein the image recognition model is trained based on the method described in any one of claims 1 to 5; the predetermined prompt words are sample prompt words used to train the image recognition model; the output result corresponding to each predetermined prompt word is used to characterize the similarity between the vector feature of the predetermined prompt word and the vector feature of the target image; The presentation state determination module is used to determine the presentation state of the target image for the target object based on the output result corresponding to each predetermined prompt word.

11. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing any of the methods described in claims 1-5 and / or any of the methods described in claims 6-8 when executing a program stored in a memory.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of claims 1 to 5 and / or the method described in any one of claims 6 to 8 is implemented.

Images