Model training method and device, image recognition method and device and electronic equipment

Abstract

The embodiment of the invention provides a model training method and device, an image recognition method and device and electronic equipment, and relates to the technical field of machine learning. The model training method comprises the following steps: acquiring a sample image from a sample image set; inputting the sample image into a to-be-trained image recognition model to obtain an output result corresponding to each sample cue word in the plurality of sample cue words; determining a loss value corresponding to the image recognition model based on the output result corresponding to each sample prompt word and the image label of the sample image; and when it is judged that the image recognition model does not meet the model convergence condition based on the loss value, performing parameter adjustment on the current weight parameter of the target full-connection layer and the image encoder, and returning to the step of obtaining the sample image from the sample image set. According to the scheme, one image recognition model can be used for carrying out image recognition on various different objects, so that computing resources occupied by the model during training and reasoning are reduced.

Description

Technical Field

[0001] The present application relates to the field of machine learning technology, and in particular to a model training method, image recognition method, device and electronic equipment. Background Art

[0002] In order to maintain the network environment of the network platform, images or videos uploaded to the platform need to be reviewed to avoid spreading images or videos that may cause bad guidance through the platform, for example, spreading images of objects such as tattoos and smoke. Usually, an image recognition model for image recognition is used to recognize an image, that is, to recognize the presentation state of an image for an object, such as whether an object exists in the image.

[0003] In the related art, when it is necessary to identify the presentation state of a specified object, the pre-trained image recognition model is usually trained in a targeted manner using sample prompt words and sample images corresponding to the specified object. This targeted training enables the trained image recognition model to identify the presentation state of the specified object in the input image based on the output results of the image recognition model without the need to input prompt words in the subsequent application process. The sample prompt words corresponding to the specified object are image description texts with the presentation state of the specified object as the description content, and the output results of the image recognition model are the similarity between the features of each sample prompt word used in the model training and the features of the input image.

[0004] However, the image recognition model obtained through the above targeted training can only be trained and reasoned for one task, that is, it can only recognize the presentation state of a specified object. If multiple objects need to be recognized, it is necessary to train each object to obtain a highly targeted image recognition model that can recognize the presentation state of the object, which results in a large amount of computing resources being required during model training and model reasoning. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a model training method, an image recognition method, a device and an electronic device, so as to realize image recognition of multiple different objects using a trained image recognition model, thereby reducing the computing resources occupied by the image recognition model during model training and model reasoning. The specific technical solution is as follows:

[0006] In a first aspect provided by an embodiment of the present application, first, a model training method is provided, the method comprising:

[0007] Acquire a sample image from a sample image set; wherein the sample image set includes at least one sample image corresponding to each of a plurality of designated objects, and each sample image is provided with an image label for characterizing a presentation state of the corresponding designated object;

[0008] The sample image is input into the image recognition model to be trained to obtain an output result corresponding to each sample prompt word in a plurality of sample prompt words; wherein the image recognition model includes a target fully connected layer with a predetermined initial weight parameter and an image encoder in a pre-trained contrastive learning model; the plurality of sample prompt words include at least two sample prompt words corresponding to each designated object, and the at least two sample prompt words corresponding to each designated object include sample prompt words whose presentation states are mutually exclusive for the designated object; the output result corresponding to each sample prompt word is used to characterize the similarity between the vector feature of the sample prompt word and the vector feature of the sample image; the initial weight parameter is predetermined based on the vector features of the plurality of sample prompt words;

[0009] Determine a loss value corresponding to the image recognition model based on the output result corresponding to each sample prompt word and the image label of the sample image;

[0010] When it is determined based on the loss value that the image recognition model does not meet the model convergence condition, the current weight parameters of the target fully connected layer and the image encoder are adjusted, and the process returns to the step of obtaining the sample image from the sample image set.

[0011] In a second aspect provided by an embodiment of the present application, an image recognition method is also provided, the method comprising:

[0012] Acquire a target image; wherein the target image is an image used to identify a target object;

[0013] The target image is input into the trained image recognition model to obtain an output result corresponding to each sample prompt word in each sample prompt word; wherein the image recognition model is trained based on the model training method provided in the first aspect; each sample prompt word is a prompt word used to train the image recognition model; the target object belongs to one of multiple specified objects; the output result corresponding to each sample prompt word is used to characterize the similarity between the vector feature of the sample prompt word and the vector feature of the target image;

[0014] Based on the output result corresponding to each sample prompt word, a presentation state of the target image for the target object is determined.

[0015] In a third aspect provided by an embodiment of the present application, a model training device is also provided, the device comprising:

[0016] An acquisition module, used to acquire a sample image from a sample image set; wherein the sample image set includes at least one sample image corresponding to each of a plurality of designated objects, and each sample image is provided with an image label for characterizing a presentation state of the corresponding designated object;

[0017] A similarity determination module is used to input the sample image into the image recognition model to be trained, and obtain an output result corresponding to each sample prompt word in a plurality of sample prompt words; wherein the image recognition model includes a target fully connected layer with a predetermined initial weight parameter and an image encoder in a pre-trained contrastive learning model; the plurality of sample prompt words include at least two sample prompt words corresponding to each designated object, and the at least two sample prompt words corresponding to each designated object include sample prompt words whose presentation states are mutually exclusive for the designated object; the output result corresponding to each sample prompt word is used to characterize the similarity between the vector feature of the sample prompt word and the vector feature of the sample image; the initial weight parameter is predetermined based on the vector features of the plurality of sample prompt words;

[0018] A loss value determination module, used to determine the loss value corresponding to the image recognition model based on the output result corresponding to each sample prompt word and the image label of the sample image;

[0019] A parameter adjustment module is used to adjust the current weight parameters of the target fully connected layer and the image encoder when it is judged that the image recognition model does not meet the model convergence conditions based on the loss value, and return to the step of obtaining the sample image from the sample image set.

[0020] In a fourth aspect provided by an embodiment of the present application, an image recognition device is further provided, the device comprising:

[0021] An acquisition module, used for acquiring a target image; wherein the target image is an image used for identifying a target object;

[0022] A similarity determination module is used to input the target image into a trained image recognition model to obtain an output result corresponding to each sample prompt word in each sample prompt word; wherein the image recognition model is trained based on the model training method provided in the first aspect; the sample prompt words are prompt words used to train the image recognition model; the target object belongs to one of multiple specified objects; the output result corresponding to each sample prompt word is used to characterize the similarity between the vector feature of the sample prompt word and the vector feature of the target image;

[0023] The recognition module is used to determine the presentation status of the target image to the target object based on the output result corresponding to each sample prompt word.

[0024] In the fifth aspect provided by the embodiment of the present application, there is also provided an electronic device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; the memory is used to store computer programs; the processor is used to implement any model training method provided in the first aspect when executing the program stored in the memory, and / or implement any image recognition method provided in the second aspect.

[0025] In another aspect provided by the embodiments of the present application, a computer-readable storage medium is also provided, in which a computer program is stored. When the computer program is executed by a processor, it implements any model training method provided in the first aspect above, and / or implements any image recognition method provided in the second aspect above.

[0026] In another aspect of the embodiments of the present application, a computer program product comprising instructions is also provided, which, when executed on a computer, enables the computer to execute any model training method provided in the first aspect above, and / or any image recognition method provided in the second aspect above.

[0027] As can be seen from the above, in the model training method provided by the embodiment of the present application, the image recognition model to be trained includes a target fully connected layer with a predetermined initial weight parameter, and an image encoder in a pre-trained contrastive learning model. Among them, the initial weight parameter of the target fully connected layer is determined in advance based on the vector features of multiple sample prompt words; the above-mentioned multiple sample prompt words include at least two sample prompt words corresponding to each specified object, and the at least two sample prompt words corresponding to each specified object include sample prompt words whose presentation states are mutually exclusive. In other words, the target fully connected layer can provide corresponding vector features for multiple specified objects. Therefore, no matter what kind of specified object is contained in the input sample image, the target fully connected layer can obtain the similarity between the vector features of each sample prompt word and the vector features of the sample image as the output result corresponding to the sample prompt word. Then, based on the output result corresponding to each sample prompt word and the image label of the sample image, the current weight parameter of the target fully connected layer and the loss value of the image encoder are determined, and the current weight parameter of the target fully connected layer and the image encoder are adjusted. In this way, by reconstructing the model structure of the image recognition model, targeted training can be achieved for multiple specified objects in one model. There is no need to train a targeted model separately for each specified object, thereby reducing the amount of computing resources occupied by the model during model training.

[0028] In addition, in the image recognition method provided in the embodiment of the present application, the image recognition model used is obtained by training the training samples for multiple specified objects, and the trained image recognition model can perform image recognition on multiple different specified objects. Therefore, when the target object in the input target image is any one of the multiple specified objects, the image recognition model can recognize the target image, thereby achieving the purpose of using one image recognition model to perform image recognition on multiple different specified objects and reducing the amount of computing resources occupied by the model during model inference.

[0029] Furthermore, the above-mentioned image recognition model is trained based on a model training method provided in an embodiment of the present application, so the sample prompt words when the image recognition model performs image recognition are the prompt words obtained by training the image recognition model. Therefore, when the image recognition model is used for image recognition, there is no need to input prompt words, and only the target image can be input to realize automatic recognition of the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0031] Figure 1 A flowchart of a model training method provided in an embodiment of the present application;

[0032] Figure 2 A schematic diagram of a specific embodiment;

[0033] Figure 3 A flowchart of an image recognition method provided in an embodiment of the present application;

[0034] Figure 4 A schematic diagram of the structure of a model training device provided in an embodiment of the present application;

[0035] Figure 5 A schematic diagram of the structure of an image recognition device provided in an embodiment of the present application;

[0036] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0037] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0038] In order to better understand the present application, the professional terms involved in the embodiments of the present application are first introduced below.

[0039] The text-image pre-training model based on contrastive learning (also called contrastive learning model) is a model pre-trained on 400 million pairs of image-text matching pairs. It has strong generalization ability. By giving the contrastive learning model any prompt word, the similarity between the features of the prompt word and the features of the input image can be obtained as the output result of the contrastive learning model. Specifically, the contrastive learning model identifies the uploaded image, that is, identifies the presentation state of the image for the object, such as whether there is an object in the image (whether there is a cat in the image), or whether there is an object in a certain state in the image (whether there is a white cat in the image), etc. In the recognition process, the contrastive learning model can classify and identify images based on pre-set prompt words. Among them, the above-mentioned text-image pre-training model based on contrastive learning can be a CLIP (Contrastive Language-Image Pre-Training) model, a BLIP (Bootstrapping Language-Image Pre-training) model or a SigLip (Sigmoid Loss for Language-Image Pre-training) model, etc., which is not specifically limited in the embodiments of the present application.

[0040] Taking the above contrastive learning model as the CLIP model as an example, when the CLIP model is used to classify images, by giving the CLIP model two prompt words "people with tattoos" and "people without tattoos", the CLIP model is allowed to determine which prompt word has a closer similarity to the image, so as to achieve image classification. Specifically:

[0041] After receiving each prompt word, the Text Encoder of the CLIP model extracts the text features of each prompt word and calls its own text multimodal network to convert the extracted text features into text multimodal features. After receiving an image, the Image Encoder of the CLIP model extracts the image features of the image and calls its own image multimodal network to convert the extracted image features into image multimodal features.

[0042] Afterwards, the CLIP model can match the text multimodal features corresponding to each prompt word with the obtained image multimodal features for feature similarity, obtain the similarity between the text multimodal features and the image multimodal features of each prompt word, and use the prompt word with the highest similarity as the label of the image. In other words, the classification result for the image is: the presentation state of the image for the object is the presentation state described by the prompt word with the highest similarity.

[0043] Among them, the above-mentioned image multimodal features are the vector features of the image in the embodiment of the present application; the above-mentioned text multimodal features (text multimodal features) are the vector features of the prompt words in the embodiment of the present application.

[0044] In order to maintain the network environment of the network platform, it is necessary to review the images or videos uploaded to the platform to avoid the spread of images or videos that may cause bad guidance through the platform, for example, the spread of images of objects such as tattoos and smoke. However, with the huge increase in the number of images or videos that need to be reviewed, the cost of manually reviewing each image or each frame of video uploaded to the platform is huge. Therefore, usually, an image recognition model is used to identify the image and identify the presentation state of the image for the object, so as to automatically filter out images or videos without the above-mentioned objects.

[0045] However, when it is necessary to identify the presentation state of a specified object, the relevant technology uses the sample prompt words and sample images corresponding to the specified object to conduct targeted training on the pre-trained image recognition model. This targeted training enables the trained image recognition model to identify the presentation state of the specified object in the input image based on the output results of the image recognition model without the need to input prompt words in the subsequent application process. The sample prompt words corresponding to the specified object are image description texts with the presentation state of the specified object as the description content, and the output results of the image recognition model are the similarity between the features of the sample prompt words used in the model training and the features of the input image.

[0046] However, the image recognition model obtained through the above targeted training can only be trained and reasoned for one task, that is, it can only recognize the presentation state of one specified object. If multiple specified objects need to be recognized, it is necessary to train each specified object to obtain a highly targeted image recognition model that can recognize the presentation state of the specified object, which results in a large amount of computing resources being required during model training and model reasoning.

[0047] Based on this, how to use a trained image recognition model to perform image recognition on multiple different specified objects, thereby reducing the computing resources occupied by the image recognition model during model training and model inference, has become a technical problem that urgently needs to be solved.

[0048] In order to solve the above technical problems, an embodiment of the present application provides a model training method.

[0049] Among them, a model training method provided in an embodiment of the present application can be applied to an electronic device, and the electronic device is specifically used for training the model. In a specific application, the electronic device can be a smart phone, a tablet computer, etc., which is all reasonable.

[0050] A model training method provided in an embodiment of the present application may include the following steps:

[0051] Acquire a sample image from a sample image set; the sample image set includes at least one sample image corresponding to each of a plurality of designated objects, and each sample image is provided with an image label for characterizing a presentation state of the corresponding designated object;

[0052] The sample image is input into the image recognition model to be trained to obtain an output result corresponding to each sample prompt word in a plurality of sample prompt words; wherein the image recognition model includes a target fully connected layer with a predetermined initial weight parameter and an image encoder in a pre-trained contrastive learning model; the plurality of sample prompt words include at least two sample prompt words corresponding to each designated object, and the at least two sample prompt words corresponding to each designated object include sample prompt words whose presentation states are mutually exclusive for the designated object; the output result corresponding to each sample prompt word is used to characterize the similarity between the vector feature of the sample prompt word and the vector feature of the sample image; the initial weight parameter is predetermined based on the vector features of the plurality of sample prompt words;

[0053] Determine a loss value corresponding to the image recognition model based on the output result corresponding to each sample prompt word and the image label of the sample image;

[0054] When it is determined based on the loss value that the image recognition model does not meet the model convergence condition, the current weight parameters of the target fully connected layer and the image encoder are adjusted, and the process returns to the step of obtaining the sample image from the sample image set.

[0055] As can be seen from the above, in the model training method provided by the embodiment of the present application, the image recognition model to be trained includes a target fully connected layer with a predetermined initial weight parameter, and an image encoder in a pre-trained contrastive learning model. Among them, the initial weight parameter of the target fully connected layer is determined in advance based on the vector features of multiple sample prompt words; the above-mentioned multiple sample prompt words include at least two sample prompt words corresponding to each specified object, and the at least two sample prompt words corresponding to each specified object include sample prompt words whose presentation states are mutually exclusive. In other words, the target fully connected layer can provide corresponding vector features for multiple specified objects. Therefore, no matter what kind of specified object is contained in the input sample image, the target fully connected layer can obtain the similarity between the vector features of each sample prompt word and the vector features of the sample image as the output result corresponding to the sample prompt word. Then, based on the output result corresponding to each sample prompt word and the image label of the sample image, the current weight parameter of the target fully connected layer and the loss value of the image encoder are determined, and the current weight parameter of the target fully connected layer and the image encoder are adjusted. In this way, by reconstructing the model structure of the image recognition model, targeted training can be achieved for multiple specified objects in one model. There is no need to train a targeted model separately for each specified object, thereby reducing the amount of computing resources occupied by the model during model training.

[0056] The following is a detailed description of a model training method provided in an embodiment of the present application in conjunction with the accompanying drawings.

[0057] Figure 1 A flow chart of a model training method provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the method may include the following steps:

[0058] S101: Obtain a sample image from a sample image set;

[0059] The sample image set includes at least one sample image corresponding to each of the multiple specified objects, and each sample image is provided with an image label for characterizing the presentation state of the corresponding specified object.

[0060] In the present application, in order to train an image recognition model for image recognition of multiple specified objects, at least one sample image corresponding to each of the multiple specified objects is obtained in advance, and the obtained sample images are combined to obtain a sample image set for model training. In addition, in order to determine which specified object each obtained sample image corresponds to, each sample image is provided with an image label for characterizing the presentation state of the corresponding specified object.

[0061] It is understandable that the number of sample images acquired may be one or more, and this is not specifically limited in the embodiments of the present application.

[0062] Among them, the presentation state of the above-mentioned sample object can be understood as whether a specified object exists in the sample image, for example, whether there is smoke in the image, or whether there is a cat in the image, etc.; it can also be understood as whether there is a specified object in a certain state or belonging to a certain attribute in the sample image, for example, whether there is smoke in a burning state in the image, or whether there is a white cat in the image, etc., which are all reasonable.

[0063] It should be noted that the above-mentioned image labels can be digital labels or text labels, and the embodiments of the present application do not make specific limitations on this. For example, the first designated object is a cat. If there is a cat in the sample image, the image label of the sample image can be set to 11, wherein the first digit "1" represents that the designated object is a cat, and the second digit "1" represents that there is a cat in the sample image; if there is no cat in the sample image, the image label of the sample image can be set to 10, wherein the first digit "1" represents that the designated object is a cat, and the second digit "0" represents that there is no cat in the sample image. The second designated object is a dog. If there is a dog in the sample image, the image label of the sample image can be set to 21, wherein the first digit "2" represents that the designated object is a dog, and the second digit "1" represents that there is a dog in the sample image; if there is no dog in the sample image, the image label of the sample image can be set to 20, wherein the first digit "2" represents that the designated object is a dog, and the second digit "0" represents that there is no dog in the sample image.

[0064] S102: inputting the sample image into the image recognition model to be trained, and obtaining an output result corresponding to each sample prompt word in a plurality of sample prompt words;

[0065] Among them, the image recognition model includes a target fully connected layer with a predetermined initial weight parameter and an image encoder in a pre-trained contrastive learning model; the multiple sample prompt words include at least two sample prompt words corresponding to each specified object, and the at least two sample prompt words corresponding to each specified object include sample prompt words whose presentation states are mutually exclusive for the specified object; the output result corresponding to each sample prompt word is used to characterize the similarity between the vector feature of the sample prompt word and the vector feature of the sample image; the initial weight parameter is predetermined based on the vector features of the multiple sample prompt words.

[0066] In the present application, it is considered that in the prior art, a contrastive learning model is usually used as an image recognition model for image recognition. When performing image recognition, this type of model calls a text encoder and an image encoder to process the input prompt word and image respectively, that is, calls the text encoder to process the input prompt word to obtain the vector features of the prompt word, and calls the image encoder to process the input image to obtain the vector features of the image. In this way, this type of model can recognize the image by determining the similarity between the vector features of the prompt word and the vector features of the image.

[0067] That is to say, the key point of the image recognition model to recognize the image is to obtain the text multimodal features of the prompt word (also called the vector features of the prompt word) and the image multimodal features of the image (also called the vector features of the image), as well as to determine the similarity between the text multimodal features and the image multimodal features. At this time, if the image recognition model is trained for a certain type of object or a certain type of practical application scenario, then it can be understood that the prompt word related to a certain type of object or a certain type of practical application scenario can be fixed, and then, the vector features of the predetermined dimension of the generated prompt word are also fixed within a certain range.

[0068] Furthermore, considering that the role of the fully connected layer in the entire image recognition model is to process multimodal data, the data processing process in the embodiment of the present application is: using the vector features of each sample prompt word and the vector features of the sample image, the similarity between the two regarding the vector features is calculated. Therefore, when the vector features of the generated sample prompt words are fixed within a certain range, the parameters determined based on the vector features of each sample prompt word can be used as the current weight parameters of the fully connected layer itself.

[0069] Therefore, the fully connected layer can be spliced ​​after the image encoder through data association to achieve the reconstruction of the content structure of the image recognition model, and obtain the image recognition model to be trained. Before the model training starts, each image description text with the presentation state of the specified object as the description content is used as each sample prompt word about the specified object. Then, the vector feature for each sample prompt word is obtained, and the matrix determined based on the vector features of each sample prompt word is used as the initial weight parameter of the fully connected layer to provide the vector features of the sample prompt word for subsequent data processing.

[0070] That is to say, the image recognition model to be trained provided in the embodiment of the present application includes a target fully connected layer with predetermined initial weight parameters, and an image encoder in a pre-trained contrastive learning model.

[0071] It should be noted that the model training is performed for a variety of specified objects in the present application. Therefore, the multiple sample prompt words used for model training are composed of at least two sample prompt words corresponding to each specified object, and the at least two sample prompt words corresponding to each specified object include sample prompt words whose presentation states are mutually exclusive for the specified object. For example, one specified object is smoke, and the other specified object is tattoo. When the specified object is smoke, the first sample prompt word in the corresponding prompt word group can be that smoke exists in the image, and the second sample prompt word can be that smoke does not exist in the image; when the specified object is tattoo, the first sample prompt word in the corresponding prompt word group can be that tattoo exists in the image, and the second sample prompt word can be that tattoo does not exist in the image. Correspondingly, the current weight parameters of the target fully connected layer itself in the embodiment of the present application are also determined based on the vector features of the above-mentioned multiple sample prompt words. The target fully connected layer can provide vector features of at least one sample prompt word for each specified object for subsequent data processing.

[0072] Furthermore, for each designated object, at least two sample prompt words corresponding to the designated object refer to sample prompt words that describe mutually exclusive presentation states for the designated object. That is, any sample prompt word corresponding to each designated object belongs to one of two prompt word types, the presentation states described by the two prompt word types for the same designated object are mutually exclusive, and the sample prompt words belonging to the same prompt word type have similar semantics.

[0073] For example, for a specified object: tattoo, the prompt phrase group corresponding to the specified object includes sample prompt word 1: tattoo and sample prompt word 2: no tattoo, as well as sample prompt word 3: cyan tattoo and sample prompt word 4: no cyan tattoo. Among them, sample prompt word 1: tattoo and sample prompt word 3: cyan tattoo both belong to the presentation state described by the sample prompt word as the presentation state of having tattoo, that is, the above sample prompt words 1 and 3 belong to the same prompt word type W, and the semantics of each sample prompt word in the prompt word type W are similar; sample prompt word 2: no tattoo and sample prompt word 4: no cyan tattoo both belong to the presentation state described by the sample prompt word as the presentation state of having no tattoo, that is, the above sample prompt words 2 and 4 belong to the same prompt word type M, and the semantics of each sample prompt word in the prompt word type M are similar. In summary, the presentation states described by the above prompt word type W and the above prompt word type M are mutually exclusive.

[0074] In this way, when determining the similarity between the vector features of each sample prompt word and the vector features of the sample image, based on the at least two sample prompt words whose presentation states for the same specified object are mutually exclusive, it can be determined which sample prompt word's presentation state the object's presentation state in the sample image is more consistent with, and the presentation state described by the more consistent sample prompt word is used as the presentation state of the sample image for the object, so as to achieve recognition of the sample image.

[0075] After obtaining the sample image, the sample image is input into the above-mentioned image recognition model to be trained to obtain the vector feature of each sample prompt word among multiple sample prompt words and the similarity with the vector feature of the sample image as the output result corresponding to the sample prompt word.

[0076] Optionally, in one implementation, the above S102, inputting the sample image into the image recognition model to be trained, and obtaining an output result corresponding to each sample prompt word in the plurality of sample prompt words, may include the following steps:

[0077] Step A: Input the sample image into the image encoder in the image recognition model to be trained so that the image encoder generates vector features of the sample image, and transmits the vector features of the sample image to the target fully connected layer to process the vector features of the sample image based on the current weight parameters of the target fully connected layer to obtain the output result corresponding to each sample prompt word.

[0078] In this implementation, after acquiring the sample image, the sample image is input into the image encoder in the above-mentioned image recognition model to be trained, so that the image encoder generates vector features of the sample image, and transmits the generated vector features of the sample image to the target fully connected layer in the image recognition model, so as to process the vector features of the sample image based on the current weight parameters of the target fully connected layer itself, that is, to calculate the similarity between the vector features of each sample prompt word in multiple sample prompt words and the vector features of the sample image, thereby obtaining the output result corresponding to each sample prompt word. Among them, the output result corresponding to each sample prompt word is characterized by a feature score, and this is not specifically limited in the embodiment of the present application.

[0079] Optionally, the cosine similarity is used to calculate the distance between the vector feature of each sample prompt word and the vector feature of the sample image. If the distance between the vector feature of the sample prompt word and the vector feature of the sample image is closer, the similarity between the two vector features is higher. Conversely, if the distance between the vector feature of the sample prompt word and the vector feature of the sample image is farther, the similarity between the two vector features is lower.

[0080] Optionally, considering that different data categories, such as text data, image data, etc., have corresponding vector features in different dimensions, in order to facilitate the subsequent determination of the similarity of each sample prompt word to the sample image, the vector features corresponding to different data types can be unified into the same dimension. Therefore, optionally, after obtaining the sample image, the image encoder of the image recognition model extracts the image features of the sample image, and generates vector features of a predetermined dimension of the sample image based on the extracted image features. Among them, the vector features of the image features are represented by the feature matrix, and this embodiment of the present application does not make any specific limitation.

[0081] In addition, considering that the image sizes of the acquired sample images are different, in order to ensure that each acquired sample image can be converted into image features of the same dimension, before further converting into vector features of a predetermined dimension, the sample image can be scaled to a predetermined size for use in the vector feature generation process. Based on this, optionally, after acquiring the sample image, the sample image is scaled to a predetermined size, and after image features are extracted from the sample image of the predetermined size, the extracted image features are converted into vector features of a predetermined dimension.

[0082] Exemplarily, after acquiring a sample image, the image size of the sample image is scaled to a fixed size, for example, 224×224, or 336×336, and then the scaled sample image is input into a module for generating image vector features to generate vector features of the sample image.

[0083] In this implementation, the dimensions of the image features of the sample image are converted to obtain vector features of the sample image at a predetermined dimension, which facilitates the subsequent determination of the similarity of each sample prompt word to the sample image, that is, determining the output result corresponding to each sample prompt word.

[0084] It should be noted that the embodiments of the present application involve a text-image pre-training model based on contrastive learning, and the main ability of this type of model is to align image features and text features in the same feature space (dimension). Therefore, in theory, only the trained model layer in the pre-training model can be used to convert image or text features to vector features, that is, the image features are converted to vector features through the image encoder in the pre-training model, and the text features are converted to vector features through the text encoder in the pre-training model. Other untrained methods cannot be used, otherwise the vector features of the obtained image and the vector features of the text will not be in the same feature space from a mathematical perspective.

[0085] Based on this, optionally, the sample image is input into the image encoder of the pre-constructed image recognition model to be trained, so that the image encoder extracts the image features of the sample image, and converts the extracted image features to obtain the vector features of the sample image; each sample prompt word is input into the text encoder of the pre-trained contrastive learning model corresponding to the image recognition model before construction, so that the text encoder extracts the text features of the sample prompt word, and converts the extracted text features to obtain the vector features of the sample prompt word as the matrix elements contained in the initial weight parameters of the fully connected layer.

[0086] In this way, since the generation method of the vector features of the image and the vector features of the text is pre-trained, the obtained vector features of the image and the vector features of the text are in the same feature space from a mathematical perspective, that is, the vector features obtained by the image encoder of the above-mentioned image recognition model and the text encoder of the contrastive learning model are in the same predetermined dimension.

[0087] S103: Determine a loss value corresponding to the image recognition model based on the output result corresponding to each sample prompt word and the image label of the sample image.

[0088] In this application, since the sample image is provided with an image label for representing the presentation state of the corresponding designated object, it can be understood that the presentation state of the designated object represented by the label of the sample image is the actual state presented by the designated object in the sample image. Therefore, after obtaining the output result corresponding to each sample prompt word, the loss value corresponding to the image recognition model can be determined based on the output result and the image label of the sample image.

[0089] S104: When it is determined based on the loss value that the image recognition model does not meet the model convergence condition, the current weight parameters of the target fully connected layer and the image encoder are adjusted, and the step of obtaining the sample image from the sample image set is returned.

[0090] In the present application, after determining the loss value corresponding to the image recognition model, it is possible to further determine whether the image recognition model meets the model convergence conditions, that is, determine whether the loss value reaches a predetermined loss value.

[0091] When it is determined based on the loss value that the image recognition model does not meet the model convergence conditions, the current weight parameters and image encoder of the target fully connected layer are adjusted, and the process returns to step S101, and the process continues to adjust the current weight parameters and image encoder of the target fully connected layer based on the next sample image using a model training method provided in an embodiment of the present application.

[0092] When it is determined based on the loss value that the image recognition model meets the model convergence condition, the training of the model is terminated.

[0093] Optionally, when it is determined based on the loss value that the image recognition model does not meet the model convergence condition, the current weight parameters of the target fully connected layer and the image encoder are adjusted according to a predetermined amplitude, that is, the parameters are adjusted according to a predetermined parameter adjustment amplitude. The above-mentioned predetermined parameter adjustment amplitude can be an amplitude represented by any value between [0, 1], and this application does not make specific limitations on this.

[0094] Optionally, when it is determined based on the loss value that the image recognition model does not meet the model convergence condition, the current weight parameters of the target fully connected layer and the image encoder are adjusted according to a predetermined direction, that is, the parameters are adjusted according to a predetermined parameter adjustment direction. For example, the parameters are adjusted in the direction of increasing the parameter value, or in the direction of reducing the parameter value. Among them, the specific adjusted parameter value can be any positive number, and this embodiment of the present application does not make specific limitations.

[0095] Optionally, when it is determined based on the loss value that the image recognition model does not meet the model convergence condition, the current weight parameters of the target fully connected layer and the image encoder are adjusted according to a predetermined direction and a predetermined amplitude, that is, the parameters are adjusted according to a predetermined parameter adjustment direction and with a predetermined parameter adjustment amplitude. For example, based on the current weight parameters of the target fully connected layer and the encoder parameters of the image encoder, the parameters are adjusted with an amplitude represented by 0.5 in the direction of increasing the parameter value.

[0096] In order to improve the efficiency of model training, optionally, the number of model training times reaching a predetermined number of times is set as a model training end condition, that is, when the number of model training times reaches the predetermined number of times, the training of the model is ended.

[0097] Optionally, a back propagation algorithm is used to calculate the gradient values ​​of the feature vectors corresponding to the weight parameters of the target fully connected layer and the image encoder based on the loss value, and the weight parameters of the target fully connected layer and the encoder parameters of the image encoder are adjusted using the calculated gradient values. Thus, during the model training process, the model parameters are adjusted according to the determined loss value to achieve the purpose of balancing the above loss value and optimizing the model performance.

[0098] As can be seen from the above, in the model training method provided by the embodiment of the present application, the image recognition model to be trained includes a target fully connected layer with a predetermined initial weight parameter, and an image encoder in a pre-trained contrastive learning model. Among them, the initial weight parameter of the target fully connected layer is determined in advance based on the vector features of multiple sample prompt words; the above-mentioned multiple sample prompt words include at least two sample prompt words corresponding to each specified object, and the at least two sample prompt words corresponding to each specified object include sample prompt words whose presentation states are mutually exclusive. That is to say, the target fully connected layer can provide corresponding vector features for multiple specified objects. Therefore, no matter what kind of specified object is contained in the input sample image, the target fully connected layer can obtain the similarity between the vector features of each sample prompt word and the vector features of the sample image as the output result corresponding to the sample prompt word. Then, based on the output result corresponding to each sample prompt word and the image label of the sample image, the current weight parameter of the target fully connected layer and the loss value of the image encoder are determined, and the current weight parameter of the target fully connected layer and the image encoder are adjusted. In this way, by reconstructing the model structure of the image recognition model, targeted training can be achieved for multiple specified objects in one model. There is no need to train a targeted model separately for each specified object, thereby reducing the amount of computing resources occupied by the model during model training.

[0099] Optionally, in one embodiment, the target fully connected layer includes a plurality of sub-fully connected layers, each sub-fully connected layer corresponds to a specified object, and each sub-fully connected layer is used to process the vector features of the sample image based on the current weight parameters of the sub-fully connected layer to obtain the similarity between the vector features of each sample prompt word corresponding to the specified object and the vector features of the sample image;

[0100] The initial weight parameters of each sub-fully connected layer are determined by:

[0101] Step B: Obtain the vector features of each sample prompt word of the sub-fully connected layer corresponding to the specified object, and concatenate each obtained vector feature into a matrix to obtain the initial weight parameters of the sub-fully connected layer.

[0102] In this embodiment, considering that the present application is a model training for multiple specified objects, the target fully connected layer in the reconstructed image recognition model includes multiple sub-fully connected layers; wherein each sub-fully connected layer corresponds to a specified object, and each sub-fully connected layer is used to process the vector features of the sample image based on the current weight parameters of the sub-fully connected layer, and obtain the similarity between the vector features of each sample prompt word corresponding to the specified object and the vector features of the sample image.

[0103] Therefore, for each sub-fully connected layer, the vector feature of each sample prompt word corresponding to the specified object of the sub-fully connected layer is obtained, and then each obtained vector feature is spliced ​​into a matrix to obtain the initial weight parameter of the sub-fully connected layer.

[0104] Taking into account different data types, such as text data, image data, etc., their corresponding vector features are in different dimensions. In order to facilitate the subsequent determination of the similarity of each sample prompt word to the sample image, the vector features corresponding to data of different data types can be unified into the same dimension. Therefore, after obtaining the text features of each sample prompt word, the vector features of the predetermined dimension of the sample prompt word can be generated.

[0105] Considering that the feature matrix can be used to represent the vector features of the sample prompt words, after obtaining the vector features of each sample prompt word of the specified object, each vector feature can be spliced ​​into a matrix in a manner of splicing each vector feature as a row of matrix elements to obtain the initial weight parameters of the sub-fully connected layer corresponding to the specified object.

[0106] For example, the vector feature of each sample prompt word is a feature matrix of [1, 512]. Therefore, when there are 10 sample prompt words, the weight parameter of the fully connected layer is the concatenated matrix [10, 512]. Among them, the first row of matrix elements is the vector feature corresponding to the first sample prompt word, the second row of matrix elements is the vector feature corresponding to the second sample prompt word, and so on. The tenth row of matrix elements is the vector feature corresponding to the tenth sample prompt word.

[0107] In this embodiment, before training the image recognition model, by predetermining the initial weight parameters of the target fully connected layer, the number of training samples required for the training process can be reduced while providing pre-training parameters for the target fully connected layer for model training. In this way, when training the model, only sample images need to be used to enable the trained image recognition model to obtain better training results.

[0108] Furthermore, it is understandable that although the target fully connected layer can provide vector features of multiple sample prompt words, considering that the vector features of sample prompt words corresponding to different specified objects are quite different, if the vector features of sample prompt words corresponding to different specified objects are mixed, the vector features of various specified objects will interfere with each other, resulting in a low accuracy rate of the similarity of each sample prompt word to the sample image, thus affecting the recognition accuracy rate of the image recognition model.

[0109] Therefore, in this embodiment, the vector features of at least two sample prompt words belonging to the same specified object are clustered by means of a sub-fully connected layer. In this way, when the parameters of the target fully connected layer are subsequently adjusted, targeted adjustments can be made only to the current weight parameters of the sub-fully connected layer that is the same as the specified object in the sample image.

[0110] Optionally, each sub-fully connected layer belongs to a parallel network layer and is connected to the image encoder.

[0111] Exemplarily, the image recognition model to be trained is composed of an image encoder and a target fully connected layer. For example, the image recognition model can be composed of an image encoder in a CLIP model and a target fully connected layer obtained by splicing vector features of multiple sample prompt words extracted based on a text encoder in the CLIP model.

[0112] However, the target fully connected layer includes a sub-fully connected layer corresponding to each specified object. Therefore, the process of the image encoder transmitting the vector features of the generated sample image to the target fully connected layer is actually the process of transmitting the vector features of the sample image to each sub-fully connected layer. In other words, each sub-fully connected layer belongs to a parallel network layer and is connected to the image encoder.

[0113] It should be clear that the model training process and the model inference process have the same way of processing input data. Therefore, in the inference process of the image recognition model, after obtaining the vector features of the target image, since it is unclear what kind of specified object the target image contains, the vector features of the target image can be transmitted to each sub-fully connected layer to determine the similarity between the vector features of each sample prompt word in the sub-fully connected layer and the vector features of the sample image, thereby determining the similarity between the vector features of each sample prompt word in the entire target fully connected layer and the vector features of the sample image.

[0114] Afterwards, the vector features of the sample image are processed through the sub-fully connected layer corresponding to each specified object based on the current weight parameters of the sub-fully connected layer to obtain the similarity between the vector features of each sample prompt word corresponding to the specified object and the vector features of the sample image.

[0115] Optionally, in one implementation, the above step S103, based on the output result corresponding to each sample prompt word and the image label of the sample image, determines the loss value corresponding to the image recognition model, which may include the following steps:

[0116] Step C1: based on the output results corresponding to each sample prompt word, determine the target confidence corresponding to the target sample prompt word;

[0117] The presentation state described by the target sample prompt word is the same as the presentation state of the specified object represented by the image label of the sample image; the target confidence is used to represent the presentation state described in the target sample prompt word, which is the predicted probability of the presentation state of the specified object in the sample image;

[0118] Step C2: Based on the target confidence and the image label of the sample image, determine the loss value corresponding to the image recognition model.

[0119] In this implementation, since the sample image is provided with an image label for characterizing the presentation state of the corresponding designated object, and each designated object corresponds to at least two sample prompt words with the presentation state of the designated object as the description content, that is, the sample prompt word corresponding to each designated object can be understood as the image label set to characterize the presentation state of the designated object. However, in a model training process, each sample image will only be provided with one image label, that is, among multiple sample prompt words, there will only be one target sample prompt word corresponding to the sample image, and the presentation state described by the target sample prompt word is the same as the presentation state of the designated object represented by the image label of the sample image.

[0120] In summary, based on the output results corresponding to each sample prompt word and the loss value determined by the image label of the sample image, it can be understood that it is caused by the low accuracy of the vector features of the target sample prompt word in the target fully connected layer of the image recognition model. Therefore, the loss value corresponding to the image recognition model can be determined based on the target confidence and the image label of the sample image.

[0121] In this way, after determining the output result corresponding to each sample prompt word, the target confidence corresponding to the target sample prompt word is further determined based on each output result to determine the prediction probability that the presentation state described in the target sample prompt word is the presentation state of the specified object in the sample image. In other words, the higher the target confidence obtained, the higher the prediction probability that the presentation state described in the target sample prompt word is the presentation state of the specified object in the sample image.

[0122] In this way, based on the target confidence corresponding to the target sample prompt word and the label of the sample image, the loss value corresponding to the image recognition model can be determined. The smaller the determined loss value is, the closer the presentation state described in the target sample prompt word is to the presentation state of the specified object in the sample image.

[0123] Among them, the image label set by the sample image for representing the presentation state of the specified object can be understood as the image label used to represent the actual state presented by the specified object in the sample image. Therefore, the corresponding confidence can be set directly based on the actual state presented by the specified object. For example, if the specified object is a tattoo, one presentation state represented by the image label is the presence of a tattoo, and the corresponding confidence of the presentation state represented by the image label can be set to 1; conversely, one presentation state represented by the image label is the absence of a tattoo, and the corresponding confidence of the presentation state represented by the image label can be set to 0.

[0124] Optionally, the target confidence corresponding to the target sample prompt word and the confidence corresponding to the image label of the sample image are input into a predetermined loss function to calculate the loss value corresponding to the image recognition model. The predetermined loss function may include a cross entropy loss function, etc., which is not specifically limited in the embodiments of the present application.

[0125] In this implementation, the confidence level is used to digitize the probability that the presentation state described in the target sample prompt word is the presentation state of the specified object corresponding to the sample image, so as to determine the loss value of the image recognition model.

[0126] Optionally, in one implementation, before the above step S104 adjusts the current weight parameters of the target fully connected layer and the image encoder, a model training method provided by an embodiment of the present application further includes the following steps:

[0127] Step D: Determine the target sub-fully connected layer to be parameter adjusted;

[0128] Among them, the target sub-fully connected layer is a sub-fully connected layer corresponding to the specified object represented by the image label of the sample image;

[0129] Accordingly, the above step S104, adjusting the current weight parameters of the target fully connected layer and the image encoder parameters, may include the following steps:

[0130] Step E: Adjust the current weight parameters of the target sub-fully connected layer and the image encoder in the target fully connected layer.

[0131] In this embodiment, the vector features of at least two sample prompt words belonging to the same specified object are clustered by dividing the sub-fully-connected layers. In this way, when it is judged that the image recognition model does not meet the model convergence conditions based on the loss value, the sub-fully-connected layer that is the same as the specified object represented by the image label of the sample image can be determined as the target sub-fully-connected layer to be parameter adjusted. In this way, when adjusting the current weight parameters and image encoder of the target fully-connected layer, it is only necessary to adjust the parameters of the current weight parameters and image encoder of the target sub-fully-connected layer, and there is no need to adjust the parameters of all sub-fully-connected layers in the entire target fully-connected layer.

[0132] Optionally, the above step C1, determining the target confidence corresponding to the target sample prompt word based on the output result corresponding to each sample prompt word, may include the following steps:

[0133] Step C11: determining the target confidence corresponding to the target sample prompt word based on the output result corresponding to the target sample prompt word and the ratio of the target and value;

[0134] The target and value are obtained by summing the output results corresponding to multiple target prompt words; the designated object described by the target prompt word is the same designated object, and the same designated object is the object contained in the sample image.

[0135] In this implementation, after obtaining the output result corresponding to each sample prompt word, the sum of the similarities of multiple target prompt words can be calculated as the target sum value. Thus, based on the ratio of the output result corresponding to the target sample prompt word to the above target sum value, the target confidence corresponding to the target sample prompt word can be determined.

[0136] It should be noted that when determining the target confidence of the target sample prompt word, the sample image, the target sample prompt word and the target prompt word are directed to the same designated object.

[0137] Optionally, for each sample prompt word, the output result corresponding to the sample prompt word is normalized to obtain the confidence corresponding to the sample prompt word, and then the confidence corresponding to the target sample prompt word is selected from each confidence as the target confidence.

[0138] It should be emphasized that the multiple sample prompt words include the sample prompt words corresponding to each designated object. Therefore, when normalizing the output results corresponding to each sample prompt word, if the sum of the output results of the sample prompt words corresponding to each designated object is referred to, it will affect the confidence of the sample prompt word obtained after the normalization. Therefore, when normalizing the output results corresponding to each sample prompt word, it is only necessary to refer to the output result corresponding to the sample prompt word and the sum of the output results of all sample prompt words that are the same object as the designated object described by the sample prompt word to ensure the accuracy of the obtained confidence.

[0139] Optionally, for each sample prompt word, a confidence level of the sample prompt word is calculated using a predetermined formula; wherein the predetermined formula includes:

[0140]

[0141] σ(z i ) represents the confidence level corresponding to the i-th sample prompt word of the specified object, K represents the total number of all target prompt words, z i Characterizes the feature score corresponding to the i-th sample prompt word about the specified object, z j Characterizes the feature score corresponding to the j-th sample prompt word about the specified object.

[0142] It should be noted that the designated object described by all the target prompt words is the same designated object, that is, the confidence corresponding to each sample prompt word is calculated by the feature score of the sample prompt word and the sum of the feature scores of all sample prompt words that are the same object as the designated object described by the sample prompt word. This calculation method can reduce the interference of sample prompt words of other designated objects on the calculated confidence of each sample prompt word, thereby ensuring the accuracy of the obtained confidence.

[0143] The feature score corresponding to each sample prompt word is used to characterize the similarity between the sample prompt word and the sample image, that is, the output result corresponding to each sample prompt word in the embodiment of the present application.

[0144] In this implementation, by standardizing the output results corresponding to each sample prompt word to a specific interval, for example, [0, 1] or [-1, 1], etc., the data distribution has statistical significance, and the probability distribution of similarity is more statistically intuitive, thereby improving the comparability of the output results corresponding to each sample prompt word.

[0145] The following is a specific example in which a model training method provided in the present application is specifically introduced.

[0146] Since conventional deep learning recognition models need to label large quantities of data, and in actual business, the data that needs to be recognized has a long-tail distribution, that is, the data that needs to be recognized accounts for a small proportion of the total data and is difficult to collect, resulting in high data labeling costs and slow model iteration cycles. Therefore, considering that the CLIP model does not need to collect data to complete image recognition, that is, the CLIP model and other contrastive learning models pre-trained on large quantities of data can directly predict the required classification by inputting prompt words. Such models can be directly applied to the review business according to business needs. For example, according to business needs, review whether there are tattoos in the image. However, since the pre-training data may not contain the features required by the business data, for example, the pre-training data may not contain image features and text features related to tattoos, the accuracy of the recognition results obtained by this type of model when recognizing images containing tattoos is low.

[0147] In the related art, when it is necessary to identify the presentation status of a specified object, the pre-trained image recognition model is trained in a targeted manner using sample prompt words and sample images corresponding to the specified object. This targeted training enables the trained image recognition model to identify the presentation status of the specified object in the input image based on the output results of the image recognition model without the need to input prompt words in subsequent applications.

[0148] However, the image recognition model obtained through the above targeted training can only be trained and reasoned for one task, that is, it can only recognize the presentation state of a specified object. If multiple objects need to be recognized, it is necessary to train each object to obtain a highly targeted image recognition model that can recognize the presentation state of the object, which results in a large amount of computing resources being required during model training and model reasoning.

[0149] For example, when there are tasks X and Y, the corresponding model x needs to be trained to perform reasoning for task X, and the model y needs to be trained to perform reasoning for task Y. However, in a non-current task dataset, there may be categories corresponding to the labels of the current dataset, but the corresponding labels are not annotated. For example, if task X is to classify different animals, and task Y is to classify different plants, then the image of the dataset corresponding to task X may include the plant types included in task Y, but the image is not annotated with the labels corresponding to task Y. Similarly, the image of the dataset corresponding to task Y may include the animal types included in task X, but the image is not annotated with the labels corresponding to task X.

[0150] At this point, if the above two data sets are mixed and trained on one model using the conventional deep learning model training method, the recognition effect of the model may be reduced.

[0151] Based on this, this specific embodiment provides a flow chart of a specific embodiment, such as Figure 2 As shown, taking the case where the number of specified objects is two as an example, the image recognition model includes an image encoder 210, a fully connected layer 221 and a fully connected layer 222, wherein the fully connected layer 221 and the fully connected layer 222 are parallel to each other and are both connected to the image encoder 210.

[0152] The above specific embodiment is introduced below in combination with specific implementation steps.

[0153] For the first designated object: tattoo, set the corresponding business (task) P; for the second designated object: smoke, set the corresponding business (task) Q. The two businesses are independent of each other, and both are businesses for reviewing images uploaded by users. Specifically, in the review business, business P is to review whether there are tattoos in the image, and business Q is to review whether there is smoke in the image. Moreover, in the training data set (sample image subset) annotated by business P, there are images with smoke corresponding to business Q, but the image is not annotated with the corresponding label; in the training data set (sample image subset) annotated by business Q, there are images with tattoos corresponding to business P, but the image is not annotated with the corresponding label.

[0154] In combination with the above examples, the specific implementation of this specific embodiment may include the following steps:

[0155] Step 1: Set each initial prompt word for business P and Q as each prompt word for model training, and use a text encoder to convert each prompt word into a corresponding text multimodal feature. The above initial prompt words are used to train the image recognition model, and the above initial prompt words are the sample prompt words in the embodiment of this application.

[0156] Specifically, before model training, initial prompt words are set for business P and business Q respectively.

[0157] For example, for business P, when the image has a tattoo, the initial prompt words are set to "a photo of a person with a tattoo", and when the image does not have a tattoo, the initial prompt words are set to "a photo of person with a tattoo". For business Q, when the image has cigarettes, the initial prompt words are set to "a photo with cigarettes", and when the image does not have tattoos, the initial prompt words are set to "a photo without cigarettes".

[0158] Then, the initial prompt words corresponding to the above services P and Q are respectively input into the text encoder of the CLIP model to obtain the text features of each initial prompt word, and then each text feature is converted into a text multimodal feature by passing it through the text multimodal network. Here, the text multimodal feature corresponding to each initial prompt word is a feature matrix of size [1, 512].

[0159] The specific implementation of the above step 1 is similar to the specific implementation of the above determination of the vector features of each sample prompt word, which will not be repeated here.

[0160] Step 2: Convert the text multimodal features corresponding to the initial prompt word of business P and the text multimodal features corresponding to the initial prompt word of business Q into corresponding fully connected layer weights, and splice them behind the corresponding image multimodal model (image encoder) to obtain the recognition model to be tuned. Among them, the above-mentioned recognition model to be tuned is the image recognition model to be trained in the embodiment of the present application; the above-mentioned fully connected layer weights are the initial weight parameters of the target fully connected layer in the embodiment of the present application.

[0161] Specifically, considering that the actual role of the fully connected layer in the deep learning recognition model is to calculate the similarity between the features of the content described by the specific label and the image features by dot multiplication and addition, and dividing by the vector length of the text multimodal features of the content described by each specific label and the image multimodal features of the input image. In the actual reasoning process, image recognition models such as the CLIP model also need to calculate the similarity between the text multimodal features and the image multimodal features by dot multiplication and addition, and dividing by the vector length of each text multimodal feature and the image multimodal feature. Therefore, it can be considered that the role of the acquired text multimodal features is consistent with the role of the fully connected layer in the conventional deep learning recognition model, both of which are text multimodal features that provide prompt words. Therefore, the predetermined text multimodal features corresponding to each of the business P and business Q in the above step 1 can be converted into corresponding fully connected layer weights, and the fully connected layer after adding the weights is spliced ​​behind the image encoder to obtain the recognition model to be tuned.

[0162] Since the image encoder sets a fixed preset size for the size of the input image, the image is scaled to the fixed preset size before being input into the image encoder, wherein the fixed preset size is usually 224×224 or 336×336, both of which are reasonable and are not specifically limited in the embodiments of the present application.

[0163] In this way, after receiving the input image, the image encoder in the recognition model to be tuned can extract the image features of the image, and pass the image features through the image multimodal network in the image encoder to convert the image features into image multimodal features. Here, the image multimodal features corresponding to each image are feature matrices of size [1, 512].

[0164] The specific implementation method corresponding to the above step 2 is similar to the implementation method of inputting the sample image into the image recognition model to be trained in the embodiment of the present application, so that the image encoder in the image recognition model generates vector features of the sample image, and obtains the vector features of the sample image, which will not be repeated here.

[0165] Step 3: Mix the training data sets corresponding to business P and business Q, use the back propagation algorithm to tune the spliced ​​model, and tune different parts of the model according to the different sources of the pictures to obtain a model tuned on business P and business Q based on the contrast learning model; wherein the mixed training data set is the sample image set in the embodiment of the present application, specifically:

[0166] Multiple text multimodal features and image multimodal features are sequentially point-multiplied and added, and then divided by the vector length of each text multimodal feature and image multimodal feature to calculate the similarity of each text multimodal feature and image multimodal feature, thereby obtaining the feature score between the text feature of each sample prompt word and the input image. The feature score is used to characterize the similarity between each text feature and image feature. At this time, for business P and business Q, the corresponding fully connected layer P' and fully connected layer Q' are obtained respectively; wherein, the fully connected layer P' and the fully connected layer Q' are both sub-fully connected layers in the embodiment of the present application; the fully connected layer P' and the fully connected layer Q' together constitute the target fully connected layer in the embodiment of the present application.

[0167] Then, define the feature score as z, and perform softmax (normalization) operations on the feature scores passing through the fully connected layer P' and the fully connected layer Q', respectively. Take business P and the fully connected layer P' as an example:

[0168]

[0169] σ(z) i represents the confidence corresponding to the i-th initial prompt word about business P, K represents the total number of initial prompt words about business P, z i Characterizes the feature score corresponding to the i-th initial prompt word of business P, z j Characterizes the feature score corresponding to the j-th initial prompt word about business P.

[0170] Through the above formula, the feature score corresponding to each sample prompt word can be mapped to a confidence level between 0 and 1 as the confidence level of the prediction.

[0171] Then, based on the cross entropy loss function, the loss value between the confidence predicted by the model and the image label of the image itself is calculated, and the back propagation algorithm is used to update the weight of the corresponding fully connected layer based on the loss value, that is, the text modality feature weights of the initial prompt words related to business P and the image encoder weight features are updated based on the loss value. In other words, the parameter adjustment of the weight of the fully connected layer is a targeted adjustment. For example, when the sample image is an image in business P, what needs to be adjusted is the weight of the fully connected layer P' corresponding to business P; when the sample image is an image in business Q, what needs to be adjusted is the weight of the fully connected layer Q' corresponding to business Q.

[0172] Exemplarily, when the sample image is an image in the business P data set, the confidence determined by the fully connected layer Q' is discarded, and only the confidence determined by the fully connected layer P' is used to calculate the loss value of the business P, and only the weight of the fully connected layer P' and the weight of the image encoder are updated, and the weight of the fully connected layer Q' remains unchanged; similarly, when the sample image is an image in the business Q data set, the confidence determined by the fully connected layer P' is discarded, and only the confidence determined by the fully connected layer Q' is used to calculate the loss value of the business Q, and only the weight of the fully connected layer Q' and the weight of the image encoder are updated, and the weight of the fully connected layer P' remains unchanged.

[0173] Among them, the weight of the fully connected layer is the matrix obtained by splicing the vector features corresponding to at least one sample prompt word of a specified object in the embodiment of the present application, that is, the initial weight parameter of the target sub-fully connected layer in the embodiment of the present application. It should be emphasized that if this update is not the first update, the updated weight of the fully connected layer is the current weight parameter of the target sub-fully connected layer in the embodiment of the present application.

[0174] In this way, the above steps 2 and 3 are repeated until all images in the data set are extracted, and gradient descent calculation is performed to optimize the recognition model to be tuned.

[0175] The above steps 1 to 3 can be regarded as one round of model training. Repeat multiple rounds of training until the preset number of training rounds or the loss value reaches a preset value.

[0176] Compared with the traditional model training process, a targeted model needs to be trained for reasoning for each business. This specific embodiment uses data sets of multiple independent businesses on one model and optimizes multiple independent businesses at the same time, that is, one model is used to train and train multiple different businesses, which can reduce the amount of computing resources occupied by the model during model training and reasoning. For example, when there are n independent businesses, if the traditional model training method is used, n models need to be trained, and n graphics card resources need to be occupied for online reasoning; while using the training method of this specific embodiment, only 1 graphics card resource is needed to reason about n businesses, and the average business resource occupancy is only 1 / n of the traditional method, which greatly reduces the amount of computing resources occupied by the model during model training and reasoning.

[0177] The following is a detailed introduction to an image recognition method provided in an embodiment of the present application.

[0178] Among them, an image recognition method provided in the embodiment of the present application can be applied to an electronic device, and the electronic device is specifically used for image recognition. In a specific application, the electronic device can be a smart phone, a tablet computer, etc., which is reasonable. Among them, the electronic devices used for image recognition and model training can be the same electronic device or different electronic devices, and the embodiment of the present application does not make specific limitations.

[0179] An image recognition method provided in an embodiment of the present application may include the following steps:

[0180] Acquire a target image; wherein the target image is an image used to identify a target object;

[0181] The target image is input into the trained image recognition model to obtain an output result corresponding to each sample prompt word in each sample prompt word; wherein the image recognition model is trained based on a model training method provided in an embodiment of the present application; each sample prompt word is a prompt word used to train the image recognition model; the target object belongs to one of multiple specified objects; the output result corresponding to each sample prompt word is used to characterize the similarity between the vector feature of the sample prompt word and the vector feature of the target image;

[0182] Based on the output result corresponding to each sample prompt word, a presentation state of the target image for the target object is determined.

[0183] As can be seen from the above, in the image recognition method provided by the embodiment of the present application, the image recognition model used is obtained by training the training samples for multiple specified objects, and the trained image recognition model can perform image recognition on multiple different specified objects. Therefore, when the target object in the input target image is any one of the multiple specified objects, the image recognition model can recognize the target image, thereby achieving the purpose of using one image recognition model to perform image recognition on multiple different specified objects and reducing the amount of computing resources occupied by the model during model inference.

[0184] Furthermore, the above-mentioned image recognition model is trained based on a model training method provided in an embodiment of the present application, so the sample prompt words when the image recognition model performs image recognition are the prompt words obtained by training the image recognition model. Therefore, when the image recognition model is used for image recognition, there is no need to input prompt words, and only the target image can be input to realize automatic recognition of the image.

[0185] An image recognition method provided in an embodiment of the present application is described in detail below in conjunction with the accompanying drawings.

[0186] Figure 3A flowchart of an image recognition method provided in an embodiment of the present application is shown in FIG. Figure 3 As shown, the method may include the following steps:

[0187] S301: Acquire a target image;

[0188] The target image is an image used to identify the target object.

[0189] In the present application, an image for identifying a target object is acquired as a target image, so that an image recognition method provided by an embodiment of the present application is performed on the target image.

[0190] Optionally, since the image recognition model has a size restriction range for the input image size, after acquiring the target image, the image size of the target image can be scaled to a predetermined size according to the size restriction range to be input to the model. The predetermined size is within the size restriction range to be input to the model.

[0191] It should be noted that the acquisition, storage, use, processing, transmission, provision and disclosure of images involved in the technical solution of this application are all carried out with the authorization of the user. In addition, the images of the human body involved in this application are not images of the human body of a specific user and do not reflect the personal information of a specific user.

[0192] S302: Input the target image into the trained image recognition model to obtain an output result corresponding to each sample prompt word in each sample prompt word;

[0193] The image recognition model is trained based on a model training method provided in an embodiment of the present application; each sample prompt word is a prompt word used to train the image recognition model; the target object belongs to one of multiple specified objects; the output result corresponding to each sample prompt word is used to characterize the similarity between the vector feature of the sample prompt word and the vector feature of the target image;

[0194] In the present application, the image recognition model used is trained based on a model training method provided in the embodiment of the present application, that is, the vector features of the prompt words used for image recognition in the image recognition model used are fixed, that is, the vector features of the sample prompt words used for image recognition are also the vector features used for training the image recognition model. Therefore, when using the image recognition model for image recognition, there is no need to input the prompt words, only the target image is required.

[0195] Therefore, after the target image is input into the image recognition model, the output result corresponding to each sample prompt word can be determined based on the similarity between the vector feature of each sample prompt word and the vector feature of the target image.

[0196] S303: Determine the presentation status of the target image for the target object based on the output result corresponding to each sample prompt word.

[0197] In the present application, after obtaining the output result corresponding to each sample prompt word, the similarity between the presentation state described by the sample prompt word and the presentation state of the target object in the target image is determined, wherein the higher the similarity, the higher the possibility that the presentation state described by the prompt word is the presentation state of the target object in the target image. Therefore, after determining the output result corresponding to each sample prompt word, the presentation state of the target object in the target image can be determined based on the determined output result.

[0198] As can be seen from the above, in the image recognition method provided by the embodiment of the present application, the image recognition model used is obtained by training the training samples for multiple specified objects, and the trained image recognition model can perform image recognition on multiple different specified objects. Therefore, when the target object in the input target image is any one of the multiple specified objects, the image recognition model can recognize the target image, thereby achieving the purpose of using one image recognition model to perform image recognition on multiple different specified objects and reducing the amount of computing resources occupied by the model during model inference.

[0199] Furthermore, the above-mentioned image recognition model is trained based on a model training method provided in an embodiment of the present application, so the sample prompt words when the image recognition model performs image recognition are the prompt words obtained by training the image recognition model. Therefore, when the image recognition model is used for image recognition, there is no need to input prompt words, and only the target image can be input to realize automatic recognition of the image.

[0200] Optionally, in one implementation, the above step S303, based on the output result corresponding to each sample prompt word, determining the presentation state of the target image for the target object may include the following steps:

[0201] Step L1: Based on the output result corresponding to each sample prompt word, determine the confidence level corresponding to the sample prompt word;

[0202] The confidence level corresponding to the sample prompt word is used to characterize the presentation state described in the sample prompt word, which is the probability of the presentation state of the target object corresponding to the target image;

[0203] Step L2: Based on the confidence level corresponding to each sample prompt word, analyze the presentation status of the target image for the target object.

[0204] In this implementation, for each sample prompt word, after determining the output result corresponding to the sample prompt word, the confidence corresponding to the sample prompt word is further determined to determine the probability that the presentation state described in the sample prompt word is the presentation state of the target object corresponding to the target image. In other words, the higher the confidence corresponding to the determined sample prompt word, the higher the probability that the presentation state described in the sample prompt word is the presentation state of the target object corresponding to the target image. Afterwards, based on the confidence corresponding to each sample prompt word, the presentation state of the target image for the target object is analyzed.

[0205] In this implementation, the confidence level is used to digitize the probability that the presentation state described in each sample prompt word is the presentation state of the target object corresponding to the target image, so as to match the target image with the corresponding sample prompt word.

[0206] Optionally, in one implementation, the above step L2, analyzing the presentation status of the target image for the target object based on the confidence level corresponding to each sample prompt word, may include the following steps:

[0207] Step L21: Select the highest confidence from the confidences corresponding to each sample prompt word;

[0208] Step L22: If the selected confidence level is greater than a predetermined threshold, the presentation state of the target image for the target object is determined to be: the presentation state described by the sample prompt word corresponding to the selected confidence level.

[0209] In this implementation, since the higher the confidence level corresponding to the determined sample prompt word, the higher the probability that the presentation state described in the sample prompt word is the presentation state of the target object corresponding to the target image, after determining the confidence level corresponding to each sample prompt word, the highest confidence level can be selected.

[0210] However, considering the possibility that the determined confidence is the highest confidence, but the probability that the presentation state described in the sample prompt word represented by the confidence is the presentation state of the target object corresponding to the target object is still low, for example, generally speaking, the two can be considered to match only when the confidence is above 50%. Then, when the highest confidence determined is 35%, although this confidence is the highest confidence among the confidences corresponding to the sample prompt words, for the case where the confidence is 50% for the two to be considered to match, the presentation state described by the sample prompt word corresponding to the confidence is actually not matched with the presentation state of the target object for the target object.

[0211] Therefore, in order to improve the accuracy of the recognition result corresponding to the determined target image, a predetermined threshold is set in advance for the confidence level, that is, when the selected confidence level is greater than the predetermined threshold, it is characterized that the presentation state described by the sample prompt word is the presentation state of the target object corresponding to the target image. Therefore, by comparing the selected confidence level with the predetermined threshold, it can be determined whether the presentation state described by the sample prompt word is the presentation state of the target object corresponding to the target image.

[0212] If the selected confidence level is greater than the predetermined threshold, the presentation state described by the sample prompt word corresponding to the confidence level is represented as the presentation state of the target object corresponding to the target image.

[0213] If the selected confidence level is not greater than the predetermined threshold, it indicates that the presentation state described by the sample prompt word corresponding to the confidence level is not the presentation state of the target object corresponding to the target image.

[0214] In this implementation, the analysis accuracy of the presentation state of the target object in the determined target image is further improved by presetting a predetermined threshold value regarding the confidence level.

[0215] Optionally, in one implementation, the above step L1, based on the output result corresponding to each sample prompt word, determines the confidence level corresponding to the sample prompt word, may include the following steps:

[0216] Step L11: For each sample prompt word, normalize the output result corresponding to the sample prompt word to obtain the confidence level corresponding to the sample prompt word.

[0217] In this implementation, after obtaining the output result corresponding to each sample prompt word, the output result corresponding to each sample prompt word may be normalized to obtain the confidence level corresponding to the sample prompt word.

[0218] Optionally, for each sample prompt word, a confidence level corresponding to the sample prompt word is calculated using a predetermined formula; wherein the predetermined formula includes:

[0219]

[0220] σ(z i ) represents the confidence level of the i-th sample prompt word corresponding to the target object, K represents the total number of sample prompt words that describe the same specified object, and z i Characterize the feature score corresponding to the i-th sample prompt word, z j Characterize the feature score corresponding to the j-th sample prompt word.

[0221] The feature score corresponding to each sample prompt word is used to characterize the similarity between the sample prompt word and the target image, that is, the output result corresponding to each sample prompt word in the embodiment of the present application.

[0222] This implementation method, by standardizing the output result corresponding to each sample prompt word to a specific interval, for example, [0, 1] or [-1, 1], etc., makes the data distribution statistically significant and the probability distribution of similarity more statistically intuitive, thereby improving the comparability of the output results corresponding to each sample prompt word.

[0223] Considering that for an image, there may be multiple target objects to be identified in the image area it contains, in order to improve the accuracy of overall image recognition, it is necessary to identify the image area corresponding to each target object in the image. Based on this, optionally, in one embodiment, a region detection algorithm for the carrier area is used to perform region detection on the initial image to obtain a region of interest, and the obtained region of interest is intercepted to obtain a target image; wherein the region of interest is the carrier area where the target object to be identified is located in the initial image when the target object to be identified exists in the initial image.

[0224] In this embodiment, considering that the target object to be identified exists in the initial image, the carrier region where the target object is located in the initial image will not change much. Therefore, after acquiring the initial image, the region detection algorithm for the carrier region is used to perform region detection on the initial image to obtain the region of interest, and the region of interest in the initial image is intercepted to obtain the target image. In other words, the image region contained in the obtained target image is the region of interest.

[0225] Considering that the types of objects to be identified are different, the corresponding carrier areas are different, and accordingly, the region detection algorithms for the carrier areas are also different. For example, if the object to be identified is a tattoo, the corresponding carrier area is the human body area, and accordingly, the region detection algorithm for the carrier area can be a human body detection algorithm; if the object to be identified is a trademark, the corresponding carrier area is the edge area of ​​the target image, and accordingly, the region detection algorithm for the carrier area can be a trademark detection algorithm. Therefore, optionally, based on the type of object to be identified, the corresponding region detection algorithm is selected, and the target image is subjected to region detection to obtain the region of interest.

[0226] Among them, the above-mentioned human body detection algorithm can be an anchor-base (based on anchor frame) human body detection algorithm, and the embodiment of the present application does not make any specific limitation on this.

[0227] In this embodiment, by intercepting the region of interest in the initial image, the interference of other image elements in the image on the recognition of the target image can be reduced, thereby improving the accuracy of automatic recognition of the image.

[0228] Considering that for an initial image, there may be one or more regions of interest contained therein. When there is one region of interest, the number of intercepted target images is one; correspondingly, when there are multiple regions of interest, the number of intercepted target images is multiple. The same method can be used to process each target image to obtain the recognition result corresponding to each target image.

[0229] That is to say, after determining the output result corresponding to each sample prompt word, the degree of similarity between the presentation state described by the GIA sample prompt word and the presentation state of the target object corresponding to the target image can be determined. Among them, the higher the similarity, the higher the possibility that the presentation state described by the sample prompt word is the presentation state of the target object in the target image. Therefore, after the output result corresponding to each sample prompt word, the presentation state described by the sample prompt word with high similarity between the vector features of the sample prompt word represented by the output result and the vector features of the target image is determined as the presentation state of the target object corresponding to the target image, thereby determining the recognition result corresponding to the target image. Then, based on the above recognition result, the presentation state of at least one target image for the target object is determined, thereby determining the presentation state of the entire initial image for the target object to obtain the recognition result corresponding to the target object.

[0230] Optionally, in one implementation, the step of determining the presentation state of the initial image for the target object based on the recognition result corresponding to the target image may include the following steps:

[0231] Step G1: In response to the number of target images being one, the presentation state represented by the recognition result corresponding to the target image is used as the presentation state of the initial image for the target object.

[0232] In this implementation, if the number of captured target images is one, it means that there is only one area in the initial image where the target object exists. At this time, the presentation state represented by the recognition result corresponding to the target image can be directly used as the presentation state of the overall image for the target object to realize automatic recognition of the image.

[0233] In this implementation, when the number of target images is one, the recognition result corresponding to the obtained target image can be directly used as the recognition result corresponding to the entire image, thereby improving the efficiency of automatic image recognition.

[0234] Optionally, in one implementation, the step of determining the presentation state of the initial image for the target object based on the recognition result corresponding to the target image may include the following steps:

[0235] Step G2: in response to the number of target images being multiple, determining the presentation state of the initial image for the target object based on the recognition results corresponding to each target image in a predetermined analysis manner;

[0236] Among them, the predetermined analysis methods include:

[0237] If the different presentation states of the target object include a first presentation state indicating the existence of the object and a second presentation state indicating the non-existence of the object, then when there is a target image whose recognition result indicates the first presentation state among the target images, the presentation state of the initial image for the target object is determined to be the first presentation state; otherwise, the presentation state of the initial image for the target object is determined to be the second presentation state;

[0238] If the different presentation states of the target object include different presentation states regarding specified attributes, then determine the target recognition result with the highest frequency among the recognition results corresponding to each target image, and determine the presentation state of the initial image for the target object as: the presentation state represented by the target recognition result.

[0239] In this implementation, if the number of captured target images is multiple, it indicates that the initial image contains multiple areas with target objects; at this time, it is necessary to determine the recognition result corresponding to each captured target image, and thus, based on the recognition results of each target image, determine the presentation state of the overall initial image for the target object.

[0240] Considering that different presentation states of the target object may include presentation states for whether the target object exists, for example, the presence of a tattoo and the absence of a tattoo; it may also include presentation states for specified attributes of the target object, for example, the presence of a black cat and the presence of a white cat.

[0241] Therefore, a predetermined analysis method is set in advance for different presentation states of the target object, so that the presentation state of the initial image for the target object is determined based on the recognition results corresponding to each target image according to the predetermined analysis method. Specifically:

[0242] If the different presentation states of the target object include a first presentation state representing the existence of the object, and a second presentation state representing the non-existence of the object; when determining that among the target images, there is a target image whose recognition result represents the first presentation state, the presentation state of the initial image for the target object can be determined as the first presentation state; otherwise, the presentation state of the initial image for the target object can be determined as the second presentation state.

[0243] If the different presentation states of the target object include different presentation states regarding specified attributes, it is necessary to determine the target recognition result with the highest frequency in the recognition results corresponding to each target image, thereby determining the presentation state of the initial image for the target object as the presentation state represented by the target recognition result.

[0244] For example, different presentation states of the target object include different presentation states regarding tattoo colors. The target recognition result with the highest frequency indicates that the tattoo color in the image is dark. At this time, the presentation state of the initial image for the target object is determined as the tattoo color in the image is dark, and the initial image is determined to be an image including a dark tattoo.

[0245] In the present implementation, considering that when there are multiple target images, the determination of the presentation state of the target object of the overall initial image needs to be combined with the recognition results of each target image, and considering that the presentation state of the target object includes multiple presentation methods, therefore, when there are multiple target images, the presentation state of the target object of the overall initial image can be determined based on the presentation state of the target object using a corresponding analysis method, thereby improving the accuracy of automatic image recognition.

[0246] Optionally, in an embodiment, an image recognition method provided in an embodiment of the present application may further include the following steps:

[0247] Step T1: Determine whether the presentation state of the target image for the target object meets the predetermined requirements; if not, execute step T2; otherwise, execute step T3;

[0248] Step T2: performing a predetermined processing operation on the target image;

[0249] The predetermined processing operation includes: using a predetermined image processing method to cover the target object in the target image, or terminating the uploading of the target image;

[0250] Step T3: Upload the target image.

[0251] In this embodiment, in order to maintain the network environment of the network platform, a predetermined regulation is set in advance for the presentation state of the target object, and images that meet the predetermined regulation are determined as normal images, and images that do not meet the predetermined regulation are determined as illegal images. For example, an image without tattoos is a normal image, and an image with tattoos is an illegal image.

[0252] Therefore, after determining the presentation status of the target image for the target object, it is further determined whether the presentation status complies with a predetermined regulation, thereby determining whether the target image can be uploaded.

[0253] If the presentation state of the target image for the target object meets the predetermined requirements, the target image is characterized as a normal image, and the dissemination of the target image will not cause bad guidance, and the target image can be uploaded.

[0254] If the presentation state of the target image to the target object does not meet the predetermined regulations, the target image is characterized as an illegal image, and the spread of the target image may cause bad guidance. Therefore, in order to maintain the network environment of the network platform, the target image can be subjected to a predetermined processing operation.

[0255] Among them, in order to improve the image upload rate of the platform, the above-mentioned predetermined processing operation may include using a predetermined image processing method to cover the target object in the target image. In this way, when it is determined that the presentation state of the target image for the target object does not meet the predetermined regulations, the target object in the target image can be covered by using a predetermined image processing method, so that after covering the target object in the target image, an image recognition method provided by an embodiment of the present application is executed again on the covered target image until it is determined that the presentation state of the target image for the target object meets the predetermined regulations, the target image is a normal image, and the target image is uploaded.

[0256] In addition, considering that the number of images to be covered is too large, the predetermined processing operation may include terminating the uploading of the target image. In this way, when it is determined that the presentation state of the target image for the target object does not meet the predetermined regulations, the uploading of the target image is directly terminated.

[0257] Optionally, after terminating the upload of the target image, a reminder message indicating that the review has not been passed is sent to the corresponding client.

[0258] In this embodiment, target images uploaded to the network platform are classified according to predetermined regulations to ensure that the images uploaded to the network platform will not cause bad guidance and protect the network environment of the network platform.

[0259] Corresponding to the model training method provided in the above embodiment of the present application, the embodiment of the present application also provides a model training device, such as Figure 4 As shown, the device comprises:

[0260] The acquisition module 410 is used to acquire a sample image from a sample image set; wherein the sample image set includes at least one sample image corresponding to each of a plurality of designated objects, and each sample image is provided with an image label for characterizing a presentation state of the corresponding designated object;

[0261] A similarity determination module 420 is used to input the sample image into the image recognition model to be trained, and obtain an output result corresponding to each sample prompt word in a plurality of sample prompt words; wherein the image recognition model includes a target fully connected layer with a predetermined initial weight parameter and an image encoder in a pre-trained contrastive learning model; the plurality of sample prompt words include at least two sample prompt words corresponding to each designated object, and the at least two sample prompt words corresponding to each designated object include sample prompt words whose presentation states are mutually exclusive for the designated object; the output result corresponding to each sample prompt word is used to characterize the similarity between the vector feature of the sample prompt word and the vector feature of the sample image; the initial weight parameter is predetermined based on the vector features of the plurality of sample prompt words;

[0262] A loss value determination module 430, configured to determine a loss value corresponding to the image recognition model based on an output result corresponding to each sample prompt word and an image label of the sample image;

[0263] The parameter adjustment module 440 is used to adjust the current weight parameters of the target fully connected layer and the image encoder when it is determined based on the loss value that the image recognition model does not meet the model convergence conditions, and return to the step of obtaining the sample image from the sample image set.

[0264] As can be seen from the above, in the model training method provided by the embodiment of the present application, the image recognition model to be trained includes a target fully connected layer with a predetermined initial weight parameter, and an image encoder in a pre-trained contrastive learning model. Among them, the initial weight parameter of the target fully connected layer is determined in advance based on the vector features of multiple sample prompt words; the above-mentioned multiple sample prompt words include at least two sample prompt words corresponding to each specified object, and the at least two sample prompt words corresponding to each specified object include sample prompt words whose presentation states are mutually exclusive. In other words, the target fully connected layer can provide corresponding vector features for multiple specified objects. Therefore, no matter what kind of specified object is contained in the input sample image, the target fully connected layer can obtain the similarity between the vector features of each sample prompt word and the vector features of the sample image as the output result corresponding to the sample prompt word. Then, based on the output result corresponding to each sample prompt word and the image label of the sample image, the current weight parameter of the target fully connected layer and the loss value of the image encoder are determined, and the current weight parameter of the target fully connected layer and the image encoder are adjusted. In this way, by reconstructing the model structure of the image recognition model, targeted training can be achieved for multiple specified objects in one model. There is no need to train a targeted model separately for each specified object, thereby reducing the amount of computing resources occupied by the model during model training.

[0265] Optionally, in a specific implementation, the target fully connected layer includes multiple sub-fully connected layers, each sub-fully connected layer corresponds to a specified object, and each sub-fully connected layer is used to process the vector features of the sample image based on the current weight parameters of the sub-fully connected layer to obtain the similarity between the vector features of each sample prompt word corresponding to the specified object and the vector features of the sample image;

[0266] The initial weight parameters of each sub-fully connected layer are determined by:

[0267] The vector feature of each sample prompt word of the sub-fully connected layer corresponding to the specified object is obtained, and each obtained vector feature is spliced ​​into a matrix to obtain the initial weight parameter of the sub-fully connected layer.

[0268] Optionally, in a specific implementation, the loss value determination module 430 is specifically used to:

[0269] Based on the output results corresponding to each sample prompt word, determine the target confidence corresponding to the target sample prompt word; wherein the presentation state described by the target sample prompt word is the same as the presentation state of the specified object represented by the image label of the sample image; the target confidence is used to represent the presentation state described in the target sample prompt word, which is the predicted probability of the presentation state of the specified object in the sample image;

[0270] Based on the target confidence and the image label of the sample image, a loss value corresponding to the image recognition model is determined.

[0271] Optionally, in a specific implementation, the device further includes:

[0272] A sub-connected layer determination module, used for determining a target sub-fully connected layer to be adjusted in parameters before adjusting the current weight parameters of the target fully connected layer and the image encoder; wherein the target sub-fully connected layer is a sub-fully connected layer corresponding to a specified object represented by the image label of the sample image;

[0273] The parameter adjustment module 440 is specifically used for:

[0274] Parameter adjustment is performed on current weight parameters of the target sub-fully connected layer in the target fully connected layer and the image encoder.

[0275] Optionally, in a specific implementation, the similarity determination module 420 is specifically configured to:

[0276] The sample image is input into the image encoder in the image recognition model to be trained so that the image encoder generates vector features of the sample image, and the vector features of the sample image are transmitted to the target fully connected layer so as to process the vector features of the sample image based on the current weight parameters of the target fully connected layer to obtain the output result corresponding to each sample prompt word.

[0277] Corresponding to the image recognition method provided in the above embodiment of the present application, the embodiment of the present application also provides an image recognition device, such as Figure 5 As shown, the device comprises:

[0278] The acquisition module 510 is used to acquire a target image; wherein the target image is an image used to identify a target object;

[0279] The similarity determination module 520 is used to input the target image into the trained image recognition model to obtain an output result corresponding to each sample prompt word in each sample prompt word; wherein the image recognition model is trained based on a model training method provided in the above-mentioned embodiment of the present application; the sample prompt words are prompt words used to train the image recognition model; the target object belongs to one of multiple specified objects; the output result corresponding to each sample prompt word is used to characterize the similarity between the vector feature of the sample prompt word and the vector feature of the target image;

[0280] The recognition module 530 is used to determine the presentation status of the target image with respect to the target object based on the output result corresponding to each sample prompt word.

[0281] As can be seen from the above, in the image recognition method provided by the embodiment of the present application, the image recognition model used is obtained by training the training samples for multiple specified objects, and the trained image recognition model can perform image recognition on multiple different specified objects. Therefore, when the target object in the input target image is any one of the multiple specified objects, the image recognition model can recognize the target image, thereby achieving the purpose of using one image recognition model to perform image recognition on multiple different specified objects and reducing the amount of computing resources occupied by the model during model inference.

[0282] Furthermore, the above-mentioned image recognition model is trained based on a model training method provided in an embodiment of the present application, so the sample prompt words when the image recognition model performs image recognition are the prompt words obtained by training the image recognition model. Therefore, when the image recognition model is used for image recognition, there is no need to input prompt words, and only the target image can be input to realize automatic recognition of the image.

[0283] Optionally, in a specific implementation, the identification module 530 includes:

[0284] A confidence determination submodule is used to determine the confidence corresponding to each sample prompt word based on the output result corresponding to the sample prompt word; wherein the confidence corresponding to the sample prompt word is used to characterize the presentation state described in the sample prompt word, which is the probability of the presentation state of the target object corresponding to the target image;

[0285] The analysis submodule is used to analyze the presentation status of the target image with respect to the target object based on the confidence level corresponding to each sample prompt word.

[0286] Optionally, in a specific implementation, the analysis submodule is specifically used to:

[0287] From the confidence levels corresponding to each sample prompt word, select the highest confidence level;

[0288] If the selected confidence level is greater than a predetermined threshold, the presentation state of the target image for the target object is determined to be: the presentation state described by the sample prompt word corresponding to the selected confidence level.

[0289] The present application also provides an electronic device, such as Figure 6 As shown, it includes a processor 601, a communication interface 602, a memory 603 and a communication bus 604, wherein the processor 601, the communication interface 602, and the memory 603 communicate with each other through the communication bus 604.

[0290] Memory 603, used for storing computer programs;

[0291] The processor 601 is used to implement the model training method and / or the image recognition method described in any of the above embodiments when executing the program stored in the memory 603.

[0292] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0293] The communication interface is used for communication between the above terminal and other devices.

[0294] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0295] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0296] In another embodiment provided in the present application, a computer-readable storage medium is also provided, in which a computer program is stored. When the computer program is executed by a processor, the model training method and / or the image recognition method described in any one of the above embodiments are implemented.

[0297] In another embodiment provided in the present application, a computer program product comprising instructions is also provided, which, when executed on a computer, enables the computer to execute the model training method and / or the image recognition method described in any one of the above embodiments.

[0298] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive Solid State Disk (SSD)), etc.

[0299] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0300] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, electronic device embodiment, computer-readable storage medium embodiment, and computer program product embodiment, since they are basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0301] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A model training method, characterized in that: The method comprises: Acquire a sample image from a sample image set; wherein the sample image set includes at least one sample image corresponding to each of a plurality of designated objects, and each sample image is provided with an image label for characterizing a presentation state of the corresponding designated object; The sample image is input into the image recognition model to be trained to obtain an output result corresponding to each sample prompt word in a plurality of sample prompt words; wherein the image recognition model includes a target fully connected layer with a predetermined initial weight parameter and an image encoder in a pre-trained contrastive learning model; the plurality of sample prompt words include at least two sample prompt words corresponding to each designated object, and the at least two sample prompt words corresponding to each designated object include sample prompt words whose presentation states are mutually exclusive for the designated object; the output result corresponding to each sample prompt word is used to characterize the similarity between the vector feature of the sample prompt word and the vector feature of the sample image; the initial weight parameter is predetermined based on the vector features of the plurality of sample prompt words; Determine a loss value corresponding to the image recognition model based on the output result corresponding to each sample prompt word and the image label of the sample image; When it is determined based on the loss value that the image recognition model does not meet the model convergence condition, the current weight parameters of the target fully connected layer and the image encoder are adjusted, and the process returns to the step of obtaining the sample image from the sample image set.

2. The method according to claim 1, characterized in that The target fully connected layer includes a plurality of sub-fully connected layers, each of which corresponds to a specified object, and each of which is used to process the vector features of the sample image based on the current weight parameters of the sub-fully connected layer to obtain the similarity between the vector features of each sample prompt word corresponding to the specified object and the vector features of the sample image; The initial weight parameters of each sub-fully connected layer are determined by: The vector feature of each sample prompt word of the sub-fully connected layer corresponding to the specified object is obtained, and each obtained vector feature is spliced ​​into a matrix to obtain the initial weight parameter of the sub-fully connected layer.

3. The method according to claim 2, characterized in that The determining the loss value corresponding to the image recognition model based on the output result corresponding to each sample prompt word and the image label of the sample image includes: Based on the output results corresponding to each sample prompt word, determine the target confidence corresponding to the target sample prompt word; wherein the presentation state described by the target sample prompt word is the same as the presentation state of the specified object represented by the image label of the sample image; the target confidence is used to represent the presentation state described in the target sample prompt word, which is the predicted probability of the presentation state of the specified object in the sample image; Based on the target confidence and the image label of the sample image, a loss value corresponding to the image recognition model is determined.

4. The method according to claim 2, characterized in that: Before adjusting the current weight parameters of the target fully connected layer and the parameters of the image encoder, the method further includes: Determine a target sub-fully connected layer for which parameters are to be adjusted; wherein the target sub-fully connected layer is a sub-fully connected layer corresponding to a specified object represented by an image label of the sample image; The step of adjusting the current weight parameters of the target fully connected layer and the image encoder comprises: Parameter adjustment is performed on current weight parameters of the target sub-fully connected layer in the target fully connected layer and the image encoder.

5. The method according to claim 1, characterized in that The step of inputting the sample image into the image recognition model to be trained to obtain an output result corresponding to each sample prompt word in the plurality of sample prompt words includes: The sample image is input into the image encoder in the image recognition model to be trained so that the image encoder generates vector features of the sample image, and the vector features of the sample image are transmitted to the target fully connected layer so as to process the vector features of the sample image based on the current weight parameters of the target fully connected layer to obtain the output result corresponding to each sample prompt word.

6. An image recognition method, characterized in that: The method comprises: Acquire a target image; wherein the target image is an image used to identify a target object; The target image is input into a trained image recognition model to obtain an output result corresponding to each sample prompt word; wherein the image recognition model is trained based on the method described in any one of claims 1 to 5; the sample prompt words are prompt words used to train the image recognition model; the target object belongs to one of a plurality of specified objects; the output result corresponding to each sample prompt word is used to characterize the similarity between the vector feature of the sample prompt word and the vector feature of the target image; Based on the output result corresponding to each sample prompt word, a presentation state of the target image for the target object is determined.

7. The method according to claim 6, characterized in that The determining, based on the output result corresponding to each sample prompt word, a presentation state of the target image for the target object, includes: Based on the output result corresponding to each sample prompt word, determine the confidence corresponding to the sample prompt word; wherein the confidence corresponding to the sample prompt word is used to characterize the presentation state described in the sample prompt word, which is the probability of the presentation state of the target object corresponding to the target image; Based on the confidence level corresponding to each sample prompt word, the presentation status of the target image with respect to the target object is analyzed.

8. The method according to claim 7, characterized in that The analyzing the presentation status of the target image to the target object based on the confidence level corresponding to each sample prompt word includes: From the confidence levels corresponding to each sample prompt word, select the highest confidence level; If the selected confidence level is greater than a predetermined threshold, the presentation state of the target image for the target object is determined to be: the presentation state described by the sample prompt word corresponding to the selected confidence level.

9. A model training device, characterized in that: The device comprises: An acquisition module, used to acquire a sample image from a sample image set; wherein the sample image set includes at least one sample image corresponding to each of a plurality of designated objects, and each sample image is provided with an image label for characterizing a presentation state of the corresponding designated object; A similarity determination module is used to input the sample image into the image recognition model to be trained, and obtain an output result corresponding to each sample prompt word in a plurality of sample prompt words; wherein the image recognition model includes a target fully connected layer with a predetermined initial weight parameter and an image encoder in a pre-trained contrastive learning model; the plurality of sample prompt words include at least two sample prompt words corresponding to each designated object, and the at least two sample prompt words corresponding to each designated object include sample prompt words whose presentation states are mutually exclusive for the designated object; the output result corresponding to each sample prompt word is used to characterize the similarity between the vector feature of the sample prompt word and the vector feature of the sample image; the initial weight parameter is predetermined based on the vector features of the plurality of sample prompt words; A loss value determination module, used to determine the loss value corresponding to the image recognition model based on the output result corresponding to each sample prompt word and the image label of the sample image; A parameter adjustment module is used to adjust the current weight parameters of the target fully connected layer and the image encoder when it is judged that the image recognition model does not meet the model convergence conditions based on the loss value, and return to the step of obtaining the sample image from the sample image set.

10. An image recognition device, characterized in that: The device comprises: An acquisition module, used for acquiring a target image; wherein the target image is an image used for identifying a target object; A similarity determination module, used for inputting the target image into a trained image recognition model to obtain an output result corresponding to each sample prompt word; wherein the image recognition model is trained based on the method described in any one of claims 1 to 5; the sample prompt words are prompt words used to train the image recognition model; the target object belongs to one of a plurality of specified objects; the output result corresponding to each sample prompt word is used to characterize the similarity between the vector feature of the sample prompt word and the vector feature of the target image; The recognition module is used to determine the presentation status of the target image to the target object based on the output result corresponding to each sample prompt word.

11. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1-5 and / or the method steps described in any one of claims 6-8 when executing a program stored in a memory.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1-5 and / or the method steps described in any one of claims 6-8 are implemented.

Images