Image recognition method and device and electronic equipment

Abstract

The embodiment of the invention provides an image recognition method and device and electronic equipment, and relates to the technical field of image recognition, and the method comprises the steps: obtaining a target image; inputting the target image into a to-be-utilized image recognition model to obtain an output result corresponding to each predetermined prompt word in the predetermined prompt words; wherein the to-be-utilized image recognition model is obtained by additionally arranging a similarity calculation module in a pre-trained target image recognition model for recognizing a specified object; the output result is used for representing the similarity between the second modal feature of the predetermined prompt word and the first modal feature of the target image; and determining a presentation state of the target image for the target object based on the output result corresponding to each predetermined prompt word. Therefore, the application flexibility of the image recognition model can be improved through the scheme.

Description

Technical Field

[0001] The present application relates to the field of image recognition technology, and in particular to an image recognition method, device and electronic equipment. Background Art

[0002] In order to maintain the network environment of the network platform, images or videos uploaded to the platform need to be reviewed to avoid the dissemination of images or videos that may cause bad guidance through the platform, such as the dissemination of images of objects such as tattoos and smoke. Usually, images are recognized based on pre-trained image recognition models, that is, the presentation state of the image for the object is recognized, such as whether there is an object in the image.

[0003] In the related art, when it is necessary to identify the presentation state of a specified object, a pre-trained image recognition model is usually trained specifically using sample prompt words and sample images related to the specified object. This targeted training enables the trained image recognition model to identify the presentation state of the specified object in the input image based on the output result of the image recognition model without the need to input prompt words in the subsequent application process. The output result of the image recognition model is the similarity between the features of each sample prompt word and the features of the input image.

[0004] However, this will cause the image recognition model to be unable to recognize the presentation status of objects other than the specified object, thereby reducing the application flexibility of the image recognition model.

[0005] Based on this, how to improve the application flexibility of the image recognition model while ensuring the recognition accuracy of the image recognition model for images containing specified objects has become a technical problem that needs to be solved urgently. Summary of the invention

[0006] The purpose of the embodiments of the present application is to provide an image recognition method, device and electronic device to improve the application flexibility of the image recognition model while ensuring the recognition accuracy of the image recognition model for images containing specified objects. The specific technical solution is as follows:

[0007] In a first aspect provided by an embodiment of the present application, first, an image recognition method is provided, the method comprising:

[0008] Acquire a target image; wherein the target image is an image used to identify a target object;

[0009] The target image is input into the image recognition model to be used, and an output result corresponding to each predetermined prompt word in each predetermined prompt word is obtained; wherein the image recognition model to be used is obtained by adding a similarity calculation module to the target image recognition model that has been pre-trained for identifying the specified object, each predetermined prompt word includes at least one other prompt word and a sample prompt word used to train the target image recognition model, the other prompt word is an image description text that uses the presentation state of other objects other than the specified object as the description content, and the other objects and the specified objects include the target object; the similarity calculation module is used to generate an output result corresponding to each other prompt word based on the second modal feature of each other prompt word and the first modal feature of the target image generated by the image encoder of the target image recognition model; the output result is used to characterize the similarity between the second modal feature of the predetermined prompt word and the first modal feature;

[0010] Based on the output result corresponding to each predetermined prompt word, a presentation state of the target image for the target object is determined.

[0011] In a second aspect provided by an embodiment of the present application, an image recognition device is also provided, the device comprising:

[0012] An image acquisition module, used to acquire a target image; wherein the target image is an image used to identify a target object;

[0013] A similarity determination module is used to input the target image into the image recognition model to be used, and obtain an output result corresponding to each predetermined prompt word in each predetermined prompt word; wherein the image recognition model to be used is obtained by adding a similarity calculation module to the target image recognition model that has been pre-trained for identifying the specified object; each predetermined prompt word includes at least one other prompt word and a sample prompt word used to train the target image recognition model, and the other prompt word is an image description text that uses the presentation state of other objects other than the specified object as the description content, and the other objects and the specified objects include the target object; the similarity calculation module is used to generate an output result corresponding to each other prompt word based on the second modal feature of each other prompt word and the first modal feature of the target image generated by the image encoder of the target image recognition model; the output result is used to characterize the similarity between the second modal feature of the predetermined prompt word and the first modal feature;

[0014] The presentation state determination module is used to determine the presentation state of the target image for the target object based on the output result corresponding to each predetermined prompt word.

[0015] In the third aspect provided by the embodiment of the present application, there is also provided an electronic device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; the memory is used to store computer programs; the processor is used to implement any image recognition method provided in the first aspect when executing the program stored in the memory.

[0016] In another aspect provided by the embodiments of the present application, a computer-readable storage medium is provided, wherein a computer program is stored in the computer storage medium, and when the computer program is executed by a processor, any image recognition method provided in the first aspect is implemented.

[0017] In another aspect provided by the embodiments of the present application, a computer program product comprising instructions is also provided, which, when executed on a computer, enables the computer to execute any image recognition method provided in the first aspect above.

[0018] By applying the solution provided by the embodiment of the present application, before image recognition is performed, sample prompt words are used in advance to train a target image recognition model for identifying a specified object, and then a similarity calculation module is added to the target image recognition model. The similarity calculation module is used to generate an output result corresponding to each other prompt word based on the second modal feature of each other prompt word and the first modal feature of the target image generated by the image encoder of the target image recognition model. In other words, the image recognition model to be used can use each predetermined prompt word including at least one other prompt word and sample prompt words used to train the target image recognition model to recognize the input target image. Among them, the other prompt words are image description texts that use the presentation status of other objects other than the specified object as the description content, and the other objects and the specified objects include the target object in the target image.

[0019] In this way, in the process of image recognition, after obtaining the image used to identify the target object as the target image, the obtained target image can be input into the image recognition model to be used to obtain the output result corresponding to each of the predetermined prompt words. The output result is used to characterize the similarity between the second modal feature and the first modal feature of the predetermined prompt word. Therefore, based on the output result corresponding to each predetermined prompt word, the presentation state of the target image for the target object can be determined to achieve recognition of the target image.

[0020] Based on this, the solution provided in the embodiment of the present application is applied. By adding a similarity calculation module for determining the similarity of other objects to the target image recognition model pre-trained for the specified object, it is possible to achieve the image recognition accuracy of the image recognition model for images containing the specified object while expanding the image recognition capability of the image recognition model for images other than the specified object, thereby improving the application flexibility of the image recognition model. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0022] Figure 1 A schematic diagram of the classification process of the existing CLIP model;

[0023] Figure 2 A schematic diagram of the process of the first image recognition method provided in the embodiment of the present application;

[0024] Figure 3 A schematic diagram of the structure of an image recognition model to be used in an embodiment of the present application;

[0025] Figure 4 A schematic diagram of a second image recognition method provided in an embodiment of the present application;

[0026] Figure 5 A schematic diagram of a training process of a target image recognition model provided in an embodiment of the present application;

[0027] Figure 6 A schematic diagram of the structure of a model structure of an image recognition model provided in an embodiment of the present application;

[0028] Figure 7 A schematic diagram of the structure of an image recognition device provided in an embodiment of the present application;

[0029] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0031] In order to better understand the present application, the professional terms involved in the embodiments of the present application are first introduced below.

[0032] Text-image pre-training model based on contrastive learning: It is a model pre-trained on 400 million pairs of image-text matching pairs, and has strong generalization ability. By giving the corresponding text prompt words to the pre-trained model, the model can obtain the similarity between the features of the text prompt words and the features of the corresponding image as the output result of the model. Specifically, the pre-training model recognizes the uploaded image, that is, recognizes the presentation state of the image for the object, such as whether there is an object in the image (whether there is a cat in the image), or whether there is an object in a certain state in the image (whether there is a white cat in the image), etc. In the recognition process, the pre-training model can classify and recognize the image based on the pre-set prompt words. The above-mentioned text-image pre-training model based on contrastive learning can be a CLIP (Contrastive Language-Image Pre-Training, contrastive language-image pre-training) model, a BLIP (Bootstrapping Language-Image Pre-training, bootstrap language image pre-training) model or a SigLip (Sigmoid Loss for Language-Image Pre-training, language-image pre-training) model, etc., which is not specifically limited in the embodiments of the present application.

[0033] Taking the above-mentioned pre-trained model as a CLIP model as an example, when the CLIP model is used to classify an image, the CLIP model can be given two prompt words, "people with tattoos" and "people without tattoos", to classify the image by letting the CLIP model determine which prompt word has a closer similarity.

[0034] Combine the following Figure 1 This paper introduces the classification process of the CLIP model. Figure 1 As shown in the figure, plane, car, dog and bird are all target objects (or specified objects, other objects) in the embodiment of the present application, and "a photo of a {object}" is the prompt word in the embodiment of the present application. The byte position of object is the byte position occupied by the target object (or specified object, other objects) in the prompt word.

[0035] After receiving the prompt word, the Text Encoder of the CLIP model can convert the text multimodal features corresponding to the prompt word, including T1, T2, T3...T N That is to say, after receiving the prompt word, the text encoder of the CLIP model extracts the text features of the prompt word and calls its own text multimodal network to convert the extracted text features into text multimodal features.

[0036] After receiving the image, the Image Encoder of the CLIP model can convert and obtain the image multimodal features I1 corresponding to the above image. In other words, after receiving the image, the image encoder of the CLIP model extracts the image features of the image, and calls its own image multimodal network to convert the extracted image features into image multimodal features.

[0037] The CLIP model can then associate the obtained image multimodal feature I1 with the text multimodal features T1, T2, T3, ... T corresponding to the prompt word. N Perform feature similarity matching to obtain the similarities I1·T1, I1·T2, I1·T3, I1·T N Among them, I1·T3 has the highest similarity, and the object in the prompt word corresponding to I1·T3 is dog. Therefore, it can be determined that the object in the image input to the Image Encoder is dog, and the output result of the model is "a photo of a dog".

[0038] Among them, the above-mentioned image multimodal feature is the first modal feature (or first sample modal feature) in the embodiment of the present application; the above-mentioned text multimodal feature is the second modal feature in the embodiment of the present application.

[0039] In order to maintain the network environment of the network platform, it is necessary to review the images or videos uploaded to the platform to avoid the dissemination of images or videos that may cause bad guidance through the platform, for example, the dissemination of images including tattoos, smoke and other objects.

[0040] Therefore, each image or video frame uploaded to the platform can be manually reviewed. However, for online review services, manual review is very costly. Therefore, in related technologies, a pre-trained image recognition model is used to identify images or videos that need to be reviewed, and the image recognition model is used to automatically filter images or videos that do not contain the above objects.

[0041] Exemplarily, the pre-trained CLIP model is used to identify the image, that is, to identify the presentation state of the image for the object. When it is necessary to identify the presentation state of a specified object, the pre-trained image recognition model is usually trained specifically using sample prompt words and sample images related to the specified object. This targeted training enables the trained image recognition model to identify the presentation state of the specified object for the input image based on the output results of the image recognition model without the need to input prompt words in subsequent applications.

[0042] The output result of the CLIP model is the similarity between the sample prompt words used in model training and the input image. The similarity between any prompt word and the image represents the similarity between the features of the prompt word and the features of the image. In other words, the output result of the trained image recognition model is the presentation state of the image for the specified object, and the presentation state belongs to the classification result of a prompt word in the sample prompt words, that is, each sample prompt word used to train the image recognition model provides a classification standard for the image recognition of the image recognition model, and the classification of the input image is based on the sample prompt words.

[0043] However, through the above-mentioned targeted training process, the trained image recognition model can only learn features related to the presentation state of the specified object, thereby losing the ability to recognize the presentation state of other objects except the specified object, resulting in the image recognition model being unable to recognize the presentation state of other objects except the specified object, thereby reducing the application flexibility of the image recognition model.

[0044] Based on this, how to improve the application flexibility of the image recognition model while ensuring the recognition accuracy of the image recognition model for images containing specified objects has become a technical problem that needs to be solved urgently.

[0045] In order to solve the above technical problems, the embodiments of the present application provide an image recognition method, device and electronic device.

[0046] Among them, the method can be applied to various application scenarios for automatic recognition of video frames in images or videos. For example, automatic recognition of tattoos in images, and automatic recognition of black dogs in video frames. In addition, the method can be applied to various electronic devices such as laptops, tablets, and desktop computers, hereinafter referred to as electronic devices. Based on this, the embodiments of the present application do not limit the application scenarios and execution subjects of the method.

[0047] An image recognition method provided in an embodiment of the present application may include the following steps:

[0048] Acquire a target image; wherein the target image is an image used to identify a target object;

[0049] The target image is input into the image recognition model to be used, and an output result corresponding to each predetermined prompt word in each predetermined prompt word is obtained; wherein the image recognition model to be used is obtained by adding a similarity calculation module to the target image recognition model that has been pre-trained for identifying the specified object; each predetermined prompt word includes at least one other prompt word and a sample prompt word for training the target image recognition model, and the other prompt word is an image description text that uses the presentation state of other objects other than the specified object as the description content, and the other objects and the specified objects include the target object; the similarity calculation module is used to generate an output result corresponding to each other prompt word based on the second modal feature of each other prompt word and the first modal feature of the target image generated by the image encoder of the target image recognition model; the output result is used to characterize the similarity between the second modal feature of the predetermined prompt word and the first modal feature;

[0050] Based on the output result corresponding to each predetermined prompt word, a presentation state of the target image for the target object is determined.

[0051] As can be seen from the above, by applying the solution provided by the embodiment of the present application, before image recognition is performed, sample prompt words are used in advance to train a target image recognition model for identifying a specified object, and then a similarity calculation module is added to the target image recognition model. The similarity calculation module is used to generate an output result corresponding to each other prompt word based on the second modal feature of each other prompt word and the first modal feature of the target image generated by the image encoder of the target image recognition model. In other words, the image recognition model to be used can use each predetermined prompt word including at least one other prompt word and sample prompt words used to train the target image recognition model to recognize the input target image. Among them, the other prompt words are image description texts that use the presentation status of other objects other than the specified object as the description content, and the other objects and the specified objects include the target object in the target image.

[0052] In this way, in the process of image recognition, after obtaining the image used to identify the target object as the target image, the obtained target image can be input into the image recognition model to be used to obtain the output result corresponding to each of the predetermined prompt words. The output result is used to characterize the similarity between the second modal feature and the first modal feature of the predetermined prompt word. Therefore, based on the output result corresponding to each predetermined prompt word, the presentation state of the target image for the target object can be determined to achieve recognition of the target image.

[0053] Based on this, the solution provided in the embodiment of the present application is applied. By adding a similarity calculation module for determining the similarity of other objects to the target image recognition model pre-trained for the specified object, it is possible to achieve the image recognition accuracy of the image recognition model for images containing the specified object while expanding the image recognition capability of the image recognition model for images other than the specified object, thereby improving the application flexibility of the image recognition model.

[0054] An image recognition method provided in an embodiment of the present application is described in detail below in conjunction with the accompanying drawings.

[0055] Figure 2 A flowchart of an image recognition method provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the method comprises the following steps:

[0056] S201: Acquire a target image;

[0057] The target image is an image used to identify the target object.

[0058] In the present application, when performing image recognition, an image used to identify a target object can be obtained as a target image, thereby executing an image recognition method provided by an embodiment of the present application on the target image.

[0059] Optionally, since the image recognition model has a size restriction range for the input image size, after acquiring the target image, the image size of the target image can be scaled to a predetermined size according to the size restriction range of the image recognition model to be input. The predetermined size is within the size restriction range of the image recognition model to be input.

[0060] It should be noted that the acquisition, storage, use, processing, transmission, provision and disclosure of images involved in the technical solution of this application are all carried out with the authorization of the user. In addition, the images of the human body involved in this application are not images of the human body of a specific user and do not reflect the personal information of a specific user.

[0061] S202: inputting the target image into the image recognition model to be used, and obtaining an output result corresponding to each of the predetermined prompt words;

[0062] Among them, the image recognition model to be used is obtained by adding a similarity calculation module to a target image recognition model that has been pre-trained for identifying a specified object; each predetermined prompt word includes at least one other prompt word and a sample prompt word used to train the target image recognition model, and the other prompt word is an image description text that uses the presentation state of other objects other than the specified object as a description content, and the other objects and the specified objects include the target object; the similarity calculation module is used to generate an output result corresponding to each other prompt word based on the second modal feature of each other prompt word and the first modal feature of the target image generated by the image encoder of the target image recognition model; the output result is used to characterize the similarity between the second modal feature and the first modal feature of the predetermined prompt word.

[0063] In the present application, the image recognition model to be used is obtained by adding a similarity calculation module to a target image recognition model that has been pre-trained for recognizing a designated object.

[0064] Among them, the target image recognition model is trained using sample images related to the specified object and sample prompt words of the image description text with the presentation state of the specified object as the description content. Therefore, when the target object is the specified object, the target image can be identified by comparing the similarity between the second modal feature of each sample prompt word and the first modal feature of the target image.

[0065] Exemplarily, the target image recognition model can be composed of an image encoder and a fully connected layer. For example, the target image recognition model can be composed of the image encoder in the above CLIP model, and the fully connected layer obtained by splicing the second modality features of each sample prompt word extracted based on the text encoder in the CLIP model. In this way, the first modality features of the target image can be generated by the above image encoder, and the second modality features of each sample prompt word can be obtained by the weight parameters of the above fully connected layer, thereby determining the similarity between the second modality features of each sample prompt word and the above first modality features, and obtaining the output result corresponding to each sample prompt word.

[0066] It should be noted that the image recognition model to be used is based on the structure of the target image recognition model, with the addition of a similarity calculation module, that is, the structure of the image recognition model to be used is an image encoder, a fully connected layer and a similarity calculation module. Among them, the similarity calculation module is a module parallel to the fully connected layer and connected to the image encoder.

[0067] In this way, after the first modal features of the target image are generated by the image encoder, the first modal features can be transmitted to the fully connected layer and the similarity calculation module, so as to use the fully connected layer to determine the feature similarity between each sample prompt word and the target image. At the same time, the similarity calculation module is used to determine the similarity between each other prompt word and the target image, so as to obtain the output result corresponding to each of the predetermined prompt words.

[0068] For ease of reading, the specific implementation method of training the target image recognition model is detailed in steps S501-S504 below and will not be repeated here.

[0069] The similarity calculation module is used to generate an output result corresponding to each other prompt word based on the second modal feature of each other prompt word and the first modal feature of the target image generated by the image encoder of the target image recognition model. The other prompt word is an image description text with the presentation state of other objects other than the specified object as the description content.

[0070] That is to say, before performing image recognition, the similarity calculation module stores the second modal features of each other prompt word related to other objects. Therefore, when the target object is another object, the similarity calculation module can be used to determine the similarity between the second modal features of each other prompt word and the first modal features of the target image to identify the target image.

[0071] It can be determined that for the entire image recognition model to be used, the prompt words used when performing image recognition are fixed, that is, before performing image recognition, each predetermined prompt word in the image recognition model to be used contains at least one other prompt word and a sample prompt word used to train the target image recognition model, and there is a second modal feature corresponding to each predetermined prompt word in the image recognition model to be used. In this way, when performing image recognition on the target image, there is no need to input the prompt word, only the target object contained belongs to other objects, or the target image belongs to a specified object can be input.

[0072] Therefore, after acquiring the target image, the target image can be input into the image recognition model to be used. After the image encoder of the image recognition model to be used generates the first modal features of the target image, the first modal features can be transmitted to the target image recognition model and the similarity calculation module respectively. In this way, the similarity between the second modal features and the first modal features of each sample prompt word of the target image recognition model itself is determined; at the same time, the similarity between the second modal features and the first modal features of each other prompt word of the similarity calculation module is determined through the second modal features and the first modal features of each other prompt word, thereby obtaining the output result corresponding to each predetermined prompt word in each predetermined prompt word.

[0073] The number of the above-mentioned designated prompt words can be one or more. When the number of the above-mentioned designated prompt words is more than one, the multiple designated prompt words can be used to describe the same other object other than the designated object; the multiple designated prompt words can also be used to describe multiple other objects other than the designated object, and this embodiment of the application does not make specific limitations on this.

[0074] S203: Determine the presentation status of the target image for the target object based on the output result corresponding to each predetermined prompt word.

[0075] In the present application, after the output result corresponding to each predetermined prompt word is obtained, the degree of similarity between the presentation state described by the predetermined prompt word and the presentation state of the target image for the target object can be determined. The higher the degree of similarity, the higher the possibility that the presentation state described by the prompt word is the presentation state of the target object in the target image. Therefore, after determining the output result corresponding to each predetermined prompt word, the presentation state of the target image for the target object can be determined based on the determined output result.

[0076] Among them, the presentation state of the above-mentioned target object can be understood as whether the target object exists in the target image, for example, whether there is smoke in the image, or whether there is a cat in the image, etc.; it can also be understood as whether there is a target object in a certain state or belonging to a certain attribute in the target image, for example, whether there is smoke in a burning state in the image, or whether there is a white cat in the image, etc., which are all reasonable.

[0077] As can be seen from the above, by applying the solution provided by the embodiment of the present application, before image recognition is performed, sample prompt words are used in advance to train a target image recognition model for identifying a specified object, and then a similarity calculation module is added to the target image recognition model. The similarity calculation module is used to generate an output result corresponding to each other prompt word based on the second modal feature of each other prompt word and the first modal feature of the target image generated by the image encoder of the target image recognition model. In other words, the image recognition model to be used can use each predetermined prompt word including at least one other prompt word and sample prompt words used to train the target image recognition model to recognize the input target image. Among them, the other prompt words are image description texts that use the presentation status of other objects other than the specified object as the description content, and the other objects and the specified objects include the target object in the target image.

[0078] In this way, in the process of image recognition, after obtaining the image used to identify the target object as the target image, the obtained target image can be input into the image recognition model to be used to obtain the output result corresponding to each of the predetermined prompt words. The output result is used to characterize the similarity between the second modal feature and the first modal feature of the predetermined prompt word. Therefore, based on the output result corresponding to each predetermined prompt word, the presentation state of the target image for the target object can be determined to achieve recognition of the target image.

[0079] Based on this, the solution provided in the embodiment of the present application is applied. By adding a similarity calculation module for determining the similarity of other objects to the target image recognition model pre-trained for the specified object, it is possible to achieve the image recognition accuracy of the image recognition model for images containing the specified object while expanding the image recognition capability of the image recognition model for images other than the specified object, thereby improving the application flexibility of the image recognition model.

[0080] Optionally, in one embodiment, the first modal feature of the target image is a vector feature of a predetermined dimension; and the method for determining the second modal feature of each other prompt word includes:

[0081] The text features of the other prompt words are extracted, and based on the text features of the other prompt words, vector features of a predetermined dimension of the other prompt words are generated to obtain the second modal features of the other prompt words.

[0082] In this embodiment, taking into account different data categories, such as text data, image data, etc., whose corresponding vector features are in different dimensions, in order to facilitate the subsequent determination of the similarity between the second modal feature of each predetermined prompt word and the first modal feature of the target image, the vector features corresponding to data of different data categories can be unified into the same dimension.

[0083] Therefore, after acquiring the target image, the image features of the target image can be extracted, and based on the extracted image features, vector features of a predetermined dimension of the target image can be generated to obtain the first modal features of the target image.

[0084] Similarly, before image recognition is performed, for each other prompt word, the text features of the other prompt word are extracted, and based on the text features of the other prompt word, the vector features of the other prompt word with a predetermined dimension are generated to obtain the second modal features of the other prompt word. In other words, the second modal features of each other prompt word are determined after the target image recognition model is trained and before image recognition is performed.

[0085] Usually, for a text-image based contrastive learning model also used for image recognition, the encoder parameters of the image encoder of the contrastive learning model are feature matched with the encoder parameters of the text encoder of the contrastive learning model. In the present application, the trained target image recognition model is obtained by adjusting the structure of the contrastive learning model in the prior art and then training it, that is, the target image recognition model includes two parts: an image encoder and a fully connected layer.

[0086] When the target image recognition model is trained, the adjusted parameters are the weight parameters of the newly added fully connected layer in the target image recognition model, and do not involve the encoder parameters corresponding to the image encoder and text encoder of the initial contrastive learning model itself before training the target image recognition model. In other words, the encoder parameters of the image encoder in the target image recognition model are still feature-matched with the encoder parameters of the text encoder. Therefore, optionally, the text encoder in the target image recognition model is used to determine the second modal features of each other prompt word.

[0087] Since the trained target image recognition model does not need to input prompt words, there is no need to call a text encoder to process the prompt words. Therefore, in order to reduce the space resources occupied by the model structure, there is no text encoder in the trained target image recognition model. Then, optionally, the second modal feature of each other prompt word is determined using the text encoder in the initial recognition model corresponding to the target image recognition model. That is, the second modal feature of each other prompt word is determined using the text encoder in the initial contrastive learning model before the target image recognition model is trained. Among them, the vector features of the image features and text features are represented by the feature matrix, and the embodiments of the present application do not make specific limitations.

[0088] In this embodiment, by converting the dimensions of the text features of other prompt words, vector features of the other prompt words in the same predetermined dimension as the first modal features of the target image are obtained as the second modal features of the other prompt words.

[0089] Considering that images with the same image size have the same dimensions of the vector features that can be converted, therefore, in order to ensure that each acquired target image can be converted into image features of the same dimension, and then further converted into vector features of a predetermined dimension, the target image can be scaled to a predetermined size before generating the first modal feature for the target image, so as to be used in the vector feature generation process. Based on this, optionally, after acquiring the target image, the target image is scaled to a predetermined size, and after image features are extracted from the target image of the predetermined size, the extracted image features are converted into vector features of a predetermined dimension to obtain the first modal feature.

[0090] Exemplarily, after acquiring the target image, the image size of the target image is scaled to a fixed size, for example, 224×224, or 336×336, and then the scaled target image is input into a module for generating a first modal feature to generate the first modal feature of the target image.

[0091] It should be noted that the embodiments of the present application involve a text-image pre-training model based on contrastive learning, and the main ability of this type of model is to align image features and text features in the same feature space (dimension). Therefore, in theory, only the trained model layer in the pre-training model can be used to convert image or text features to vector features, that is, the image features are converted to the first modality features through the image encoder in the pre-training model, and the text features are converted to the second modality features through the text encoder in the pre-training model. Other untrained methods cannot be used, otherwise, the obtained first modality features and second modality features will not be in the same feature space from a mathematical perspective.

[0092] Based on this, optionally, the target image is input into the image encoder of the pre-trained model so that the image encoder extracts image features of the target image and converts the extracted image features to obtain the first modal features of the target image; the prompt word is input into the text encoder of the pre-trained model so that the text encoder extracts text features of the prompt word and converts the extracted text features to obtain the second modal features of the prompt word.

[0093] In this way, since the generation method of the first modal feature and the second modal feature is pre-trained, the obtained first modal feature and the second modal feature are in the same feature space from a mathematical perspective, that is, the modal features obtained by the image encoder and text encoder using the above-mentioned pre-trained model are in the same predetermined dimension.

[0094] In this embodiment, by converting the dimension of the text feature of other prompt words, the vector feature of the text feature is converted to the same predetermined dimension as the first modal feature of the target image, and the second modal feature of the other prompt words is obtained, so as to facilitate the subsequent calculation of the similarity between the second modal feature of each other prompt word and the first modal feature of the target image, thereby improving the accuracy of the determined similarity.

[0095] Optionally, in one implementation, the similarity calculation module is a module that is parallel to the fully connected layer and connected to the image encoder.

[0096] In this implementation, the similarity calculation module in the image recognition model to be used is parallel to the fully connected layer of the target image recognition model, and the similarity calculation module and the fully connected layer are both connected to the image encoder of the target image recognition model.

[0097] For example, Figure 3 A schematic diagram of the structure of an image recognition model to be used provided in an embodiment of the present application. The target image recognition model is a recognition model trained for service A. Therefore, the weight parameters of the fully connected layer 301 of the target image recognition model are a matrix determined based on the text multimodal features of multiple sample prompt words related to service A. The similarity calculation module 302 contains text multimodal features determined by the specified prompt words for service B, text multimodal features determined by the specified prompt words for service C, and text multimodal features determined by the specified prompt words for service D.

[0098] The image is input to the image encoder in the image recognition model to be used, so that the image encoder generates the first modal feature of the image, and then the first modal feature is input in parallel to the fully connected layer 301 and the similarity calculation module 302. The fully connected layer 301 processes the first modal feature based on its own weight parameter to obtain the output result corresponding to each sample prompt word in the image for each sample prompt word, that is, the similarity between the second modal feature of each sample prompt word and the first modal feature, and then obtains the label score of the image for business A through softmax (normalization) processing.

[0099] The first modal feature is processed through the text multimodal features of business B to obtain the output result corresponding to the specified prompt word of business B, that is, the similarity between the second modal feature and the first modal feature of each specified prompt word of business B. Then, the label score of the image for business B is obtained through softmax processing.

[0100] The first modal feature is processed through the text multimodal features of business C to obtain the output result corresponding to the specified prompt word of business C, that is, the similarity between the second modal feature and the first modal feature of each specified prompt word of business C. Then, the label score of the image for business C is obtained through softmax processing.

[0101] The first modal feature is processed through the text multimodal features of business D to obtain the output result corresponding to the specified prompt word of business D, that is, the similarity between the second modal feature and the first modal feature of each specified prompt word of business D. Then, the label score of the image for business D is obtained through softmax processing.

[0102] Among them, the above label score is the confidence level in the embodiment of the present application.

[0103] Optionally, in one implementation, Figure 4 As shown, in an image recognition method provided by an embodiment of the present application, the above step S203, based on the output result corresponding to each predetermined prompt word, determines the presentation state of the target image for the target object, which may include the following steps:

[0104] S2031: Determine the confidence level corresponding to each predetermined prompt word based on the output result corresponding to the predetermined prompt word;

[0105] The confidence level corresponding to each predetermined prompt word is used to characterize the presentation state described in the predetermined prompt word, which is the probability of the presentation state of the target object corresponding to the target image;

[0106] S2032: Analyze the presentation status of the target image with respect to the target object based on the confidence level corresponding to each predetermined prompt word.

[0107] In this implementation, after determining the output result corresponding to each predetermined prompt word, the similarity between the second modal feature and the first modal feature of each predetermined prompt word can be determined, thereby further determining the confidence corresponding to the predetermined prompt word, and using the presentation state described in the predetermined prompt word as the probability of determining the presentation state of the target object corresponding to the target image. In other words, the higher the confidence corresponding to the predetermined prompt word, the higher the probability that the presentation state described in the predetermined prompt word is the presentation state of the target object corresponding to the target image, thereby analyzing the presentation state of the target image for the target object based on the confidence corresponding to each predetermined prompt word.

[0108] In this implementation, the confidence level is used to digitize the probability that the presentation state described in each predetermined prompt word is the presentation state of the target object corresponding to the target image, so as to match the target image with the corresponding predetermined prompt word, thereby improving the accuracy of the analysis of the presentation state of the target image for the target object.

[0109] Optionally, in one implementation, the above step S2032, analyzing the presentation status of the target image to the target object based on the confidence level corresponding to each predetermined prompt word, may include the following steps:

[0110] Step A1: Select the highest confidence level from the confidence levels corresponding to each predetermined prompt word as the target confidence level;

[0111] Step A2: If the target confidence is greater than a predetermined threshold, the presentation state of the target image for the target object is determined to be: the presentation state described by the predetermined prompt word corresponding to the target confidence.

[0112] In this implementation, the higher the confidence level corresponding to the predetermined prompt word, the higher the probability that the presentation state described in the predetermined prompt word is the presentation state of the target object corresponding to the target image. Therefore, after determining the confidence level corresponding to each predetermined prompt word, the highest confidence level is selected as the target confidence level.

[0113] However, considering that although the determined confidence is the highest confidence, the probability that the presentation state described in the predetermined prompt word represented by the confidence is the presentation state of the target object corresponding to the target object is still low, for example, generally speaking, the two can only be considered to be matched when the confidence is above 50%. At this time, if the highest confidence determined is 35%, although this confidence is the highest confidence among the confidences corresponding to the predetermined prompt words, when the confidence is 50% for the two to be considered to be a match, the presentation state described by the predetermined prompt word corresponding to the confidence does not actually match the presentation state of the target object to the target object.

[0114] Therefore, in order to improve the accuracy of the analysis of the presentation state of the target image to the target object, a predetermined threshold can be set for the confidence in advance, so that, by comparing the determined target confidence with the predetermined threshold, it is determined whether the presentation state described by the predetermined prompt word corresponding to the target confidence is the presentation state of the target image to the target object.

[0115] If the target confidence is greater than a predetermined threshold, then the presentation state described by the predetermined prompt word corresponding to the target confidence is the presentation state of the target image for the target object.

[0116] If the target confidence is not greater than a predetermined threshold, it is ensured that the presentation state described by the predetermined prompt word corresponding to the target confidence is not the presentation state of the target image for the target object.

[0117] In this implementation, the accuracy of analyzing the presentation status of the target image to the target object is further improved by presetting a predetermined threshold value regarding the confidence level.

[0118] Optionally, in one implementation, the above step S2031, based on the output result corresponding to each predetermined prompt word, determines the confidence level corresponding to the predetermined prompt word, may include the following steps:

[0119] Step B: normalize the output result corresponding to each predetermined prompt word to obtain the confidence level corresponding to the predetermined prompt word.

[0120] In this implementation, after obtaining the output result corresponding to each predetermined prompt word, for each predetermined prompt word, the similarity between the second modal feature and the first modal feature of the predetermined prompt word is normalized to obtain the confidence level corresponding to the predetermined prompt word.

[0121] Optionally, for each predetermined prompt word, a predetermined formula is used to calculate the confidence level corresponding to the predetermined prompt word; wherein the predetermined formula includes:

[0122]

[0123] σ(z i ) represents the confidence corresponding to the i-th predetermined prompt word about the target object, K represents the total number of predetermined prompt words about the target object, z i Characterize the feature score corresponding to the i-th predetermined prompt word, z j Characterize the feature score corresponding to the j-th predetermined prompt word.

[0124] Among them, the feature score corresponding to each of the above-mentioned predetermined prompt words is used to characterize the similarity between the predetermined prompt word and the target image, that is, the output result corresponding to each predetermined prompt word in the embodiment of the present application, that is, the similarity between the second modal feature of the predetermined prompt word and the first modal feature of the target image.

[0125] This implementation method, by normalizing the similarity between the second modal feature of each predetermined prompt word and the first modal feature of the target image to a specific interval, for example, [0, 1] or [-1, 1], etc., makes the data distribution statistically significant and the probability distribution of similarity more statistically intuitive, thereby improving the comparability of the output results corresponding to each predetermined prompt word.

[0126] Optionally, in one embodiment, Figure 5 A schematic diagram of a training process of a target image recognition model provided in an embodiment of the present application, such as Figure 5 As shown, the training method of the target image recognition model may include the following steps:

[0127] S501: Acquire a sample image;

[0128] The sample image is provided with a label for characterizing the presentation state of the specified object.

[0129] In this embodiment, since the public data set may not contain training data related to actual business scenarios, for example, the public data set does not contain training data related to tattoos (sample images and sample prompt words). Therefore, the image recognition model trained using the public data set has a low recognition accuracy rate for objects contained in images in highly targeted, actual business scenarios. Taking the designated object as a tattoo as an example, if there are no tattoo-related images and prompt words in the public data set, the image recognition model trained using the public data set has a low recognition accuracy rate for whether there is a tattoo in the image.

[0130] Therefore, an image provided with a label characterizing the presentation state of a specified object can be acquired as the original image.

[0131] It is understandable that the number of sample images acquired may be one or more, and the embodiment of the present application does not specifically limit this.

[0132] Among them, the presentation state of the above-mentioned sample object can be understood as whether a specified object exists in the sample image, for example, whether there is smoke in the image, or whether there is a cat in the image, etc.; it can also be understood as whether there is a specified object in a certain state or belonging to a certain attribute in the sample image, for example, whether there is smoke in a burning state in the image, or whether there is a white cat in the image, etc., which are all reasonable.

[0133] It should be noted that the label used to characterize the presentation state of the specified object can be a digital label or a text label, and the embodiments of the present application do not specifically limit this. For example, if the specified formation exists in the sample image, the representation of the sample image can be set to 1; if the specified object does not exist in the sample image, the label of the sample image can be set to 0.

[0134] S502: inputting the sample image into a pre-built target image recognition model to be trained, so that an image encoder in the target image recognition model generates a vector feature of a predetermined dimension of the sample image, obtains a first sample modal feature of the sample image, and transmits the first sample modal feature to a fully connected layer in the target image recognition model, so as to process the first sample modal feature based on a weight parameter of the fully connected layer, and obtain an output result corresponding to each sample prompt word in each sample prompt word;

[0135] The output result corresponding to each sample prompt word is used to represent the similarity between the second modality feature of the sample prompt word and the first sample modality feature of the sample image.

[0136] In this embodiment, considering that the contrastive learning model used for image recognition in the prior art, when performing image recognition, calls the text encoder to process the input prompt word respectively to obtain the text multimodal feature as the second modal feature of the prompt word; at the same time, calls the image encoder to process the input image to obtain the image multimodal feature as the first modal feature of the image, thereby realizing the recognition of the image by determining the similarity between the first modal feature and the second modal feature.

[0137] That is to say, the key point of the model used for image recognition is to obtain the text multimodal features of the prompt word and the image multimodal features of the image, and to determine the similarity between the text multimodal features and the image multimodal features. At this time, if the model is trained for a specified object, it can be understood that the prompt word related to the specified object can be fixed and the vector features of the predetermined dimension of the generated prompt word are also fixed within a certain range.

[0138] Furthermore, considering that the role of the fully connected layer in the entire model for image recognition is to process multimodal data by calculating the similarity of multimodal data; and the process of processing multimodal data in this embodiment is: by calculating the similarity between the second modal feature of each sample prompt word and the first sample modal feature of the sample image, the similarity of each sample prompt word to the sample image is determined. It can be seen that the processing process of the fully connected layer for multimodal data is the same as the processing process for multimodal data in this embodiment. Therefore, when the vector features of the predetermined dimensions of the generated sample prompt words are fixed within a certain range, the matrix determined based on the second modal features of multiple sample prompt words can be used as the weight parameters of the fully connected layer, thereby using the fully connected layer to replace the text encoder in the existing contrastive learning model, and providing the required second modal features for the image recognition process through the weight parameters of the fully connected layer.

[0139] In this way, by splicing the fully connected layer after the image encoder through data association, the internal structure of the contrastive learning model is reconstructed to obtain the target image recognition model to be trained. In addition, before the model training starts, each image description text with the presentation state of the specified object as the description content is used as each sample prompt word, and then the vector feature of the predetermined dimension used to characterize each sample prompt word is obtained as the second modal feature of the sample prompt word, so that the matrix determined based on the second modal features of multiple sample prompt words is used as the weight parameter of the fully connected layer in advance to provide the second modal features of the sample prompt words for the subsequent multimodal data processing.

[0140] Among them, the specific implementation method for determining the weight parameters of the fully connected layer is detailed in steps C1-C2 below and will not be repeated here.

[0141] After acquiring the sample image, the sample image is input into the above-mentioned pre-built target image recognition model to be trained, so that the image encoder in the target image recognition model generates a vector feature of a predetermined dimension of the sample image, and obtains the first sample modal feature of the sample image; and the first sample modal feature is transmitted to the fully connected layer in the target image recognition model, so as to process the first sample modal feature based on the weight parameter of the fully connected layer itself, that is, to calculate the similarity between the second modal feature of each sample prompt word and the first sample modal feature of the sample image, thereby obtaining the output result corresponding to each sample prompt word in each sample prompt word. Among them, the output result corresponding to each sample prompt word is characterized by a feature score, and this is not specifically limited in the embodiment of the present application.

[0142] Optionally, the cosine similarity is used to calculate the distance between the second modal feature of each sample prompt word and the first sample modal feature. If the distance between the second modal feature of the sample prompt word and the first sample modal feature is closer, the similarity between the two modal features is higher. Conversely, if the distance between the second modal feature of the sample prompt word and the first sample modal feature is farther, the similarity between the two modal features is lower.

[0143] S503: Based on the output result corresponding to each sample prompt word and the label of the sample image, determine the loss value corresponding to the target image recognition model.

[0144] In this embodiment, since the sample image is provided with a label for representing the presentation state of the specified object, that is, the presentation state of the specified object represented by the label of the sample image is the actual state presented by the specified object in the sample image. Therefore, after obtaining the output result corresponding to each sample prompt word, the loss value corresponding to the target image recognition model can be determined based on the output result and the label of the sample image.

[0145] S504: When it is determined based on the loss value that the target image recognition model does not meet the model convergence condition, the weight parameters of the fully connected layer are adjusted based on the loss value, and the process returns to step S501.

[0146] In this embodiment, after determining the loss value corresponding to the target image recognition model, it is possible to further determine whether the target image recognition model meets the model convergence conditions, thereby determining whether to continue model training.

[0147] When it is determined based on the loss value that the target image recognition model does not meet the model convergence conditions, the weight parameters of the fully connected layer can be adjusted based on the loss value, and the process returns to step S501 to obtain a sample image, and based on the next sample image obtained, the process executes steps S501-S504 to train the target image recognition model to be trained.

[0148] When it is determined based on the loss value that the target image recognition model meets the model convergence conditions, the model training is terminated.

[0149] Optionally, in order to improve the recognition accuracy of the trained model for images, the above-mentioned model convergence condition may be that the loss value reaches a predetermined loss value, that is, when the determined loss value reaches the predetermined loss value, the training of the model is terminated.

[0150] Optionally, in order to improve the training efficiency of the model training, the model training end condition can be set to the model training times reaching a predetermined number of times, that is, when the model training times reaches the predetermined number of times, the model training is ended.

[0151] Optionally, a back propagation algorithm is used to calculate the gradient value of the feature vector corresponding to the weight parameter of the fully connected layer based on the loss value, and the weight parameter of the fully connected layer is adjusted using the calculated gradient value. Thus, in the process of model training, the model parameters are adjusted according to the determined loss value to balance the above loss value and optimize the model performance.

[0152] Optionally, the electronic device used to perform the above steps S501-S504 and the electronic device used to perform the above steps S201-S203 may be the same electronic device or may be a different electronic device, which is not specifically limited in the embodiment of the present application.

[0153] Optionally, in one embodiment, a method for determining the weight parameters of the fully connected layer may include the following steps:

[0154] Step C1: extracting text features of each sample prompt word, and based on the text features of the sample prompt word, generating a vector feature of a predetermined dimension of the sample prompt word to obtain a second modal feature of the sample prompt word;

[0155] Step C2: concatenate the second modal features of multiple sample prompt words into a matrix to obtain the weight parameters of the fully connected layer;

[0156] Among them, each row of matrix element groups in the weight parameters represents the second modal features of a sample prompt word.

[0157] In this embodiment, image description text with the presentation status of a specified object as the description content can be obtained in advance as a sample prompt word; wherein the above-mentioned sample prompt word can be a single word or a long sentence composed of multiple words, and this embodiment of the application does not make any specific limitation on this.

[0158] In this way, after obtaining each sample prompt word, the text feature of each sample prompt word can be extracted. Considering different data types, such as text data, image data, etc., the corresponding vector features are in different dimensions. In order to facilitate the subsequent determination of the similarity of each sample prompt word with respect to the modal feature of the sample image, the vector features corresponding to data of different data types can be unified into the same predetermined dimension. Therefore, after obtaining the text feature of each sample prompt word, the vector feature of the predetermined dimension of the sample prompt word can be generated to obtain the second modal feature of the sample prompt word.

[0159] Optionally, a feature matrix is ​​used to represent the second modal features of sample prompt words. After obtaining the second modal features of multiple sample prompt words, the second modal features of the multiple sample prompt words can be concatenated into a matrix to obtain weight parameters of the fully connected layer, wherein in the obtained weight parameters, each row of matrix element groups represents the second modal features of a sample prompt word.

[0160] For example, the second modal feature of each sample prompt word is a feature matrix of [1, 512]. Therefore, when there are 10 sample prompt words, the weight parameter of the fully connected layer is the concatenated matrix [10, 512]. Among them, the first row of matrix elements is the second modal feature corresponding to the first sample prompt word, the second row of matrix elements is the second modal feature corresponding to the second sample prompt word, and so on. The tenth row of matrix elements is the second modal feature corresponding to the tenth sample prompt word.

[0161] In this embodiment, considering that the training process of the fully connected layer requires a large number of sample prompt words, before training the target image recognition model, by pre-determining the weights of the fully connected layer, while reducing the number of training samples required for the training process, pre-training parameters of the fully connected layer can be provided for the training process of the target image recognition model. In this way, when training the model, only sample images need to be used to enable the trained target image recognition model to obtain better training results.

[0162] In addition, the contrastive learning model used for image recognition in the prior art includes a text encoder for processing prompt words and an image encoder for processing images. Therefore, in the process of training the above-mentioned contrastive learning model, it is necessary to train the two encoders at the same time, during which the number of training samples and the training time required are greatly increased, resulting in a decrease in the training efficiency of the model. The training method of the target image recognition model provided in this embodiment can realize the training of the target image recognition model by adjusting the weight parameters of the fully connected layer in the reconstructed target image recognition model, so there is no need to train the text encoder and the image encoder at the same time, reducing the number of encoders required to be trained, thereby improving the efficiency of model training.

[0163] Optionally, in one implementation, the above step S503, based on the output result corresponding to each sample prompt word and the label of the sample image, determines the loss value corresponding to the target image recognition model, which may include the following steps:

[0164] Step E1: based on the output results corresponding to each sample prompt word, determine the target confidence corresponding to the target sample prompt word;

[0165] The presentation state described by the target sample prompt word is the same as the presentation state of the sample object represented by the label of the sample image; the target confidence is used to represent the presentation state described in the target sample prompt word, which is the predicted probability of the presentation state of the specified object in the sample image;

[0166] Step E2: Based on the target confidence and the label of the sample image, determine the loss value corresponding to the target image recognition model.

[0167] In this implementation, since the sample image is provided with a label for characterizing the presentation state of the specified object, and each sample prompt word is an image description text with the presentation state of the specified object as the description content, that is to say, the sample prompt word can be understood as a label set on the sample image, and in the training process of a target image recognition model, each sample image will only be provided with one label, that is, there is only one target sample prompt word corresponding to the sample image among multiple sample prompt words, and the presentation state described by the target sample prompt word is the same as the presentation state of the specified object represented by the label of the sample image.

[0168] In summary, the loss value determined based on the output results corresponding to each sample prompt word and the label of the sample image can be understood as being caused by the low accuracy of the second modal features of the target sample prompt word in the weight parameters of the fully connected layer of the target image recognition model. Therefore, the loss value corresponding to the target image recognition model can be determined based on the target confidence corresponding to the target sample prompt word and the label of the sample image.

[0169] Therefore, after determining the output result corresponding to each sample prompt word, the target confidence corresponding to the target sample prompt word can be further determined based on each output result to determine the prediction probability that the presentation state described in the target sample prompt word is the presentation state of the specified object in the sample image. In other words, the higher the target confidence obtained, the higher the prediction probability that the presentation state described in the target sample prompt word is the presentation state of the specified object in the sample image.

[0170] In this way, based on the target confidence corresponding to the target sample prompt word and the label of the sample image, the loss value corresponding to the target image recognition model can be determined. The smaller the determined loss value is, the closer the presentation state described in the target sample prompt word is to the presentation state of the specified object in the sample image.

[0171] Among them, the label set by the sample image for representing the presentation state of the specified object can be understood as a label used to represent the actual state presented by the specified object in the sample image. Therefore, the corresponding confidence can be set directly based on the actual state presented by the specified object. For example, if the specified object is a tattoo, one of the presentation states represented by the label is the presence of a tattoo, and the corresponding confidence of the presentation state represented by the label can be set to 1; conversely, if one of the presentation states represented by the label is the absence of a tattoo, the corresponding confidence of the presentation state represented by the label can be set to 0.

[0172] Optionally, in one implementation, the target confidence and the confidence corresponding to the label of the sample image are input into a predetermined loss function to calculate the loss value corresponding to the target image recognition model. The predetermined loss function may include a cross entropy loss function, etc., which is not specifically limited in the embodiments of the present application.

[0173] In this implementation, the confidence level is used to digitize the probability that the presentation state described in the target sample prompt word is the presentation state of the specified object corresponding to the sample image, so as to determine the loss value of the target image recognition model.

[0174] The following is a detailed introduction to an image recognition method provided by an embodiment of the present application in conjunction with a specific embodiment.

[0175] Since conventional deep learning recognition models need to label large quantities of data, and in actual business, the data that needs to be recognized has a long-tail distribution, that is, the data that needs to be recognized accounts for a small proportion of the total data and is difficult to collect, resulting in high data labeling costs and slow model iteration cycles. Therefore, considering that the CLIP model does not need to collect data to complete image recognition, that is, the CLIP model and other contrastive learning models pre-trained on large quantities of data can directly predict the required classification by inputting prompt words. Such models can be directly applied to the review business according to business needs, for example, according to business needs, review whether there are tattoos in the image. However, since the pre-training data may not contain the features required by the business data, for example, the pre-training data may not contain image features and text features related to tattoos, the accuracy of the recognition results obtained by this type of model when recognizing images containing tattoos is low.

[0176] In the related art, by converting the model structure of the contrastive learning model into the model structure of the traditional deep learning recognition model, and tuning the model after the structural adjustment, the cost of model tuning is greatly reduced while maintaining the excellent performance of the contrastive learning model itself. Specifically, the model is tuned by adjusting the image encoder in the model after the structural adjustment, and the feature parameters of the preset prompt words after passing through the text encoder and the multimodal feature network, that is, adjusting the encoder parameters of the image encoder in the model after the structural adjustment and the weight parameters of the fully connected layer.

[0177] However, after the structurally adjusted model is tuned using the above-mentioned related technologies, the encoder parameters of the image encoder in the model are changed and no longer match the encoder parameter features of the text encoder in the model before the adjustment, resulting in the tuned model being able to only classify the input image according to preset prompt words (i.e., the sample prompt words in the embodiment of the present application), but not according to prompt words other than the preset prompt words (i.e., the designated prompt words in the embodiment of the present application). That is, the tuned model loses the ability to classify on non-tuned samples, so that the model can no longer review the content other than the sample labels, that is, the model cannot identify the presentation status of objects other than the specified sample objects, thereby reducing the application flexibility of the model.

[0178] Based on this, this specific embodiment provides a model structure of an image recognition model. Figure 6 As shown, the image recognition model includes an image encoder 601, a fully connected layer 602 and a similarity calculation module 603, wherein the fully connected layer 602 and the similarity calculation module 603 are connected in parallel with the image encoder 601.

[0179] Taking the designated object as a tattoo as an example, the weight parameter of the fully connected layer 602 is a matrix obtained by determining the second modal features of each sample prompt word related to the tattoo, and the similarity calculation module 603 stores the second modal features of other prompt words unrelated to the tattoo.

[0180] In this way, after receiving the image, the image recognition model calls the image encoder 601 to generate the first modal features of the image, and transmits the first modal features to the fully connected layer 602 and the similarity calculation module 603, so as to realize the recognition of the image by determining the similarity between the second modal features and the first modal features of each sample prompt word and each other prompt word. Therefore, after fine-tuning the image recognition model for the specified current business label (for the specified object), it can not only ensure the algorithmic advantages of image recognition for the image of the specified object, but also ensure the application flexibility of the image recognition model, so that the image recognition model can perform image classification tasks for non-current business labels (for objects other than the specified object).

[0181] The above specific embodiment is introduced below in combination with specific implementation steps.

[0182] For example, assume that there are currently services A, B, C, and D, and the four audit tasks are all to conduct prohibited audits on images uploaded by users, and service A has a data set X that can be used for fine-tuning, while services B, C, and D do not have data sets that can be used for fine-tuning. For example, in the audit service, service A audits whether there are tattoos in the image, and services B, C, and D audit whether there are prohibited images such as barcodes, cigarettes, and alcohol in the image. In other words, for service A, there is a training sample data set that can be used for model training, while services B, C, and D do not have a training sample data set that can be used for model training.

[0183] In combination with the above examples, the specific implementation of this specific embodiment may include the following steps:

[0184] Step 1: Set each initial prompt word of business A as each sample prompt word for model training, and use a text encoder to convert each sample prompt word into a corresponding text multimodal feature (also called text multimodal feature). The above initial prompt words are used to train the target image recognition model.

[0185] Specifically, before model training, multiple initial prompt words for tattoo classification can be set. For example, if the image has a tattoo, the initial prompt word is set to "a photo of person with a tattoo"; if the image does not have a tattoo, the initial prompt word is set to "a photo of person without a tattoo". By combining the above initial prompt words, each sample prompt word for model training is obtained. Then, each sample prompt word is input into the text encoder of the CLIP model respectively to obtain the text feature of each sample prompt word, and then each text feature is converted into a text multimodal feature through a text multimodal network. Here, the text multimodal feature corresponding to each sample prompt word is a feature matrix of size [1, 512].

[0186] The specific implementation method of the above step 1 is the same as the specific implementation method of determining the second modal feature of each other prompt word, which will not be repeated here.

[0187] Step 2: Convert the obtained text multimodal features into the weights of the fully connected layer, and splice the fully connected layer behind the image encoder of the initial recognition model (also referred to as the image multimodal model) to obtain the recognition model to be tuned; wherein the above-mentioned recognition model to be tuned is the target image recognition model to be trained in the embodiment of the present application. wherein the above-mentioned initial recognition model is the contrastive learning model in the embodiment of the present application without the reconstruction of the internal structure, and can also be referred to as the initial contrastive learning model.

[0188] Specifically, considering that the actual role of the fully connected layer in the deep learning recognition model is to calculate the similarity between the features of the content described by the specific label and the image features by adding and dividing the vector length of the text multimodal features of the content described by each specific label and the image multimodal features of the input image. In contrastive learning models, such as image recognition models such as the CLIP model, in the actual reasoning process, it is also necessary to calculate the similarity between the text multimodal features and the image multimodal features by adding and dividing the vector length of each text multimodal feature and the image multimodal feature. Therefore, it can be considered that the role of the acquired text multimodal features is consistent with that of the fully connected layer in the conventional deep learning recognition model, both of which provide text multimodal features of the prompt words. Therefore, the various text multimodal features predetermined in the above step 1 can be converted into fully connected layer weights, and the fully connected layer with the weights added can be spliced ​​after the image encoder to obtain the recognition model to be tuned.

[0189] Since the image encoder sets a fixed preset size for the size of the input image, the image is scaled to the fixed preset size before being input into the image encoder, wherein the fixed preset size is usually 224×224 or 336×336, both of which are reasonable and are not specifically limited in the embodiments of the present application.

[0190] In this way, after receiving the input image, the image encoder in the recognition model to be tuned can extract the image features of the image, and pass the image features through the image multimodal network in the image encoder to convert the image features into image multimodal features. Here, the image multimodal features corresponding to each image are feature matrices of size [1, 512].

[0191] The specific implementation method corresponding to the above-mentioned step two is step S502 in the embodiment of the present application, that is, inputting the sample image into a pre-constructed target image recognition model to be trained, so that the image encoder in the target image recognition model generates a vector feature of a predetermined dimension of the sample image, and obtaining the first sample modal feature of the sample image is similar, which will not be repeated here.

[0192] Step 3: Use the back propagation algorithm to tune only the weight parameters of the fully connected layer in the recognition model to be tuned after splicing, and obtain the recognition model tuned on business A based on the initial recognition model.

[0193] Specifically, multiple text multimodal features and image multimodal features are sequentially dot-multiplied and added, and then divided by the vector length of each text multimodal feature and image multimodal feature to calculate the similarity between each text multimodal feature and image multimodal feature, thereby obtaining a feature score between the text feature of each sample prompt word and the image feature of the input image. The feature score is used to characterize the similarity between each text feature and the image feature.

[0194] Then, define the feature score as z and perform softmax operation on all feature scores:

[0195]

[0196] σ(z) i represents the confidence corresponding to the i-th sample prompt word about tattoos, K represents the total number of sample prompt words about tattoos, and z i Characterize the feature score corresponding to the i-th sample prompt word, z j Characterize the feature score corresponding to the j-th sample prompt word.

[0197] Through the above formula, the feature score of the text feature of each sample prompt word in the image can be mapped to a confidence level between 0 and 1 as the confidence level of the prediction.

[0198] Then, based on the cross-entropy loss function, the loss value between the confidence predicted by the model and the label of the image itself is calculated, and the back propagation algorithm is used to update the weights of the fully connected layer based on the loss value, that is, the text modal feature weights of multiple sample prompt words; wherein, the weights of the above-mentioned fully connected layer, and the text modal feature weights of multiple sample prompt words are both matrices obtained by splicing the second modal features of multiple sample prompt words in the embodiment of the present application, that is, the weight parameters of the fully connected layer in the embodiment of the present application.

[0199] It should be noted that in the above process of updating the weights of the fully connected layer based on the loss value, there is no need to adjust the encoder parameters of the image encoder. That is to say, the features of the image encoder used are not updated in the above process to ensure that the text multimodal features of services B, C and D can share the same image encoder with service A.

[0200] In this way, the above steps 2 and 3 are repeated until all images in the data set are extracted, and gradient descent calculation is performed to optimize the recognition model to be tuned.

[0201] The above steps 1 to 3 can be regarded as one round of model training. Repeat multiple rounds of training until the preset number of training rounds or the loss value reaches a preset value.

[0202] Step 4: After completing the model tuning training of steps 1 to 3 above, before the reasoning task is performed, the labels related to business B, C, and D are spliced ​​on the above-tuned recognition model, specifically, the labels related to business B, C, and D are spliced ​​after the image encoder in the above-tuned recognition model. Among them, the labels related to business B, C, and D are the text multimodal features of business B, C, and D.

[0203] Taking business B as an example, the above step 4 specifically includes the following steps:

[0204] Before the model starts inference:

[0205] Step 1: Set each initial prompt word of business B as each designated prompt word. For example, if business B is to verify whether there is a barcode in the image, the designated prompt words can be set as: "a photo with a bar code" and "a photo without a bar code".

[0206] Step 2: Use the text encoder in the original contrastive learning module (initial recognition model, or initial contrastive learning model) to convert each specified prompt word in step 1 into a corresponding text feature, and then convert each text feature into a text multimodal feature to obtain the text multimodal feature b_f corresponding to business B.

[0207] Step 3: Concatenate the text multimodal feature b_f behind the image encoder of the optimized model. The text multimodal feature b_f and the fully connected layer corresponding to business A are independent of each other and are calculated in parallel.

[0208] Step 4: Repeat the above steps 1-3 until all tags in services B, C and D are processed, that is, the designated prompt words corresponding to services B, C and D are processed, and the text multimodal features corresponding to each designated prompt word are spliced ​​behind the image encoder of the tuned model (that is, the target image recognition model in the embodiment of the present application) to obtain the target model. The above target model is the image recognition model to be used in the embodiment of the present application.

[0209] Optionally, after processing the designated prompt words corresponding to services B, C and D, and the text multimodal features corresponding to each designated prompt word, each text multimodal feature can be stored in a similarity calculation module. In this way, when services B, C and D need to be replaced later, the data stored in the above similarity calculation model can be directly replaced.

[0210] During model inference:

[0211] Step 5: Pass the image through the image encoder of the optimized model to obtain the image multimodal features of the image, and pass the image multimodal features through the fully connected layer corresponding to business A, and the text multimodal features corresponding to businesses B, C and D respectively, so as to obtain the independent feature scores of the image for businesses A, B, C and D.

[0212] Step 6: The independent feature scores of the image for services A, B, C and D are respectively calculated by softmax to obtain the confidence scores of each label (prompt word) of the image under each service, thereby realizing the recognition of the image.

[0213] Compared with the use of traditional models and traditional training methods, this embodiment only requires a small number of samples to obtain a higher model accuracy, greatly reducing the difficulty of data collection in scenarios with long-tail distribution data characteristics such as audit business. Moreover, compared with the training process of the existing comparative learning model, in this embodiment, only the fully connected layer of the recognition model to be tuned needs to be trained, that is, by training one image recognition model, the effect of training two encoders can be achieved. In other words, by reducing the number of encoders that need to be trained during model training, the number of training samples and training time required during training are reduced, thereby improving the training efficiency of the image recognition model.

[0214] Moreover, in the subsequent reasoning process of the trained image recognition model, the text encoder can be discarded, and only the weight parameters of the image encoder and the fully connected layer can be retained, thereby improving the image recognition efficiency of the subsequent image recognition model.

[0215] In addition, the image recognition model to be used finally obtained includes an image encoder, a fully connected layer for similarity calculation for a specified object, and a similarity calculation module for similarity calculation for other objects. The fully connected layer can ensure the recognition accuracy of the image recognition model to be used for images containing the specified object. At the same time, the similarity calculation module can expand the recognition ability of the image recognition model to be used for images containing other objects other than the specified object. Therefore, based on the model structure of the image recognition model to be used that can be fine-tuned and expanded for open set classification, the application flexibility of the subsequent image recognition model in the actual reasoning process can be improved.

[0216] Moreover, the above-mentioned image recognition model to be used can simultaneously process multiple image recognition tasks for different objects, that is, one model can be used to infer multiple tasks at the same time, which greatly reduces the consumption of online review resources.

[0217] Corresponding to the above method embodiment, the present application also provides an image recognition device, such as Figure 7FIG. 1 is a schematic diagram of the structure of an image recognition device provided in an embodiment of the present application, wherein the device comprises:

[0218] The image acquisition module 710 is used to acquire a target image; wherein the target image is an image used to identify a target object;

[0219] The similarity determination module 720 is used to input the target image into the image recognition model to be used, and obtain an output result corresponding to each predetermined prompt word in each predetermined prompt word; wherein the image recognition model to be used is obtained by adding a similarity calculation module to the target image recognition model that has been pre-trained for identifying the specified object; each predetermined prompt word includes at least one other prompt word and a sample prompt word used to train the target image recognition model, and the other prompt word is an image description text that uses the presentation state of other objects other than the specified object as the description content, and the other objects and the specified objects include the target object; the similarity calculation module is used to generate an output result corresponding to each other prompt word based on the second modal feature of each other prompt word and the first modal feature of the target image generated by the image encoder of the target image recognition model; the output result is used to characterize the similarity between the second modal feature of the predetermined prompt word and the first modal feature;

[0220] The presentation state determination module 730 is used to determine the presentation state of the target image for the target object based on the output result corresponding to each predetermined prompt word.

[0221] As can be seen from the above, by applying the solution provided by the embodiment of the present application, before image recognition is performed, sample prompt words are used in advance to train a target image recognition model for identifying a specified object, and then a similarity calculation module is added to the target image recognition model. The similarity calculation module is used to generate an output result corresponding to each other prompt word based on the second modal feature of each other prompt word and the first modal feature of the target image generated by the image encoder of the target image recognition model. In other words, the image recognition model to be used can use each predetermined prompt word including at least one other prompt word and sample prompt words used to train the target image recognition model to recognize the input target image. Among them, the other prompt words are image description texts that use the presentation status of other objects other than the specified object as the description content, and the other objects and the specified objects include the target object in the target image.

[0222] In this way, in the process of image recognition, after obtaining the image used to identify the target object as the target image, the obtained target image can be input into the image recognition model to be used to obtain the output result corresponding to each of the predetermined prompt words. The output result is used to characterize the similarity between the second modal feature and the first modal feature of the predetermined prompt word. Therefore, based on the output result corresponding to each predetermined prompt word, the presentation state of the target image for the target object can be determined to achieve recognition of the target image.

[0223] Based on this, the solution provided in the embodiment of the present application is applied. By adding a similarity calculation module for determining the similarity of other objects to the target image recognition model pre-trained for the specified object, it is possible to achieve the image recognition accuracy of the image recognition model for images containing the specified object while expanding the image recognition capability of the image recognition model for images other than the specified object, thereby improving the application flexibility of the image recognition model.

[0224] Optionally, in one implementation, the first modal feature of the target image is a vector feature of a predetermined dimension;

[0225] The method for determining the second modal feature of each other prompt word includes:

[0226] The text features of the other prompt words are extracted, and based on the text features of the other prompt words, the vector features of the predetermined dimension of the other prompt words are generated to obtain the second modal features of the other prompt words.

[0227] Optionally, in one implementation, the presentation status determination module 730 includes:

[0228] A confidence determination submodule, used to determine the confidence corresponding to each predetermined prompt word based on the output result corresponding to the predetermined prompt word; wherein the confidence corresponding to each predetermined prompt word is used to characterize the probability that the presentation state described in the predetermined prompt word is the presentation state corresponding to the target image with respect to the target object;

[0229] The presentation status analysis submodule is used to analyze the presentation status of the target image with respect to the target object based on the confidence level corresponding to each predetermined prompt word.

[0230] Optionally, in an implementation, the presentation status analysis submodule is specifically configured to:

[0231] From the confidence levels corresponding to each predetermined prompt word, select the highest confidence level as the target confidence level;

[0232] If the target confidence is greater than a predetermined threshold, the presentation state of the target image for the target object is determined to be: the presentation state described by the predetermined prompt word corresponding to the target confidence.

[0233] Optionally, in one implementation, the training method of the target image recognition model includes:

[0234] Acquire a sample image; wherein the sample image is provided with a label for characterizing a presentation state of a specified object;

[0235] The sample image is input into a pre-built target image recognition model to be trained, so that an image encoder in the target image recognition model generates a vector feature of a predetermined dimension of the sample image, obtains a first sample modality feature of the sample image, and transmits the first sample modality feature to a fully connected layer in the target image recognition model, so as to process the first sample modality feature based on a weight parameter of the fully connected layer, and obtain an output result corresponding to each sample prompt word in each sample prompt word; wherein the output result corresponding to each sample prompt word is used to characterize the similarity between the second modality feature of the sample prompt word and the first sample modality feature of the sample image;

[0236] Determine the loss value corresponding to the target image recognition model based on the output result corresponding to each sample prompt word and the label of the sample image;

[0237] When it is determined based on the loss value that the target image recognition model does not meet the model convergence condition, the weight parameters of the fully connected layer are adjusted based on the loss value, and the process returns to the step of obtaining the sample image.

[0238] Optionally, in one implementation, a method for determining the weight parameters of the fully connected layer includes:

[0239] Extracting text features of each sample prompt word, and generating vector features of a predetermined dimension of the sample prompt word based on the text features of the sample prompt word, to obtain a second modal feature of the sample prompt word;

[0240] The second modality features of multiple sample prompt words are concatenated into a matrix to obtain the weight parameters of the fully connected layer; wherein each row of matrix element groups in the weight parameters represents the second modality feature of a sample prompt word.

[0241] Optionally, in an implementation, determining the loss value corresponding to the target image recognition model based on the output result corresponding to each sample prompt word and the label of the sample image includes:

[0242] Based on the output results corresponding to each sample prompt word, determine the target confidence corresponding to the target sample prompt word; wherein the presentation state described by the target sample prompt word is the same as the presentation state of the sample object represented by the label of the sample image; the target confidence is used to represent the presentation state described in the target sample prompt word, which is the predicted probability of the presentation state of the specified object in the sample image;

[0243] Based on the target confidence and the label of the sample image, a loss value corresponding to the target image recognition model is determined.

[0244] The present application also provides an electronic device, such as Figure 8 As shown, it includes a processor 801, a communication interface 802, a memory 803 and a communication bus 804, wherein the processor 801, the communication interface 802, and the memory 803 communicate with each other through the communication bus 804.

[0245] Memory 803, used for storing computer programs;

[0246] The processor 801 is used to implement any image recognition method provided in the above-mentioned embodiments of the present application when executing the program stored in the memory 803.

[0247] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0248] The communication interface is used for communication between the above terminal and other devices.

[0249] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0250] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0251] In another embodiment provided in the present application, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the image recognition method described in any one of the above embodiments is implemented.

[0252] In another embodiment provided in the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute the image recognition method described in any one of the above embodiments.

[0253] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive Solid State Disk (SSD)), etc.

[0254] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0255] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, electronic device embodiment, computer-readable storage medium embodiment, and computer program product embodiment, since they are basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0256] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. An image recognition method, characterized in that: The method comprises: Acquire a target image; wherein the target image is an image used to identify a target object; The target image is input into the image recognition model to be used, and an output result corresponding to each predetermined prompt word in each predetermined prompt word is obtained; wherein the image recognition model to be used is obtained by adding a similarity calculation module to the target image recognition model that has been pre-trained for identifying the specified object; each predetermined prompt word includes at least one other prompt word and a sample prompt word for training the target image recognition model, and the other prompt word is an image description text that uses the presentation state of other objects other than the specified object as the description content, and the other objects and the specified objects include the target object; the similarity calculation module is used to generate an output result corresponding to each other prompt word based on the second modal feature of each other prompt word and the first modal feature of the target image generated by the image encoder of the target image recognition model; the output result is used to characterize the similarity between the second modal feature of the predetermined prompt word and the first modal feature; Based on the output result corresponding to each predetermined prompt word, a presentation state of the target image for the target object is determined.

2. The method according to claim 1, characterized in that The first modal feature of the target image is a vector feature of a predetermined dimension; The method for determining the second modal feature of each other prompt word includes: The text features of the other prompt words are extracted, and based on the text features of the other prompt words, the vector features of the predetermined dimension of the other prompt words are generated to obtain the second modal features of the other prompt words.

3. The method according to claim 1, characterized in that: The determining, based on the output result corresponding to each predetermined prompt word, a presentation state of the target image for the target object, comprises: Based on the output result corresponding to each predetermined prompt word, determine the confidence level corresponding to the predetermined prompt word; wherein the confidence level corresponding to each predetermined prompt word is used to characterize the presentation state described in the predetermined prompt word, which is the probability of the presentation state of the target object corresponding to the target image; Based on the confidence level corresponding to each predetermined prompt word, the presentation status of the target image with respect to the target object is analyzed.

4. The method according to claim 3, characterized in that The analyzing the presentation status of the target image to the target object based on the confidence level corresponding to each predetermined prompt word includes: From the confidence levels corresponding to each predetermined prompt word, select the highest confidence level as the target confidence level; If the target confidence is greater than a predetermined threshold, the presentation state of the target image for the target object is determined to be: the presentation state described by the predetermined prompt word corresponding to the target confidence.

5. The method according to claim 1, characterized in that The training method of the target image recognition model includes: Acquire a sample image; wherein the sample image is provided with a label for characterizing a presentation state of a specified object; The sample image is input into a pre-built target image recognition model to be trained, so that an image encoder in the target image recognition model generates a vector feature of a predetermined dimension of the sample image, obtains a first sample modality feature of the sample image, and transmits the first sample modality feature to a fully connected layer in the target image recognition model, so as to process the first sample modality feature based on a weight parameter of the fully connected layer, and obtain an output result corresponding to each sample prompt word in each sample prompt word; wherein the output result corresponding to each sample prompt word is used to characterize the similarity between the second modality feature of the sample prompt word and the first sample modality feature of the sample image; Determine the loss value corresponding to the target image recognition model based on the output result corresponding to each sample prompt word and the label of the sample image; When it is determined based on the loss value that the target image recognition model does not meet the model convergence condition, the weight parameters of the fully connected layer are adjusted based on the loss value, and the process returns to the step of obtaining the sample image.

6. The method according to claim 5, characterized in that The method for determining the weight parameters of the fully connected layer includes: Extracting text features of each sample prompt word, and generating vector features of a predetermined dimension of the sample prompt word based on the text features of the sample prompt word, to obtain a second modal feature of the sample prompt word; The second modality features of multiple sample prompt words are concatenated into a matrix to obtain the weight parameters of the fully connected layer; wherein each row of matrix element groups in the weight parameters represents the second modality feature of a sample prompt word.

7. The method according to claim 5, characterized in that The determining, based on the output result corresponding to each sample prompt word and the label of the sample image, a loss value corresponding to the target image recognition model includes: Based on the output results corresponding to each sample prompt word, determine the target confidence corresponding to the target sample prompt word; wherein the presentation state described by the target sample prompt word is the same as the presentation state of the sample object represented by the label of the sample image; the target confidence is used to represent the presentation state described in the target sample prompt word, which is the predicted probability of the presentation state of the specified object in the sample image; Based on the target confidence and the label of the sample image, a loss value corresponding to the target image recognition model is determined.

8. An image recognition device, characterized in that: The device comprises: An image acquisition module, used to acquire a target image; wherein the target image is an image used to identify a target object; A similarity determination module is used to input the target image into the image recognition model to be used, and obtain an output result corresponding to each predetermined prompt word in each predetermined prompt word; wherein the image recognition model to be used is obtained by adding a similarity calculation module to the target image recognition model that has been pre-trained for identifying the specified object; each predetermined prompt word includes at least one other prompt word and a sample prompt word used to train the target image recognition model, and the other prompt word is an image description text that uses the presentation state of other objects other than the specified object as the description content, and the other objects and the specified objects include the target object; the similarity calculation module is used to generate an output result corresponding to each other prompt word based on the second modal feature of each other prompt word and the first modal feature of the target image generated by the image encoder of the target image recognition model; the output result is used to characterize the similarity between the second modal feature of the predetermined prompt word and the first modal feature; The presentation state determination module is used to determine the presentation state of the target image for the target object based on the output result corresponding to each predetermined prompt word.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1 to 7 when executing a program stored in a memory.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1 to 7 are implemented.

Images