Model training method, image recognition method, device and electronic equipment

Abstract

The embodiments of the present application provide a model training method, an image recognition method, a device and an electronic device, which relate to the field of machine learning technology. The model training method includes: obtaining a sample image; inputting the sample image and each sample prompt word into a pre-trained contrastive learning model, so that the contrastive learning model generates the first modal feature of the sample image; determining the current second modal feature of each sample prompt word; calculating the similarity of the sample image for each sample prompt word; determining the loss value based on each similarity and the label of the sample image; when it is judged based on the loss value that the model does not meet the model convergence condition, based on the loss value, adjusting the parameters of the specified feature part in the current second modal feature of each sample prompt word, and returning to the step of obtaining the sample image. This solution can reduce the influence of non-keywords on the accuracy of the recognition results, thereby ensuring the accuracy of the contrastive learning model for image recognition.

Description

Technical Field

[0001] The present application relates to the field of machine learning technology, and in particular to a model training method, image recognition method, device and electronic equipment. Background Art

[0002] In order to maintain the network environment of the online platform, it is necessary to review the images or videos uploaded to the platform to avoid the dissemination of images or videos that may cause bad guidance through the platform, for example, the dissemination of images including tattoos, cigarettes, etc.

[0003] Typically, a pre-trained contrastive learning model, such as a pre-trained CLIP (Contrastive Language-Image Pre-Training) model, is used to perform image recognition, specifically identifying the image's representation of an object, such as whether the object is present in the image. During the recognition process, the contrastive learning model can identify the image based on pre-set prompt words.

[0004] However, the inventors discovered during the process of using the above contrastive learning model for image recognition that:

[0005] For the same object, if the keywords in the prompt words are the same but other non-keywords are different, the contrastive learning model will obtain different recognition results for the same image. For example, "a photo of a person with a tattoo" and "a photo of a person with tattoo" are prompt words for "image of a person with tattoo" that share the keyword "tattoo" but different non-keywords.

[0006] Based on this, how to optimize the pre-trained contrastive learning model to reduce the impact of non-keywords in the prompt words on the accuracy of recognition results, thereby ensuring the accuracy of image recognition using the contrastive learning model, has become a technical problem that needs to be solved urgently. Summary of the Invention

[0007] The purpose of the embodiments of the present application is to provide a model training method, image recognition method, device, and electronic device to reduce the impact of non-keywords in prompt words on the accuracy of recognition results, thereby ensuring the accuracy of contrastive learning models for image recognition. The specific technical solutions are as follows:

[0008] In a first aspect of the embodiments of the present application, a model training method is provided, comprising:

[0009] Acquire a sample image; wherein the sample image is provided with a label for characterizing a presentation state of the sample object;

[0010] The sample image and each predetermined sample prompt word are input into a pre-trained contrastive learning model so that the contrastive learning model generates a vector feature of the sample image, obtains a first modal feature of the sample image, determines a current second modal feature of each sample prompt word, and calculates the similarity between the first modal feature of the sample image and the second modal feature of each sample prompt word, thereby obtaining a similarity of the sample image to each sample prompt word. The pre-trained contrastive learning model is trained on the basis of a pre-trained model using images containing objects other than the sample object. Each sample prompt word is an image description text that describes the presentation state of the sample object. Any sample prompt word belongs to one of two prompt word types, the presentation states described by the two prompt word types are mutually exclusive, and sample prompt words belonging to the same prompt word type are semantically similar. Each sample prompt word is obtained by combining a target keyword and other non-keywords, wherein the target keyword is a keyword used to characterize a presentation state of the sample object. The initial second modal feature of each sample prompt word is a vector feature of the sample prompt word.

[0011] Determining a loss value corresponding to the contrastive learning model based on the similarity of the sample image to each sample prompt word and the label of the sample image;

[0012] When it is determined based on the loss value that the contrastive learning model does not meet the model convergence conditions, parameters are adjusted for the specified feature part in the current second modal feature of each sample prompt word, and the process returns to the step of obtaining the sample image; wherein the specified feature part in the current second modal feature of each sample prompt word is the feature part corresponding to other non-keywords in the current second modal feature of the sample prompt word.

[0013] In a second aspect of the embodiments of the present application, an image recognition method is further provided, the method comprising:

[0014] Cutting out a region of interest from the target image to obtain a target sub-image; wherein the region of interest is a carrier region where the target object to be identified is located in the target image when the target object exists in the target image;

[0015] The target sub-image and the prompt phrase are input into a pre-trained contrastive learning model so that the contrastive learning model generates the vector features of the target sub-image to obtain the first modal features, and generates the initial modal features of each prompt word in the prompt phrase, and, for each prompt word, splices the target sub-feature in the initial modal features of the prompt word with the designated sub-feature corresponding to the target sample prompt word to obtain the second modal features of the prompt word; calculates the similarity between the first modal features of the target sub-image and the second modal features of each prompt word, and obtains the similarity of the target sub-image to each prompt word; wherein, the prompt Each prompt word in the phrase is an image description text that uses the presentation state of the target object as its description content, and the presentation states described by different prompt words in the same prompt word group are mutually exclusive. The contrastive learning model is trained based on a model training method provided in the first aspect above. The target sample prompt word is a sample prompt word among the sample prompt words whose other non-keywords match the other non-keywords of the prompt word. The designated sub-feature is the feature content of the designated feature part of the current second modal feature of the target sample prompt word when the contrastive learning model is trained. The target sub-feature is a feature that characterizes the target keyword.

[0016] Determine a recognition result corresponding to the target sub-image based on the similarity of each prompt word; wherein the recognition result is a result regarding the presentation state of the target object;

[0017] Based on the recognition result corresponding to the target sub-image, a presentation state of the target image with respect to the target object is determined.

[0018] In a third aspect of the embodiments of the present application, a model training device is provided, the device comprising:

[0019] An image acquisition module, configured to acquire a sample image; wherein the sample image is provided with a label for characterizing a presentation state of the sample object;

[0020] a similarity acquisition module, configured to input the sample image and each predetermined sample prompt word into a pre-trained contrastive learning model, so that the contrastive learning model generates a vector feature of the sample image, obtains a first modal feature of the sample image, determines a current second modal feature of each sample prompt word, and calculates the similarity between the first modal feature of the sample image and the second modal feature of each sample prompt word, thereby obtaining the similarity of the sample image with respect to each sample prompt word; wherein the pre-trained contrastive learning model is trained on the basis of a pre-trained model using images containing objects other than the sample object; each sample prompt word is an image description text that describes the presentation state of the sample object, and any sample prompt word belongs to one of two prompt word types, the presentation states described by the two prompt word types are mutually exclusive, and sample prompt words belonging to the same prompt word type have similar semantics; each sample prompt word is obtained by combining a target keyword and other non-keywords, wherein the target keyword is a keyword used to characterize a presentation state of the sample object; and the initial second modal feature of each sample prompt word is a vector feature of the sample prompt word;

[0021] a loss value calculation module, configured to determine a loss value corresponding to the contrastive learning model based on the similarity of the sample image to each sample prompt word and the label of the sample image;

[0022] A parameter adjustment module is used to adjust the parameters of the specified feature part in the current second modal feature of each sample prompt word when it is determined based on the loss value that the contrastive learning model does not meet the model convergence conditions, and return to the step of obtaining the sample image; wherein the specified feature part in the current second modal feature of each sample prompt word is the feature part corresponding to other non-keywords in the current second modal feature of the sample prompt word.

[0023] In a fourth aspect of the embodiments of the present application, an image recognition device is provided, the device comprising:

[0024] An image capture module is configured to capture a region of interest from a target image to obtain a target sub-image; wherein the region of interest is a carrier region where the target object to be identified is located in the target image when the target object exists in the target image;

[0025] A similarity determination module is configured to input the target sub-image and the prompt phrase into a pre-trained contrastive learning model, so that the contrastive learning model generates a vector feature of the target sub-image to obtain a first modal feature, and generates an initial modal feature for each prompt word in the prompt phrase, and, for each prompt word, concatenates the target sub-feature in the initial modal feature of the prompt word with a designated sub-feature corresponding to the target sample prompt word to obtain a second modal feature of the prompt word; calculates the similarity between the first modal feature of the target sub-image and the second modal feature of each prompt word, respectively, to obtain the similarity of the target sub-image for each prompt word; wherein Each prompt word in the prompt word group is an image description text that uses the presentation state of the target object as a description content, and the presentation states described by different prompt words in the same prompt word group are mutually exclusive. The contrastive learning model is trained based on a model training method provided by the first aspect above. The target sample prompt word is a sample prompt word among each sample prompt word whose other non-keywords match the other non-keywords of the prompt word. The designated sub-feature is the feature content of the designated feature part of the current second modal feature of the target sample prompt word when the contrastive learning model is trained. The target sub-feature is a feature that characterizes the target keyword.

[0026] A result determination module is configured to determine a recognition result corresponding to the target sub-image based on the similarity of each prompt word; wherein the recognition result is a result regarding the presentation state of the target object;

[0027] The image determination module is configured to determine a presentation state of the target image with respect to the target object based on a recognition result corresponding to the target sub-image.

[0028] In the fifth aspect provided by the embodiment of the present application, an electronic device is also provided, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; the memory is used to store computer programs; the processor is used to implement any model training method provided in the first aspect and / or any image recognition method provided in the second aspect when executing the program stored in the memory.

[0029] In another aspect provided by the embodiments of the present application, a computer-readable storage medium is also provided, in which a computer program is stored. When the computer program is executed by a processor, it implements any model training method provided in the first aspect above, and / or any image recognition method provided in the second aspect above.

[0030] In another aspect provided by the embodiments of the present application, a computer program product comprising instructions is also provided, which, when run on a computer, enables the computer to execute any model training method provided in the first aspect above, and / or any image recognition method provided in the second aspect above.

[0031] In the model training method provided in the embodiment of the present application, before model training, sample prompt words can be set in advance for sample objects, and each sample prompt word is an image description text with the presentation state of the sample object as the description content, and any sample prompt word belongs to one of two prompt word types. The presentation states described by the two prompt word types are mutually exclusive, and the sample prompt words belonging to the same prompt word type have similar semantics. Each sample prompt word is obtained based on a target keyword and other non-keyword combinations, wherein the above-mentioned target keyword is a keyword used to characterize a presentation state of the sample object. Considering that other non-keywords in the prompt word have a significant impact on the second modal features of the entire prompt word, the contrastive learning model is relatively stable in generating vector features corresponding to the target keyword in the second modal features of the prompt word. However, for other non-keywords, the generated vector features may be different if the non-keywords are the same but combined differently, or if the non-keywords are different. In other words, the contrastive learning model is unstable in generating vector features corresponding to other non-keywords in the second modal features of the prompt word, and the generated vector features are quite different. Therefore, by determining the similarity of the sample images for each sample prompt word and the difference in the pre-set labels of the sample images themselves, the loss value of the current second modal feature of each sample prompt word generated by the contrastive learning model is determined. Therefore, when the contrastive learning model is judged to not meet the model convergence conditions based on the loss value, the parameters of the feature portion corresponding to other non-keywords in the current second modal features of each sample prompt word are adjusted to reduce the impact of other non-keywords in the prompt word on the accuracy of the recognition result, thereby ensuring the accuracy of image recognition using the contrastive learning model.

[0032] Furthermore, combining a target keyword with other non-keywords to create sample prompts reduces the tedious process of manually designing prompts and mitigates the potential for low accuracy associated with manually designed prompts. Furthermore, the sample prompts created through this combination method have a high accuracy rate. Therefore, during model training, a comparative learning model with high recognition performance can be trained using only a small number of sample prompts and sample images.

[0033] Furthermore, in the image recognition method provided in the embodiment of the present application, after a target sub-image is captured from a target image, the target sub-image and a prompt phrase associated with the target object are input into a pre-trained contrastive learning model. Accordingly, because the contrastive learning model reduces the influence of other non-keywords in the prompt phrase on the accuracy of the recognition result during the training process, when using the contrastive learning model to recognize the target object, for each prompt word in the prompt phrase, the target sub-feature of the prompt word is concatenated with the designated sub-feature corresponding to the target sample prompt word to obtain the second modal feature of the prompt word. Specifically, the feature representing the target keyword of the prompt word is concatenated with the feature content of the designated feature portion of the current second modal feature of the target sample prompt word when the contrastive learning model is trained, to obtain the second modal feature of the prompt word. Thus, the similarity between the first modal feature of the target sub-image and the second modal feature of each prompt word is calculated to obtain the similarity of the target sub-image with respect to each prompt word. Based on the obtained similarity, the presentation state of the target image with respect to the target object is determined. Through this solution, we can focus on the characteristic content of the keyword part in each prompt word in the prompt phrase group and use the specified characteristic part in the target sample prompt word to reduce the influence of non-keywords in the image recognition process, thereby achieving accurate recognition of the target object. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art.

[0035] Figure 1 A schematic diagram of the classification process of the CLIP model provided in an embodiment of the present application;

[0036] Figure 2 This is a diagram showing the effect of different non-keywords on the final accuracy provided in the embodiment of the present application;

[0037] Figure 3 A flowchart of a model training method provided in an embodiment of the present application;

[0038] Figure 4 A flowchart of an existing training method for a contrastive learning model is shown;

[0039] Figure 5 A schematic diagram of a process for obtaining multimodal features of text provided in an embodiment of the present application;

[0040] Figure 6 A schematic diagram of the flow of the first image recognition method provided in an embodiment of the present application;

[0041] Figure 7A schematic diagram of the flow of the second image recognition method provided in an embodiment of the present application;

[0042] Figure 8 A schematic diagram of the structure of a model training device provided in an embodiment of the present application;

[0043] Figure 9 A schematic diagram of the structure of an image recognition device provided in an embodiment of the present application;

[0044] Figure 10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0046] In order to better understand the embodiments of the present application, the professional terms involved in the embodiments of the present application are first introduced below.

[0047] The text-image pre-training model based on contrastive learning (also known as the contrastive learning model) is a model pre-trained on 400 million pairs of image-text matching pairs. It has a strong generalization ability. By giving the contrastive learning model the corresponding text prompt words, the model can obtain the similarity of the corresponding image. Specifically, the contrastive learning model recognizes the image, that is, recognizes the presentation state of the image for the object, such as whether there is an object in the image. In the recognition process, the contrastive learning model can classify and recognize the image based on pre-set prompt words. The above-mentioned contrastive learning model can be a CLIP model, a BLIP (Bootstrapping Language-Image Pre-training) model or a SigLip (Sigmoid Loss for Language-Image Pre-training) model, which is not specifically limited in the embodiments of the present application.

[0048] Taking the CLIP model as an example, when using the CLIP model to classify images, the CLIP model can be given two prompt words, "people with tattoos" and "people without tattoos", and the CLIP model can be allowed to determine which prompt word has a closer similarity to the other two words to classify the images.

[0049] The following combination Figure 1 This paper introduces the classification process of CLIP model. Figure 1As shown in the figure, plane, car, dog, and bird are all target objects (or sample objects) in the embodiment of the present application, and "a photo of a {object}" is the prompt word in the embodiment of the present application. The byte position of object is the byte position occupied by the target object in the prompt word.

[0050] After receiving the prompt word, the Text Encoder of the CLIP model can convert the text corresponding to the prompt word into multimodal features including T1, T2, T3...T N .

[0051] After receiving the image, the Image Encoder of the CLIP model can convert it into the image multimodal feature I1 corresponding to the above image.

[0052] The CLIP model can then combine the obtained image multimodal feature I1 with the text multimodal features T1, T2, T3, ... T corresponding to the prompt word. N Perform feature similarity matching to obtain the similarity I1·T1, I1·T2, I1·T3, I1·T corresponding to each prompt word N Among them, I1·T3 has the highest similarity, and the object in the prompt word corresponding to I1·T3 is dog. Therefore, it can be determined that the object in the image input to the Image Encoder is dog, and the output result of the model is "a photo of a dog".

[0053] Among them, the above-mentioned image multimodal feature is the first modal feature in the embodiment of the present application; the above-mentioned text multimodal feature (text multimodal feature) is the second modal feature in the embodiment of the present application.

[0054] However, when giving the CLIP model prompts, using different prompts can affect the final classification results. For example, when the CLIP model is given two different prompts describing "an image of a person with a tattoo," "a photo of a person with a tattoo" or "a photo of a person with a tattoo," the final accuracy of the contrastive learning model will be different. In other words, for the same object, if the keywords in the prompts are the same but other non-keywords are different, the contrastive learning model will produce different recognition results for the same image.

[0055] For example, Figure 2The effect of different non-keywords on the final accuracy is shown in the figure. Among them, CLASS (classification) is used to represent the label that needs to be classified, which can also be understood as a keyword; [V] i Used to represent non-keywords; Accuracy is used to represent the accuracy on the test set when using the corresponding prompt word; Prompt is used to represent the prompt word; Photo is the image to be recognized.

[0056] Depend on Figure 2 It can be seen that for the same image to be identified, the accuracy rates obtained by the comparative learning model are different when other non-keywords are different, or when other non-keywords are the same but the combinations of non-keywords are different.

[0057] Based on this, how to optimize the pre-trained contrastive learning model to reduce the impact of non-keywords in the prompt words on the accuracy of recognition results, thereby ensuring the accuracy of image recognition using the contrastive learning model, has become a technical problem that needs to be solved urgently.

[0058] In order to solve the above technical problems, an embodiment of the present application provides a model training method.

[0059] Among them, a model training method provided in an embodiment of the present application can be applied to an electronic device, and the electronic device is specifically used to train the model. In specific applications, the electronic device can be a smart phone, a tablet computer, etc., which is all reasonable.

[0060] The present invention provides a model training method, which may include the following steps:

[0061] Acquire a sample image; wherein the sample image is provided with a label for characterizing a presentation state of the sample object;

[0062] The sample image and each predetermined sample prompt word are input into a pre-trained contrastive learning model so that the contrastive learning model generates a vector feature of the sample image, obtains a first modal feature of the sample image, determines a current second modal feature of each sample prompt word, and calculates the similarity between the first modal feature of the sample image and the second modal feature of each sample prompt word, thereby obtaining a similarity of the sample image to each sample prompt word. The pre-trained contrastive learning model is trained on the basis of a pre-trained model using images containing objects other than the sample object. Each sample prompt word is an image description text that describes the presentation state of the sample object. Any sample prompt word belongs to one of two prompt word types, the presentation states described by the two prompt word types are mutually exclusive, and sample prompt words belonging to the same prompt word type have similar semantics. Each sample prompt word is obtained by combining a target keyword and other non-keywords, wherein the target keyword is a keyword used to characterize a presentation state of the sample object. The initial second modal feature of each sample prompt word is a vector feature of the sample prompt word.

[0063] Determining a loss value corresponding to the contrastive learning model based on the similarity of the sample image to each sample prompt word and the label of the sample image;

[0064] When it is determined based on the loss value that the contrastive learning model does not meet the model convergence conditions, parameters are adjusted for the specified feature part in the current second modal feature of each sample prompt word, and the process returns to the step of obtaining the sample image; wherein the specified feature part in the current second modal feature of each sample prompt word is the feature part corresponding to other non-keywords in the current second modal feature of the sample prompt word.

[0065] As can be seen from the above, in the model training method provided in the embodiment of the present application, sample prompt words can be set in advance for the sample objects before model training. Each sample prompt word is an image description text with the presentation state of the sample object as the description content, and any sample prompt word belongs to one of the two prompt word types. The presentation states described by the two prompt word types are mutually exclusive, and the sample prompt words belonging to the same prompt word type have similar semantics. Each sample prompt word is obtained based on a target keyword and other non-keyword combinations, wherein the above-mentioned target keyword is a keyword used to characterize a presentation state of the sample object. Considering that other non-keywords in the prompt word have a significant impact on the second modal features of the entire prompt word, the contrastive learning model is relatively stable in generating vector features corresponding to the target keyword in the second modal features of the prompt word. However, for other non-keywords, the generated vector features may be different if the non-keywords are the same but combined differently, or if the non-keywords are different. In other words, the contrastive learning model is unstable in generating vector features corresponding to other non-keywords in the second modal features of the prompt word, and the generated vector features are quite different. Therefore, by determining the similarity of the sample images for each sample prompt word and the difference in the pre-set labels of the sample images themselves, the loss value of the current second modal feature of each sample prompt word generated by the contrastive learning model is determined. Therefore, when the contrastive learning model is judged to not meet the model convergence conditions based on the loss value, the parameters of the feature portion corresponding to other non-keywords in the current second modal features of each sample prompt word are adjusted to reduce the impact of other non-keywords in the prompt word on the accuracy of the recognition result, thereby ensuring the accuracy of image recognition using the contrastive learning model.

[0066] Furthermore, combining a target keyword with other non-keywords to create sample prompts reduces the tedious process of manually designing prompts and mitigates the potential for low accuracy associated with manually designed prompts. Furthermore, the sample prompts created through this combination method have a high accuracy rate. Therefore, during model training, a comparative learning model with high recognition performance can be trained using only a small number of sample prompts and sample images.

[0067] The following is a detailed description of a model training method provided in an embodiment of the present application in conjunction with the accompanying drawings.

[0068] Figure 3 A flow chart of a model training method provided in an embodiment of the present application is shown as follows: Figure 3 As shown, the method includes the following steps:

[0069] S301: Acquire a sample image.

[0070] The sample image is provided with a label for characterizing the presentation state of the sample object.

[0071] In the present application, an image provided with a representation for representing the presentation state of the sample object may be acquired as a sample image.

[0072] It is understandable that the number of sample images obtained may be one or more, and this embodiment of the present application does not impose any specific limitation on this.

[0073] The label used to represent the presentation status of the sample object can be a numeric label or a text label, which is specifically limited in the embodiments of the present application. For example, if the sample object is present in the sample image, the label of the sample image can be set to 1; if the sample object is not present in the sample image, the label of the sample image can be set to 0.

[0074] Optionally, in one implementation, the sample object is a tattoo.

[0075] In this implementation, if the sample object is a tattoo, the sample image obtained is an image of the tattoo. Accordingly, the presentation state of the sample object may include: the presence or absence of a tattoo, the presence of a dark tattoo or the presence of a light tattoo, etc.

[0076] S302: Input the sample image and each predetermined sample prompt word into the pre-trained contrastive learning model, so that the contrastive learning model generates the vector features of the sample image, obtains the first modal features of the sample image, determines the current second modal features of each sample prompt word, and calculates the similarity between the first modal features of the sample image and the second modal features of each sample prompt word, thereby obtaining the similarity of the sample image to each sample prompt word.

[0077] Among them, the pre-trained contrastive learning model is obtained by training on the basis of the pre-trained model using images containing objects other than sample objects; each sample prompt word is an image description text with the presentation state of the sample object as the description content, and any sample prompt word belongs to one of the two prompt words, the presentation states described by the two prompt word types are mutually exclusive, and the sample prompt words belonging to the same prompt word type have similar semantics. Each sample prompt word is obtained based on a target keyword and other non-keyword combinations, and the target keyword is a keyword used to characterize a presentation state of the sample object; the initial second modal feature of each sample prompt word is the vector feature of the sample prompt word.

[0078] In the present application, a pre-trained model can be pre-trained using images containing other objects so that a feature expression of the image can be formed in the pre-trained model, and a matching mapping relationship is established between the feature expression of the image and a prompt word with similar semantic features, thereby obtaining a pre-trained contrastive learning model. In other words, the above training of the pre-trained model enables the pre-trained contrastive learning model to generate vector features about the image, and to determine the prompt word that matches the image by comparing the similarity between the vector features about the image and the vector features about the prompt word.

[0079] Therefore, after obtaining the sample image and each predetermined sample prompt word, the obtained sample image and each predetermined sample prompt word can be input into the pre-trained contrastive learning model so that the contrastive learning model can generate the vector features of the sample image and obtain the first modal features of the sample image. At the same time, the current second modal features of each of the above sample prompt words are determined. It should be noted that when the sample prompt word is input into the contrastive learning model for the first time, the generated vector features of the sample prompt word are directly determined as the initial second modal features of the sample prompt word. Thus, the initial second modal features of the sample prompt word are used as the current second modal features of the sample prompt word. Then, the similarity between the first modal features of the sample image and the second modal features of each sample prompt word is calculated to obtain the similarity of the sample image to each sample prompt word.

[0080] Among them, each sample prompt word is an image description text with the presentation status of the sample object as the description content, and any sample prompt word belongs to one of two prompt word types. The presentation states described by the two prompt word types are mutually exclusive, and the sample prompt words belonging to the same prompt word type have similar semantics.

[0081] For example, the sample prompt words include: "having a tattoo" and "not having a tattoo," "having a cyan tattoo" and "not having a cyan tattoo," and "having a patterned tattoo" and "not having a patterned tattoo." Among them, "having a tattoo," "having a cyan tattoo," and "having a patterned tattoo" all belong to the sample prompt words describing a presentation state in which a tattoo is present. That is, the three sample prompt words belong to the same prompt word type W, and the semantics of the sample prompt words in this prompt word type W are similar. "not having a tattoo," "not having a cyan tattoo," and "not having a patterned tattoo" all belong to the sample prompt words describing a presentation state in which a tattoo is absent. That is, the three sample prompt words belong to the same prompt word type M, and the semantics of the sample prompt words in this prompt word type M are similar. In summary, the presentation states described by the prompt word type W and the prompt word type M are mutually exclusive.

[0082] In this way, when using each of the above-mentioned predetermined sample prompt words to identify the sample image, it is possible to determine which prompt word's presentation state the sample object in the sample image is more consistent with based on the prompt words that describe mutually exclusive presentation states. Thus, the presentation state described by the prompt word that is more consistent is used as the presentation state of the sample image for the sample object to achieve recognition of the sample image.

[0083] Optionally, a target keyword and other non-keywords that can be combined with the target keyword can be pre-set for a presentation state of the sample object. In this way, through the combination of the target keyword and other non-keywords, an image description text with the presentation state of the sample object as the description content can be obtained as a sample prompt word, thereby determining multiple sample prompt words.

[0084] The combination of the above-mentioned target keywords and other non-keywords may include:

[0085] For each target keyword, a corresponding target keyword is set for each sample object in different presentation states. For each target keyword, a random combination of non-keywords and the target keyword is used to generate multiple sample prompts. For example, if the sample object is a tattoo, its presentation states can include: tattoo presence and tattoo absence. In this case, the target keywords that can be set are tattoo presence and tattoo absence.

[0086] For each presentation state of the same sample object, a corresponding target keyword is set. Multiple sample prompt words are generated by randomly combining the target keyword with other non-keywords. For example, if the sample object is a tattoo, its presentation states can include: tattoo presence or absence. In this case, the target keyword that can be set is tattoo presence or absence.

[0087] The embodiment of the present application does not limit the number of target keywords in the multiple sample prompt words, nor does it limit the number of other non-keywords.

[0088] Compared to the prior art method of manually designing prompt words, the combination of the target keyword and other non-keywords to obtain sample prompt words simplifies the tedious process of manually designing prompt words and reduces the possibility of low accuracy of the prompt words involved in manually designing prompt words. In addition, the sample prompt words obtained by the above combination method provided by the embodiments of the present application have a high accuracy rate. Therefore, during model training, a comparative learning model with high recognition efficiency can be trained using a small number of sample prompt words and sample images.

[0089] Considering that different data categories, such as text data and image data, have corresponding vector features of different dimensions, in order to facilitate the subsequent determination of the similarity of each prompt word with respect to the sample image, the vector features corresponding to data of different data categories can be unified into the same predetermined dimension. Therefore, after obtaining the sample image, vector features of the predetermined dimension can be generated for the sample image to obtain the first modal feature.

[0090] Optionally, after acquiring the sample image, the contrastive learning model calls the image encoder to extract image features of the sample image, and converts the extracted image features into vector features of a predetermined dimension to obtain the first modal features of the sample image.

[0091] At the same time, a text encoder is invoked to extract text features for each sample prompt word, thereby converting the text features of each sample prompt word into vector features of a predetermined dimension, thereby obtaining initial second modal features for each sample prompt word. The vector features representing text features and image features using a feature matrix are not specifically limited in this embodiment of the present application.

[0092] Afterwards, after obtaining the first modal feature and the second modal feature, the similarity between the first modal feature and the second modal feature corresponding to each sample prompt word can be calculated to obtain the similarity of the sample image for each sample prompt word, wherein the feature score can be used to represent the similarity corresponding to each sample prompt word. This embodiment of the present application does not make any specific limitations on this.

[0093] It should be noted that the embodiments of the present application involve a text-image pre-training model based on contrastive learning, and the main ability of this type of model is to align image features and text features in the same feature space (dimension). Therefore, in theory, only the trained model layer in this type of model can be used to convert image or text features to vector features, that is, the image features are converted to the first modality features through the image encoder in this type of model, and the text features are converted to the second modality features through the text encoder in this type of model. Other untrained methods cannot be used, otherwise, the obtained first modality features and second modality features will not be in the same feature space from a mathematical point of view.

[0094] Based on this, optionally, the sample image is input into the image encoder of the pre-trained contrastive learning model so that the image encoder extracts the image features of the sample image and converts the extracted image features to obtain the first modal features of the sample image; the sample prompt word is input into the text encoder of the pre-trained contrastive learning model so that the text encoder extracts the text features of the sample prompt word and converts the extracted text features to obtain the initial second modal features of the sample prompt word.

[0095] In this way, since the generation method of the first modal feature and the second modal feature is pre-trained, the obtained first modal feature and the second modal feature are in the same feature space from a mathematical perspective, that is, the modal features obtained by the image encoder and text encoder using the above-mentioned pre-trained contrastive learning model are in the same predetermined dimension.

[0096] Optionally, cosine similarity is used to calculate the distance between the first modal feature and the second modal feature corresponding to each sample prompt word. If the distance between the first modal feature and the second modal feature corresponding to the sample prompt word is closer, the similarity between the two modal features is higher. Conversely, if the distance between the first modal feature and the second modal feature corresponding to the sample prompt word is farther, the similarity between the two modal features is lower.

[0097] S303: Determine a loss value corresponding to the contrastive learning model based on the similarity of the sample image to each sample prompt word and the label of the sample image.

[0098] In this application, since the sample images are provided with labels representing the presentation state of the sample objects, that is, the presentation state of the sample objects represented by the labels of the sample images is the actual presentation state of the sample objects in the sample images. Therefore, after obtaining the similarity of the sample images for each sample prompt word, the loss value corresponding to the contrastive learning model can be determined based on this similarity and the labels.

[0099] S304: When it is determined based on the loss value that the contrastive learning model does not meet the model convergence condition, parameters of the designated feature portion of the current second modal feature of each sample prompt word are adjusted, and the process returns to step S301;

[0100] The designated feature portion in the current second modal feature of each sample prompt word is the feature portion corresponding to other non-keywords in the current second modal feature of the sample prompt word.

[0101] In this application, since the target keyword is fixed for different sample prompt words, what changes are other non-keywords, that is, the generation of vector features corresponding to the target keyword is relatively stable, while for other non-keywords, due to the different combinations of other non-keywords, the generated vector features are different and the stability is poor. Therefore, the loss value of the comparative learning model during training or inference is generated in the process of generating the feature part corresponding to other non-keywords in the second modal feature of the prompt word.

[0102] After determining the loss value corresponding to the contrastive learning model, it can be further determined whether the contrastive learning model meets the model convergence condition, that is, whether the determined loss value reaches a predetermined loss value.

[0103] When it is determined based on the loss value that the contrastive learning model does not meet the model convergence conditions, the parameters of the specified feature part in the current second modal feature of each sample prompt word are adjusted, that is, the feature part corresponding to other non-keywords in the current second modal feature of the sample prompt word is adjusted, and the process returns to step S301, and continues to use a model training method provided in an embodiment of the present application based on the next sample image to adjust the parameters of the specified feature part in the current second modal feature of each sample prompt word, so as to reduce the influence of other non-keywords in the prompt word on the accuracy of the recognition result, thereby achieving optimized training of the contrastive learning model.

[0104] Optionally, when the contrastive learning model is determined to not meet the model convergence conditions based on the loss value, parameter adjustments are made to a specified feature portion of the current second modal features of each sample prompt word according to a predetermined magnitude. In other words, when it is determined that the contrastive learning model does not meet the model convergence conditions, the user can adjust the parameters according to a predetermined parameter adjustment magnitude, for example, by magnitudes represented by values ​​such as 0.5, 1, or 2.

[0105] Optionally, when it is determined based on the loss value that the contrastive learning model does not meet the model convergence conditions, parameter adjustment is performed on the specified feature portion of the current second modal feature of each sample prompt word in a predetermined direction. That is to say, when it is determined that the contrastive learning model does not meet the model convergence conditions, the user can adjust the parameters in a predetermined parameter adjustment direction, for example, in a direction of increasing the parameter value, or in a direction of decreasing the parameter value. The specific adjusted parameter value can be any positive number, and this embodiment of the application does not make any specific limitation on this.

[0106] Optionally, when the contrastive learning model is determined to not meet the model convergence conditions based on the loss value, parameter adjustments are performed on a specified feature portion of the current second modal features of each sample prompt word in a predetermined direction and by a predetermined magnitude. That is, when it is determined that the contrastive learning model does not meet the convergence conditions, the user can adjust the parameters in a predetermined parameter adjustment direction and by a predetermined parameter adjustment magnitude. For example, based on the specified feature portion of the current second modal features of each sample prompt word, parameter adjustments are performed in a direction of increasing the parameter value and by a magnitude represented by 0.5.

[0107] When the contrastive learning model is judged to meet the model convergence conditions based on the loss value, the training of the model is terminated.

[0108] Optionally, in order to improve the training efficiency of the model training, the model training end condition can be set to when the model training times reach a predetermined number of times. That is, when the model training times reach the predetermined number of times, the model training ends.

[0109] As can be seen from the above, in the model training method provided in the embodiment of the present application, sample prompt words can be set in advance for the sample objects before model training. Each sample prompt word is an image description text with the presentation state of the sample object as the description content, and any sample prompt word belongs to one of the two prompt word types. The presentation states described by the two prompt word types are mutually exclusive, and the sample prompt words belonging to the same prompt word type have similar semantics. Each sample prompt word is obtained based on a target keyword and other non-keyword combinations, wherein the above-mentioned target keyword is a keyword used to characterize a presentation state of the sample object. Considering that other non-keywords in the prompt word have a significant impact on the second modal features of the entire prompt word, the contrastive learning model is relatively stable in generating vector features corresponding to the target keyword in the second modal features of the prompt word. However, for other non-keywords, the generated vector features may be different if the non-keywords are the same but combined differently, or if the non-keywords are different. In other words, the contrastive learning model is unstable in generating vector features corresponding to other non-keywords in the second modal features of the prompt word, and the generated vector features are quite different. Therefore, by determining the similarity of the sample images for each sample prompt word and the difference in the pre-set labels of the sample images themselves, the loss value of the current second modal feature of each sample prompt word generated by the contrastive learning model is determined. Therefore, when the contrastive learning model is judged to not meet the model convergence conditions based on the loss value, the parameters of the feature portion corresponding to other non-keywords in the current second modal features of each sample prompt word are adjusted to reduce the impact of other non-keywords in the prompt word on the accuracy of the recognition result, thereby ensuring the accuracy of image recognition using the contrastive learning model.

[0110] Furthermore, combining a target keyword with other non-keywords to create sample prompts reduces the tedious process of manually designing prompts and mitigates the potential for low accuracy associated with manually designed prompts. Furthermore, the sample prompts created through this combination method have a high accuracy rate. Therefore, during model training, a comparative learning model with high recognition performance can be trained using only a small number of sample prompts and sample images.

[0111] Optionally, in one implementation, step S303, determining the loss value corresponding to the contrastive learning model based on the similarity of the sample image to each sample prompt word and the label of the sample image, may include the following steps:

[0112] Step A1: determining the confidence level of the sample image for each sample prompt word based on the similarity level of the sample image for each sample prompt word;

[0113] The confidence level of the sample image for any sample prompt word is used to characterize the presentation state of the sample object in the sample image, and is the predicted probability of the presentation state described in the sample prompt word.

[0114] Step A2: Based on the confidence of each sample prompt word in the sample image and the label of the sample image, determine the loss value corresponding to the contrastive learning model.

[0115] In this implementation, for each sample prompt word, after determining the similarity of the sample image to the sample prompt word, the confidence score of the sample prompt word can be further determined to determine the predicted probability that the presentation state of the sample object in the sample image is the presentation state described by the sample prompt word. In other words, the higher the confidence score corresponding to the sample prompt word, the higher the predicted probability that the presentation state of the sample object in the sample image is the presentation state described by the sample prompt word.

[0116] In this way, based on the confidence level of each sample prompt word in the sample image and the label of the sample image, the loss value corresponding to the contrastive learning model can be determined. The smaller the loss value determined, the closer the presentation state of the sample object in the sample image is to the presentation state described by the sample prompt word.

[0117] Among them, the label set for the sample image to represent the presentation state of the sample object can be understood as that the label of the sample image represents the actual state presented by the sample object in the sample image. Therefore, the corresponding confidence level can be set directly based on the actual state presented by the sample object. For example, if the sample object is a tattoo, one presentation state represented by the label is the presence of a tattoo, and the corresponding confidence level of the presentation state represented by the label is set to 1.

[0118] Optionally, in one implementation, the confidence level of each sample prompt word in the sample image and the confidence level corresponding to the label of the sample image are input into a predetermined loss function to calculate the loss value corresponding to the contrastive learning model.

[0119] Among them, the above-mentioned predetermined loss function may include a cross-entropy loss function, etc., which is not specifically limited in this embodiment of the present application.

[0120] In this implementation, the confidence score is used to digitize the probability that the presentation state of the sample object corresponding to the sample image is the presentation state described in each sample prompt word, so as to facilitate matching the sample image with the corresponding sample prompt word.

[0121] Optionally, in one implementation, the above step A1, determining the confidence level of the sample image for each sample prompt word based on the similarity level of the sample image for each sample prompt word, may include the following steps:

[0122] Step A11: normalize the similarity of the sample image for each sample prompt word to obtain the confidence of the sample image for each sample prompt word.

[0123] In this implementation, after obtaining the similarity of each sample prompt word, normalization processing can be performed on the similarity corresponding to each sample prompt word to obtain the confidence corresponding to the sample prompt word.

[0124] Optionally, for each sample prompt word, a confidence level corresponding to the sample prompt word is calculated using a predetermined formula; wherein the predetermined formula includes:

[0125]

[0126] σ(z i ) represents the confidence level corresponding to the i-th sample prompt word of the sample object, K represents the total number of sample prompt words of the sample object, z i Characterize the feature score corresponding to the i-th sample prompt word, z j Characterize the feature score corresponding to the j-th sample prompt word.

[0127] The feature score corresponding to each sample prompt word is used to represent the similarity between the sample prompt word and the sample image, that is, the similarity corresponding to each prompt word in the embodiment of the present application.

[0128] This implementation method normalizes the similarity corresponding to each sample prompt word to a specific interval, such as [0, 1] or [-1, 1], so that the data distribution has statistical significance and the probability distribution of similarity is more statistically intuitive, thereby improving the comparability of the similarity corresponding to each sample prompt word.

[0129] Optionally, in one implementation, when it is determined based on the loss value that the contrastive learning model does not meet the model convergence condition, step S304 may include the following steps:

[0130] Step B: When it is determined based on the loss value that the contrastive learning model does not meet the model convergence conditions, the gradient value of the feature vector corresponding to other non-keywords in the current second modal feature of each sample prompt word is calculated based on the loss value; and the gradient value is used to adjust the feature vector corresponding to the specified feature part in the current second modal feature of the sample prompt word.

[0131] In this implementation, considering that for different sample prompt words, the target keywords are fixed, and what changes are other non-keywords, that is, the loss value of the contrastive learning model during the training and inference process is caused by the changes in other non-keywords. Therefore, when it is judged based on the loss value that the contrastive learning model does not meet the model convergence conditions, the parameters of the specified feature part in the current second modal feature of each sample prompt word can be adjusted based on the loss value. Specifically: first, based on the loss value, the gradient value of the vector feature corresponding to other non-keywords in the current second modal feature of each sample prompt word is calculated, and then, the calculated gradient value is used to adjust the feature vector corresponding to the specified feature part in the current second modal feature of the sample prompt word.

[0132] In this way, the determined loss value can provide a specific adjustment amplitude and adjustment direction for subsequent parameter adjustments. When adjusting parameters, there is no need to adjust according to a predetermined direction and / or predetermined amplitude, so as to improve the flexibility of parameter adjustment, thereby reducing the number and frequency of parameter adjustments, and further improving the efficiency of model training.

[0133] Optionally, in one implementation, each sample prompt word's current second modal feature carries a feature tag, where the feature tag includes: other non-keyword tags and target keyword tags. Thus, by identifying whether the carried feature tag is a other non-keyword tag, it is determined whether the feature portion is a designated feature portion corresponding to other non-keywords in the current second modal feature.

[0134] Optionally, in one implementation, the representation of the current second modal features of each sample prompt word obtained by calling a text encoder for conversion is in the form of a feature matrix group, which includes a first matrix of feature vectors corresponding to the target keywords of the sample prompt word and a second matrix of feature vectors corresponding to other non-keywords of the sample prompt word. Since the matrix positions of the first matrix and the second matrix are different, the feature vectors corresponding to other non-keywords in the generated current second modal features can be determined by determining the positions of the matrices. In other words, the feature vectors corresponding to the specified feature portion of the current second modal features of the sample prompt word that require parameter adjustment can be determined.

[0135] In this implementation, the feature vector corresponding to the specified feature part in the current second modal feature of the sample prompt word is adjusted to reduce the impact of non-keywords in the prompt word on the accuracy of the recognition result, so as to ensure the accuracy of image recognition using the contrastive learning model.

[0136] Optionally, a back propagation algorithm is used to calculate the gradient value of the feature vector corresponding to the non-keyword in each sample prompt word based on the loss value.

[0137] Below, in conjunction with a specific embodiment, a model training method provided in an embodiment of the present application is specifically introduced.

[0138] Conventional deep learning models require large amounts of data to be labeled. In practice, the data required for recognition often has a long-tail distribution, meaning it accounts for a small portion of the total data and is difficult to collect. This leads to high data labeling costs and slow model iteration cycles. Therefore, considering that contrastive learning models such as CLIP can perform image recognition without collecting data, they can be used for zero-shot prediction, enabling automatic image recognition.

[0139] In related technologies, when training the above-mentioned contrastive learning model, the text encoder and image encoder contained in the contrastive learning model need to be trained simultaneously. During this period, the number of training samples and the training time required are greatly increased, resulting in a decrease in the training efficiency of the model.

[0140] For example, Figure 4 The figure shows a flow chart of an existing contrastive learning model training method. In the existing contrastive learning model training process, it is necessary to obtain the key prompt words with airplane, butterfly and pizza as the objects respectively, combine the key prompt words with other non-key words, and then output each prompt word obtained after the above combination to the Text Encoder of the CLIP model to obtain the Text features (text features) corresponding to each prompt word; then, input the image into the Image Encoder of the CLIP model to obtain the Image features (image features) corresponding to the image; after that, calculate the Similarityscores (similarity coefficient) of the text features and image features as the similarity between the two. By Figure 4 It can be seen that when training the existing contrastive learning model, the text encoder and the image encoder need to be trained at the same time, which makes the training efficiency of the model low.

[0141] Based on this, an embodiment of the present application provides a model training method to achieve optimized training of the entire model by optimizing the feature vectors of other non-keywords in the prompt words. In this way, there is no need to optimize the text encoder and image encoder, and the model can obtain a higher recognition accuracy, thereby improving the training efficiency of the model.

[0142] The following is a detailed introduction to a model training method provided by an embodiment of the present application, using a specific example. In this specific example, the sample object is a tattoo, and the pre-trained contrastive learning model is a CLIP model. The specific process includes the following steps:

[0143] Step 1: Set the initial keywords, combine the keywords with the learnable vectors, and convert them into text multimodal features through a text encoder; wherein the above-mentioned learnable vectors are the non-keywords in the embodiment of the present application, and the above-mentioned initial keywords are the target keywords in the embodiment of the present application.

[0144] Step 2: Input the image and its corresponding label, and convert the image into image multimodal features through the image encoder.

[0145] Step 3: Use the back propagation algorithm to tune the parameters corresponding to the learnable vector in the prompt word; wherein, the parameters corresponding to the learnable variables in the prompt word are the specified feature parts in the current second modal features of the sample prompt word in the embodiment of the present application.

[0146] In this embodiment, feature engineering is used to tune the parameters corresponding to the learnable vectors in the prompt words. That is, when the keywords "tattoo" and "not tattoo" are given in the prompt words, other non-keywords are filled with learnable vector parameters. In this way, the filling vector parameters can be tuned and learned by using a small number of samples to obtain the best model effect.

[0147] The following is a detailed introduction to steps one to three:

[0148] The step 1 of obtaining multimodal features of text may include the following steps:

[0149] Step D1: setting a plurality of keywords for tattoo classification. For example, when the image has a tattoo, the prompt word is set to "tattoo"; when the image does not have a tattoo, the prompt word is set to "not tattoo".

[0150] Step D2: Add multiple learnable vectors before and after the prompt word, and combine them with the keyword in step D1 to form a complete prompt word;

[0151] That is, a plurality of complete prompt words are obtained as the plurality of prompt words by combining a keyword with a plurality of other non-keywords.

[0152] Step D3: Pass the combined multiple prompt words through the text encoder of the CLIP model to obtain text features, and then pass the text features through the text multimodal network to convert the text features into text multimodal features. Here, the text multimodal features corresponding to each text are feature matrices of size [1, 512].

[0153] For example, Figure 5 This is a flowchart for obtaining multimodal features of text. Figure 5 The learnable context in the model is the learnable vector. Airplane, butterfly, and pizza can all be sample objects. After combining the learnable vector with any sample object, a complete prompt word is obtained. This complete prompt word is then input into the text encoder of the CLIP model to obtain the multimodal text features corresponding to the input prompt word.

[0154] The conversion of the image into image multimodal features in step 2 may include the following steps:

[0155] Step E1: Randomly extract multiple images from the dataset to form a batch.

[0156] The dataset described above is a collection of sample images in the embodiments of this application. Each image in the dataset depicts a tattoo and is assigned a tattoo-related label. Therefore, a batch of images can be randomly selected from the dataset to form a set of images used for model training. This allows the model to be trained using these images and the resulting combination of prompt words.

[0157] Step E2: scaling the image to a fixed size; wherein the fixed size may be 224×224 or 336×336, etc., which is not specifically limited in the embodiment of the present application.

[0158] Step E3: Use the image encoder of the CLIP model to extract the image features, and then pass the image features through the image multimodal network to convert the image features into image multimodal features. Here, the image multimodal features corresponding to each image are feature matrices of size [1, 512].

[0159] In this way, multiple text multimodal features and image multimodal features are sequentially multiplied and added, and then divided by the vector length of each text multimodal feature and image multimodal feature to calculate the similarity between the text multimodal features and the image multimodal features, and obtain the feature scores corresponding to the features of the image and each text feature; wherein the above feature scores are used to represent the similarity between the image and each text feature.

[0160] Then, define the feature score as z and perform softmax (normalization) operation on all feature scores:

[0161]

[0162] σ(z i ) represents the confidence level of the i-th prompt word about tattoos, K represents the total number of sample prompt words about tattoos, z i Characterize the feature score corresponding to the i-th prompt word, z j Represents the feature score corresponding to the j-th prompt word.

[0163] Through the above formula, the feature score of each text in the image can be mapped to a confidence score between 0 and 1 as the predicted confidence score.

[0164] In particular, for step 3 above, using the back propagation algorithm to tune the parameters corresponding to the learnable vector in the prompt word may include the following steps:

[0165] The cross-entropy loss function is used to calculate the loss value between the confidence score predicted by the model and the label of the image itself. The backpropagation algorithm is used to calculate the gradient value of the learnable vector in the prompt word based on the loss value and update the parameter value of the learnable vector.

[0166] That is to say, the predicted confidence score and the confidence score corresponding to the label of the image itself are input into the cross-entropy loss function calculation model, and the loss value between the predicted confidence score and the label of the image itself is calculated. Then, the backpropagation algorithm is used to calculate the gradient value of the learnable vector in the prompt word based on the above loss value, and the obtained gradient value is used to tune the text features of the learnable vector.

[0167] After tuning the parameters of the text features of the learnable vectors, repeat steps 2 and 3 above until all images in the dataset are extracted, and perform gradient descent calculations to optimize the contrastive learning model.

[0168] The above steps can be considered as one round of model training. Repeat multiple rounds of training until the preset number of training rounds is reached or the loss value reaches a preset value.

[0169] Compared with the use of traditional models and traditional training methods, this embodiment only requires a small number of samples to obtain a higher model accuracy, greatly reducing the difficulty of data collection in scenarios with long-tail distribution data characteristics such as audit business. Moreover, compared with the use of manually designed prompt words, the prompt words obtained by feature engineering can obtain a higher accuracy rate, while also greatly reducing the tedious process of manually designing prompt words.

[0170] The following is a detailed introduction to an image recognition method provided in an embodiment of the present application.

[0171] Among them, an image recognition method provided in an embodiment of the present application can be applied to an electronic device, and the electronic device is specifically used for image recognition. In specific applications, the electronic device can be a smart phone, a tablet computer, etc., which is all reasonable.

[0172] An image recognition method provided in an embodiment of the present application may include the following steps:

[0173] Cutting out a region of interest from the target image to obtain a target sub-image; wherein the region of interest is a carrier region where the target object to be identified is located in the target image when the target object exists in the target image;

[0174] The target sub-image and the prompt phrase are input into a pre-trained contrastive learning model so that the contrastive learning model generates the vector features of the target sub-image to obtain the first modal features, and generates the initial modal features of each prompt word in the prompt phrase, and, for each prompt word, splices the target sub-feature in the initial modal features of the prompt word with the designated sub-feature corresponding to the target sample prompt word to obtain the second modal features of the prompt word; calculates the similarity between the first modal features of the target sub-image and the second modal features of each prompt word, and obtains the similarity of the target sub-image to each prompt word; wherein, the prompt Each prompt word in the phrase is an image description text that uses the presentation state of the target object as a description content, and the presentation states described by different prompt words in the same prompt word group are mutually exclusive. The contrastive learning model is trained based on a model training method provided in an embodiment of the present application. The target sample prompt word is a sample prompt word among each sample prompt word whose other non-keywords match the other non-keywords of the prompt word. The designated sub-feature is the feature content of the designated feature part of the current second modal feature of the target sample prompt word when the contrastive learning model is trained. The target sub-feature is a feature that characterizes the target keyword.

[0175] Determine a recognition result corresponding to the target sub-image based on the similarity of each prompt word; wherein the recognition result is a result regarding the presentation state of the target object;

[0176] Based on the recognition result corresponding to the target sub-image, a presentation state of the target image with respect to the target object is determined.

[0177] As can be seen from the above, in the image recognition method provided by the embodiment of the present application, after a target sub-image is captured from a target image, the target sub-image and a prompt phrase associated with the target object are input into a pre-trained contrastive learning model. Accordingly, because the contrastive learning model reduces the influence of other non-keywords in the prompt phrase on the accuracy of the recognition result during the training process, when using the contrastive learning model to recognize the target object, for each prompt word in the prompt phrase, the target sub-feature of the prompt word is concatenated with the designated sub-feature corresponding to the target sample prompt word to obtain the second modal feature of the prompt word. That is, the feature representing the target keyword of the prompt word is concatenated with the feature content of the designated feature portion of the current second modal feature of the target sample prompt word when the contrastive learning model is trained, to obtain the second modal feature of the prompt word. Thus, the similarity between the first modal feature of the target sub-image and the second modal feature of each prompt word is calculated to obtain the similarity of the target sub-image with respect to each prompt word. Based on the obtained similarity, the presentation state of the target image with respect to the target object is determined. Through this solution, we can focus on the characteristic content of the keyword part in each prompt word in the prompt phrase group and use the specified characteristic part in the target sample prompt word to reduce the influence of non-keywords in the image recognition process, thereby achieving accurate recognition of the target object.

[0178] An image recognition method provided by an embodiment of the present application is described below with reference to the accompanying drawings.

[0179] Figure 6 A flowchart of an image recognition method provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, the method may include the following steps:

[0180] S601: Cut out a region of interest from a target image to obtain a target sub-image.

[0181] The region of interest is the carrier region where the target object is located in the target image when the target object to be identified exists in the target image.

[0182] In this application, it is taken into consideration that when the target object to be identified exists in the target image, the carrier area where the target object is located in the target image will not change much. For example, the carrier area corresponding to the trademark is the edge area of ​​the overall image, the carrier area corresponding to the watch is the arm area, and the carrier area corresponding to the car is the ground area.

[0183] Therefore, when performing image recognition, the carrier area can be determined from the target image as the region of interest, and the region of interest can be intercepted to obtain a target sub-image. In other words, the image area contained in the target sub-image is the region of interest.

[0184] Furthermore, a target image may contain one or more regions of interest. When there is one region of interest, one target sub-image is captured; correspondingly, when there are multiple regions of interest, multiple target sub-images are captured. The same method can be used to process each target sub-image to obtain a corresponding recognition result.

[0185] Optionally, in one embodiment, the target object is a tattoo, and the carrier area is a human body area.

[0186] In this embodiment, when the target object is a tattoo, the carrier region where the tattoo exists is a human body region, and therefore, the captured target sub-image is an image containing a human body.

[0187] It should be noted that the acquisition, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution of this application are all performed with the user's authorization. Furthermore, the images of human bodies in this embodiment are not images of a specific user and do not reflect the personal information of a specific user. Furthermore, the two-dimensional human body images in this embodiment are from a public dataset.

[0188] S602: Input the target sub-image and the prompt phrase into a pre-trained contrastive learning model, so that the contrastive learning model generates a vector feature of the target sub-image to obtain a first modal feature, and generates an initial modal feature for each prompt word in the prompt phrase. Furthermore, for each prompt word, the target sub-feature in the initial modal feature of the prompt word is concatenated with a designated sub-feature corresponding to the target sample prompt word to obtain a second modal feature of the prompt word. The similarity between the first modal feature of the target sub-image and the second modal feature of each prompt word is calculated to obtain a similarity of the target sub-image to each prompt word.

[0189] Among them, each prompt word in the prompt phrase group is an image description text with the presentation state of the target object as the description content, and the presentation states described by different prompt words in the same prompt phrase group are mutually exclusive. The contrastive learning model is trained based on a model training method provided in an embodiment of the present application. The target sample prompt word is a sample prompt word among each sample prompt word, whose other non-keywords match the other non-keywords of the prompt word. The designated sub-feature is the feature content of the designated feature part in the current second modal feature of the target sample prompt word when the contrastive learning model is trained. The target sub-feature is the feature that characterizes the target keyword.

[0190] In this application, a prompt phrase describing the target object is pre-obtained. Each prompt phrase in the prompt phrase is an image description text that describes the target object's presentation state. Moreover, the presentation states described by different prompt phrases in the same prompt phrase are mutually exclusive. For example, when the target object is a tattoo, the target object's presentation state can include the presence of a tattoo and the absence of a tattoo, and the corresponding prompt phrases can be "the image has a person with a tattoo" and "the image does not have a person with a tattoo"; for another example, when the target object is a tattoo, the target object's presentation state can include the presence of a dark tattoo and the absence of a dark tattoo, and the corresponding prompt phrases can be "the image has a person with a dark tattoo" and "the image does not have a person with a dark tattoo"; for another example, when the target object is a cat, the target object's presentation state can include the presence of a cat and the absence of a cat, and the corresponding prompt phrases can be "the image has a cat" and "the image does not have a cat", etc.

[0191] In this way, by using different prompt words in the prompt phrase group to describe the mutually exclusive presentation states of the target object, the presentation state of the target object in the target sub-image can be determined, and which prompt word in the prompt words is more consistent with the presentation state described. Thus, the presentation state described by the prompt word that is more consistent is used as the presentation state of the target sub-image for the target object, and the recognition result corresponding to the target sub-image is obtained. Furthermore, based on the presentation state of the target sub-image for the target object, the presentation state of the overall target image for the target object can be further determined to realize automatic recognition of the image.

[0192] Therefore, the acquired target sub-image and prompt phrase can be input into a contrastive learning model pre-trained using a model training method provided in an embodiment of the present application, so that the contrastive learning model generates vector features of the target sub-image, obtains the first modal features, and generates the initial modal features of each prompt word in the prompt phrase.

[0193] Because the trained contrastive learning model stores various designated sub-features—that is, it stores the feature content of the designated feature portion of the current second modal features of each sample prompt word when the contrastive learning model is trained—for each prompt word, a sample prompt word whose other non-keywords match the other non-keywords of the prompt word can be selected from each sample prompt word as the target sample prompt word. Thus, the target sub-feature in the initial modal features of the prompt word is concatenated with the designated sub-feature corresponding to the selected target sample prompt word to obtain the second modal features of the prompt word. In this way, the trained designated sub-feature replaces the vector features corresponding to other non-keywords in the initial modal features, thereby reducing the impact of the non-keywords in the prompt word on the accuracy of the recognition results.

[0194] Afterwards, similarities between the first modal feature of the target sub-image and the second modal feature of each prompt word are calculated to obtain the similarity of the target sub-image to each prompt word in the prompt word group.

[0195] In this way, by focusing on the characteristic content of the keyword part in each prompt word in the prompt phrase group and using the specified characteristic part in the target sample prompt word, the influence of non-keywords in the image recognition process is reduced, thereby achieving accurate recognition of the target object.

[0196] The specific implementation of step S602 generating the vector features of the target sub-image, obtaining the first modal features, and generating the initial modal features of each prompt word in the prompt phrase group is similar to the specific implementation of step S302 and will not be repeated here.

[0197] Considering that the target sub-images captured from different target images, or the target sub-images captured from the same target image, may have different image sizes. However, for images with the same image size, the dimensions of the vector features that can be converted are also the same. Therefore, in order to ensure that any target sub-image obtained can be converted into image features of the same dimension, and then further converted into vector features of a predetermined dimension, the target sub-image can be scaled to a predetermined size before generating the first modal feature for the target sub-image, so as to be used in the vector feature generation process. Based on this, optionally, after acquiring the target sub-image, the target sub-image is scaled to a predetermined size, and after image feature extraction is performed on the target sub-image of the predetermined size, the extracted image features are converted into vector features of a predetermined dimension to obtain the first modal feature.

[0198] Exemplarily, after acquiring the target sub-image, the image size of the target sub-image is scaled to a fixed size, for example, 224×224, or 336×336, and then the scaled target sub-image is input into a module for generating the first modal feature to generate the first modal feature of the target sub-image.

[0199] In this implementation, by converting the dimensions of the image features of the target sub-image to obtain vector features of the target sub-image in a predetermined dimension, it is convenient to determine the similarity of each prompt word to the target sub-image.

[0200] S603: Determine the recognition result corresponding to the target sub-image based on the similarity of each prompt word.

[0201] The recognition result is a result about the presentation status of the target object.

[0202] In the present application, after obtaining the similarity corresponding to each prompt word, the degree of similarity between the presentation state of the target object corresponding to the target sub-image and the presentation state described by the prompt word can be determined. The higher the similarity, the higher the possibility that the presentation state of the target object in the target sub-image is the presentation state described by the prompt word. Therefore, after determining the similarity corresponding to each prompt word, the presentation state of the target object described by the prompt word with high similarity can be determined as the presentation state of the target object corresponding to the target sub-image, thereby determining the recognition result corresponding to the target sub-image.

[0203] S604: Determine the presentation state of the target image with respect to the target object based on the recognition result corresponding to the target sub-image.

[0204] In this application, it is considered that a target image may contain multiple regions of interest, that is, a target image may contain at least one target sub-image. Therefore, after obtaining the recognition result for the target sub-image, the presentation state of the at least one target sub-image with respect to the target object can be determined based on the above recognition result. Thus, the presentation state of the entire target image with respect to the target object can be determined to obtain the recognition result corresponding to the target image.

[0205] As can be seen from the above, in the image recognition method provided by the embodiment of the present application, after a target sub-image is captured from a target image, the target sub-image and a prompt phrase associated with the target object are input into a pre-trained contrastive learning model. Accordingly, because the contrastive learning model reduces the influence of other non-keywords in the prompt phrase on the accuracy of the recognition result during the training process, when using the contrastive learning model to recognize the target object, for each prompt word in the prompt phrase, the target sub-feature of the prompt word is concatenated with the designated sub-feature corresponding to the target sample prompt word to obtain the second modal feature of the prompt word. That is, the feature representing the target keyword of the prompt word is concatenated with the feature content of the designated feature portion of the current second modal feature of the target sample prompt word when the contrastive learning model is trained, to obtain the second modal feature of the prompt word. Thus, the similarity between the first modal feature of the target sub-image and the second modal feature of each prompt word is calculated to obtain the similarity of the target sub-image with respect to each prompt word. Based on the obtained similarity, the presentation state of the target image with respect to the target object is determined. Through this solution, we can focus on the characteristic content of the keyword part in each prompt word in the prompt phrase group and use the specified characteristic part in the target sample prompt word to reduce the influence of non-keywords in the image recognition process, thereby achieving accurate recognition of the target object.

[0206] Optionally, in one implementation, the above step S604, determining the presentation state of the target image with respect to the target object based on the recognition result corresponding to the target sub-image, may include the following steps:

[0207] Step F1: In response to the number of the target sub-image being one, the presentation state represented by the recognition result corresponding to the target sub-image is used as the presentation state of the target image for the target object.

[0208] In this implementation method, if the number of captured target sub-images is one, it means that there is only one area in the target image where the target object exists. At this time, the presentation state represented by the recognition result corresponding to the target sub-image can be directly used as the presentation state of the overall target image for the target object to realize automatic image recognition.

[0209] In this implementation, when the number of target sub-images is one, the recognition result corresponding to the obtained target sub-image can be directly used as the recognition result corresponding to the entire target image, thereby improving the efficiency of automatic image recognition.

[0210] Optionally, in one implementation, the above step S604, determining the presentation state of the target image with respect to the target object based on the recognition result corresponding to the target sub-image, may include the following steps:

[0211] Step F2: In response to the number of target sub-images being multiple, determining the presentation state of the target image with respect to the target object based on the recognition results corresponding to the respective target sub-images in accordance with a predetermined analysis method;

[0212] Among them, the predetermined analysis methods include:

[0213] If the different presentation states of the target object include a first presentation state indicating the presence of the object and a second presentation state indicating the absence of the object, then when, in each target sub-image, there is a target sub-image whose recognition result indicates the first presentation state, the presentation state of the target image with respect to the target object is determined to be the first presentation state; otherwise, the presentation state of the target image with respect to the target object is determined to be the second presentation state;

[0214] If the different presentation states of the target object include different presentation states regarding specified attributes, then determine the target recognition result with the highest frequency among the recognition results corresponding to each target sub-image, and determine the presentation state of the target image for the target object as: the presentation state represented by the target recognition result.

[0215] In this implementation, if there are multiple target sub-images captured, it indicates that the target image contains multiple areas with target objects. At this time, it is necessary to determine the corresponding recognition results of each captured target sub-image, and thus, based on the recognition results of each target sub-image, determine the presentation status of the target object to the target object.

[0216] Considering that different presentation states of the target object may include presentation states for whether the target object exists, for example, whether a tattoo exists or not; it may also include presentation states for specified attributes of the target object, for example, whether a black cat exists or a white cat exists.

[0217] Therefore, a predetermined analysis method is set in advance for different presentation states of the target object, so as to determine the presentation state of the target image for the target object based on the recognition results corresponding to each target sub-image. Specifically:

[0218] If the different presentation states of the target object include a first presentation state indicating the presence of the object and a second presentation state indicating the absence of the object, then if, in each target sub-image, there is a target sub-image whose recognition result indicates the first presentation state, the presentation state of the target image with respect to the target object may be determined as the first presentation state. Otherwise, the presentation state of the target image with respect to the target object may be determined as the second presentation state.

[0219] If the different presentation states of the target object include different presentation states regarding specified attributes, it is necessary to determine the target recognition result with the highest frequency among the corresponding recognition results in each target sub-image, so as to determine the presentation state of the target image for the target object as the presentation state represented by the target recognition result.

[0220] For example, the different presentation states of the target object include different presentation states regarding the color of the tattoo. The target recognition result with the highest frequency indicates that the color of the tattoo in the image is dark. In this case, the presentation state of the target image for the target object is determined as the color of the tattoo in the image is dark, and the target image is determined to be an image including a dark tattoo.

[0221] In this implementation, considering that when there are multiple target sub-images, the determination of the presentation state of the target object of the overall target image needs to be combined with the recognition results of each target sub-image; and considering that the presentation state of the target object includes multiple presentation methods, therefore, when there are multiple target sub-images, based on the different presentation states of the target object, the presentation state of the target object of the overall target image is determined using the corresponding analysis method, thereby improving the accuracy of automatic image recognition.

[0222] Optionally, in one implementation, the above step S603, determining the recognition result corresponding to the target sub-image based on the similarity of each prompt word, may include the following steps:

[0223] Step G1: for each prompt word, determine the confidence score corresponding to the prompt word based on the similarity corresponding to the prompt word;

[0224] The confidence score corresponding to the prompt word is used to characterize the presentation state of the target object corresponding to the target sub-image, which is the probability of the presentation state described in the prompt word;

[0225] Step G2: Based on the confidence score corresponding to each prompt word, determine the recognition result corresponding to the target sub-image.

[0226] In this implementation, after determining the similarity corresponding to each prompt word, the confidence score corresponding to the prompt word can be further determined, thereby determining the probability that the target sub-image's presentation state of the target object is the presentation state described by the prompt word. In other words, the higher the confidence score determined for the prompt word, the higher the probability that the target sub-image's presentation state of the target object is the presentation state described by the prompt word. Thus, based on the confidence score corresponding to each prompt word, the recognition result corresponding to the target sub-image is determined.

[0227] In this implementation, the confidence score is used to digitize the probability that the presentation state of the target object corresponding to the target sub-image is the presentation state described in each prompt word, so as to facilitate matching the target sub-image with the corresponding prompt word, thereby improving the accuracy of the recognition result corresponding to the target sub-image.

[0228] Optionally, in one implementation, step G2, determining the recognition result corresponding to the target sub-image based on the confidence score corresponding to each prompt word, may include the following steps:

[0229] Step G21: Select the highest confidence score from the confidence scores corresponding to each prompt word as the target confidence score;

[0230] Step G22: If the target confidence score is greater than a predetermined threshold, the recognition result corresponding to the target sub-image is determined as: the presentation state described by the prompt word corresponding to the target confidence score.

[0231] In this implementation, since the higher the confidence score corresponding to the determined prompt word, the higher the probability that the presentation state of the target object corresponding to the target sub-image is the presentation state described in the prompt word, after determining the confidence score corresponding to each prompt word, the highest confidence score is selected as the target confidence score.

[0232] However, although the determined confidence score is the highest, the probability that the target object's corresponding presentation state represented by this confidence score is the presentation state described in the prompt word is still low. For example, generally speaking, a confidence score of 50% or higher is required for the two to be considered a match. In this case, if the highest confidence score determined is 35%, although this confidence score is the highest confidence score among the target image's confidence scores for each prompt word, given that a confidence score of 50% is required for a match to be considered, the presentation state described by the prompt word corresponding to this confidence score does not actually match the presentation state of the target object.

[0233] Therefore, to improve the accuracy of the recognition result corresponding to the determined target sub-image, a predetermined threshold can be set for the confidence score. That is, when the determined confidence score is greater than the predetermined threshold, it is indicated that the presentation state described by the prompt word is the presentation state of the target object corresponding to the target sub-image. Therefore, by comparing the determined target confidence score with the predetermined threshold, it can be determined whether the presentation state described by the prompt word is the presentation state of the target object corresponding to the target sub-image.

[0234] If the target confidence score is greater than the above-mentioned predetermined threshold, the presentation state described by the prompt word corresponding to the target confidence score is the presentation state of the target object corresponding to the target sub-image, and the recognition result corresponding to the target sub-image can be determined as: the presentation state described by the prompt word corresponding to the target confidence score.

[0235] If the target confidence score is not greater than the predetermined threshold, the presentation state described by the prompt word corresponding to the target confidence score is not the presentation state of the target object corresponding to the target sub-image.

[0236] In this implementation, the accuracy of the recognition result corresponding to the determined target sub-image is further improved by pre-setting a predetermined threshold value for the confidence score.

[0237] Optionally, in one implementation, step G1, for each prompt word, determining the confidence score corresponding to the prompt word based on the similarity corresponding to the prompt word, may include the following steps:

[0238] Step G11: For each prompt word, normalize the similarity corresponding to the prompt word to obtain the confidence score corresponding to the prompt word.

[0239] In this implementation, after obtaining the similarity of each prompt word, the similarity corresponding to each prompt word may be normalized to obtain a confidence score corresponding to the prompt word.

[0240] Optionally, for each prompt word, a confidence score corresponding to the prompt word is calculated using a predetermined formula; wherein the predetermined formula includes:

[0241]

[0242] σ(z i ) represents the confidence score corresponding to the i-th prompt word about the target object, K represents the total number of prompt words in the prompt phrase group about the target object, z i Characterize the feature score corresponding to the i-th prompt word, z j Represents the feature score corresponding to the j-th prompt word.

[0243] The feature score corresponding to each prompt word is used to represent the similarity between the prompt word and the target sub-image, that is, the similarity corresponding to each prompt word in the embodiment of the present application.

[0244] In this implementation, by normalizing the similarity corresponding to each prompt word to a specific interval, for example, [0, 1] or [-1, 1], etc., the data distribution has statistical significance, and the probability distribution of similarity is more statistically intuitive, thereby improving the comparability of the similarity corresponding to each prompt word.

[0245] Optionally, in one implementation, the above step S601 of intercepting the region of interest from the target image to obtain the target sub-image may include the following steps:

[0246] Step H1: Using a region detection algorithm for the carrier region, perform region detection on the target image to obtain a region of interest;

[0247] Step H2: intercepting the region of interest in the target image to obtain a target sub-image.

[0248] In this implementation, after acquiring the target image, the region detection algorithm for the carrier region can be used to perform region detection on the target image to obtain the region of interest, and the region of interest in the target image can be intercepted to obtain the target sub-image.

[0249] Considering that the types of objects to be identified are different, the corresponding carrier regions are different, and accordingly, the region detection algorithms for the carrier regions are also different. For example, if the object to be identified is a tattoo, the corresponding carrier region is the human body region, and the corresponding region detection algorithm for the carrier region can be a human body detection algorithm; if the object to be identified is a trademark, the corresponding carrier region is the edge region of the target image, and the corresponding region detection algorithm for the carrier region can be a trademark detection algorithm. Therefore, optionally, based on the type of object to be identified, the corresponding region detection algorithm is selected, and region detection is performed on the target image to obtain the region of interest.

[0250] Among them, the above-mentioned human detection algorithm can be an anchor-base (based on anchor frame) human detection method, which is not specifically limited in the embodiments of the present application.

[0251] This implementation method can reduce the interference of other image elements in the image on image recognition by intercepting the region of interest in the target image, thereby improving the accuracy of automatic image recognition.

[0252] Next, in conjunction with a specific embodiment, the above step H1 is described in detail, using a region detection algorithm for the carrier region to perform region detection on the target image to obtain the region of interest. In this specific embodiment, the target object is a tattoo and the region of interest is a human body region.

[0253] For the target image to be identified, the size information of the target image is determined. If the size information is not equal to the first size adapted by the convolutional neural network, the target image is scaled to the first size to obtain a first image, and the convolutional neural network is used to extract features from the obtained first image, thereby obtaining a first feature image corresponding to the above-mentioned first image.

[0254] In this way, multiple first anchor frames about each pixel point in the obtained first feature image can be generated, so that the confidence level of the existence of a human body in each first anchor frame can be determined.

[0255] Among them, the so-called anchor frame can be understood as: when identifying the target existing in the feature image obtained after processing by the convolutional neural network, if it is determined that there is a target belonging to the detected human body in a certain area in the feature image, the area can be identified using a graphic frame. Then, the graphic frame used to identify the area where the human body target exists in the feature image is the anchor frame.

[0256] Because some areas of the feature image may not contain a human body, or the area framed by the first anchor frame may contain only a portion of a human body, a first preset score can be pre-set to ensure the accuracy of the determined image position information of the human body. This can greatly reduce the number of anchor frames that need to be processed later. Therefore, for each pixel in the first feature image, after determining the confidence level corresponding to each first anchor frame for that pixel, the anchor frames with a confidence level greater than the first preset score are retained as the first target anchor frames.

[0257] In this way, based on the position information between each first target anchor frame, the IOU (Intersection over Union) value between every two first target anchor frames in all the first target anchor frames can be calculated, where the so-called intersection over union ratio refers to the ratio of the area of ​​the overlapping area of ​​the two first target anchor frames divided by the area of ​​the union area of ​​the two first target anchor frames.

[0258] For example, the area of ​​the first target anchor box J1 is S J1 , the area of ​​the first target anchor box J2 is S J2 , the overlapping area of ​​the first target anchor frame J1 and the first target anchor frame J2 is S J1J2 Then the intersection-and-union ratio between the first target anchor frame J1 and the first target anchor frame J2 can be calculated as

[0259] Thus, in each group of first target anchor frames whose calculated IOU value is greater than the first preset IOU value, the position information corresponding to the first target anchor frame with the confidence score is selected and determined as the region of interest in the target image.

[0260] For example, for a target image, if the calculated IOU between any two first target anchor frames is greater than a first preset IOU value, then the human body framed by the two first target anchor frames can be determined to belong to the same person. In this way, the position information corresponding to the first target anchor frame with the highest confidence among the two first target anchor frames can be determined as the position information of the region of interest in the target image.

[0261] Optionally, since the position of the preset first anchor frame is offset relative to the actual position of the human body, a first relative offset can be pre-set, so that the first offset of each first anchor frame is determined simultaneously with the confidence score of each first anchor frame. In this way, each first target anchor frame whose first offset is not greater than the first relative offset can be retained, thereby achieving secondary filtering of the obtained first target anchor frames using the first relative offset. Then, through the two filtering steps, each first anchor frame whose confidence score is greater than the first preset score and whose first offset is not greater than the first relative offset is determined as a first target anchor frame, further improving the accuracy of the image position information of the determined region of interest in the target image.

[0262] Optionally, in one embodiment, as Figure 7 As shown, an image recognition method provided in an embodiment of the present application may further include the following steps:

[0263] S605: Determine whether the presentation state of the target image for the target object complies with a predetermined requirement; if not, execute step S606; otherwise, execute step S607;

[0264] S606: performing a predetermined processing operation on the target image;

[0265] The predetermined processing operation includes: using a predetermined image processing method to cover the target object in the target image, or terminating the upload of the target image;

[0266] S607: Upload the target image.

[0267] In this embodiment, to maintain the network environment of the network platform, it is necessary to pre-set predetermined rules for the presentation status of the target object. That is, images that meet the predetermined rules are considered normal images, while images that do not meet the predetermined rules are considered illegal images. For example, an image of a person without tattoos is considered normal, while an image of a person with tattoos is considered illegal.

[0268] Therefore, after determining the presentation status of the target image to the target object, it is possible to further determine whether the presentation status complies with the above-mentioned predetermined regulations, thereby determining whether the target image can be uploaded.

[0269] If the presentation state of the target image for the target object meets the predetermined requirements, the target image is characterized as a normal image, and the dissemination of the target image will not cause bad guidance, and the target image can be uploaded.

[0270] If the target image's presentation state for the target object does not conform to predetermined regulations, the target image is characterized as an illegal image, and the dissemination of the target image may cause bad guidance. Therefore, in order to maintain the network environment of the network platform, a predetermined processing operation can be performed on the target image.

[0271] Among them, in order to improve the image upload rate of the platform, the above-mentioned predetermined processing operation may include using a predetermined image processing method to cover the target object in the target image; in this way, when it is determined that the presentation state of the target image for the target object does not comply with the predetermined regulations, the target object in the target image can be covered using a predetermined image processing method, and thus, after covering the target object in the target image, an image recognition method provided by an embodiment of the present application is executed again on the covered target image until it is determined that the presentation state of the target image for the target object complies with the predetermined regulations, the target image is a normal image, and the target image is uploaded.

[0272] In addition, considering that the number of images to be identified is too large, the above-mentioned predetermined processing operation may include terminating the upload of the target image; in this way, when it is determined that the presentation status of the target image for the target object does not meet the predetermined regulations, the upload of the target image is directly terminated.

[0273] Optionally, after the upload of the target image is terminated, a reminder message indicating that the review has failed is sent to the corresponding client.

[0274] In this embodiment, target images uploaded to the network platform are classified according to predetermined regulations to ensure that the images uploaded to the network platform will not cause bad guidance and protect the network environment of the network platform.

[0275] Corresponding to the model training method provided in the above embodiment of the present application, the embodiment of the present application also provides a model training device, such as Figure 8 As shown, the device includes:

[0276] The image acquisition module 810 is used to acquire a sample image; wherein the sample image is provided with a label for characterizing the presentation state of the sample object;

[0277] Similarity acquisition module 820 is configured to input the sample image and each predetermined sample prompt word into a pre-trained contrastive learning model, so that the contrastive learning model generates a vector feature of the sample image, obtains a first modal feature of the sample image, determines a current second modal feature of each sample prompt word, and calculates the similarity between the first modal feature of the sample image and the second modal feature of each sample prompt word, thereby obtaining the similarity of the sample image with respect to each sample prompt word. The pre-trained contrastive learning model is trained based on a pre-trained model using images containing objects other than the sample object. Each sample prompt word is an image description text that describes the presentation state of the sample object. Furthermore, any sample prompt word belongs to one of two prompt word types, the presentation states described by the two prompt word types are mutually exclusive, and sample prompt words belonging to the same prompt word type have similar semantics. Each sample prompt word is obtained by combining a target keyword and other non-keywords, wherein the target keyword is a keyword used to characterize a presentation state of the sample object. The initial second modal feature of each sample prompt word is a vector feature of the sample prompt word.

[0278] a loss value calculation module 830 for determining a loss value corresponding to the contrastive learning model based on the similarity of the sample image to each sample prompt word and the label of the sample image;

[0279] The parameter adjustment module 840 is used to adjust the parameters of the specified feature part in the current second modal feature of each sample prompt word when it is determined based on the loss value that the contrastive learning model does not meet the model convergence conditions, and return to the step of obtaining the sample image; wherein the specified feature part in the current second modal feature of each sample prompt word is the feature part corresponding to other non-keywords in the current second modal feature of the sample prompt word.

[0280] As can be seen from the above, in the model training method provided in the embodiment of the present application, sample prompt words can be set in advance for the sample objects before model training. Each sample prompt word is an image description text with the presentation state of the sample object as the description content, and any sample prompt word belongs to one of the two prompt word types. The presentation states described by the two prompt word types are mutually exclusive, and the sample prompt words belonging to the same prompt word type have similar semantics. Each sample prompt word is obtained based on a target keyword and other non-keyword combinations, wherein the above-mentioned target keyword is a keyword used to characterize a presentation state of the sample object. Considering that other non-keywords in the prompt word have a significant impact on the second modal features of the entire prompt word, the contrastive learning model is relatively stable in generating vector features corresponding to the target keyword in the second modal features of the prompt word. However, for other non-keywords, the generated vector features may be different if the non-keywords are the same but combined differently, or if the non-keywords are different. In other words, the contrastive learning model is unstable in generating vector features corresponding to other non-keywords in the second modal features of the prompt word, and the generated vector features are quite different. Therefore, by determining the similarity of the sample images for each sample prompt word and the difference in the pre-set labels of the sample images themselves, the loss value of the current second modal feature of each sample prompt word generated by the contrastive learning model is determined. Therefore, when the contrastive learning model is judged to not meet the model convergence conditions based on the loss value, the parameters of the feature portion corresponding to other non-keywords in the current second modal features of each sample prompt word are adjusted to reduce the impact of other non-keywords in the prompt word on the accuracy of the recognition result, thereby ensuring the accuracy of image recognition using the contrastive learning model.

[0281] Furthermore, combining a target keyword with other non-keywords to create sample prompts reduces the tedious process of manually designing prompts and mitigates the potential for low accuracy associated with manually designed prompts. Furthermore, the sample prompts created through this combination method have a high accuracy rate. Therefore, during model training, a comparative learning model with high recognition performance can be trained using only a small number of sample prompts and sample images.

[0282] Optionally, in a specific implementation, the loss value calculation module 830 includes:

[0283] a confidence acquisition submodule, configured to determine the confidence of the sample image for each sample prompt word based on the similarity of the sample image to each sample prompt word; wherein the confidence of the sample image for any sample prompt word is used to represent the presentation state of the sample object in the sample image, and is the predicted probability of the presentation state described in the sample prompt word;

[0284] The loss value determination submodule is used to determine the loss value corresponding to the contrastive learning model based on the confidence of the sample image for each sample prompt word and the label of the sample image.

[0285] Optionally, in a specific implementation, the parameter adjustment module 840 is specifically configured to:

[0286] When it is determined based on the loss value that the contrastive learning model does not meet the model convergence condition, calculating, based on the loss value, the gradient value of the feature vector corresponding to the other non-keywords in the current second modal feature of each sample prompt word;

[0287] The gradient value is used to adjust the feature vector corresponding to the specified feature part in the current second modal feature of the sample prompt word.

[0288] Optionally, in a specific implementation, the confidence acquisition submodule is specifically configured to:

[0289] The similarity of the sample image with respect to each sample prompt word is normalized to obtain the confidence of the sample image with respect to each sample prompt word.

[0290] Optionally, in a specific implementation, the sample object is a tattoo.

[0291] Corresponding to the image recognition method provided in the above embodiment of the present application, the embodiment of the present application also provides an image recognition device, such as Figure 9 As shown, the device includes:

[0292] The image capture module 910 is configured to capture a region of interest from a target image to obtain a target sub-image; wherein the region of interest is a carrier region where the target object to be identified is located in the target image when the target object exists in the target image;

[0293] The similarity determination module 920 is used to input the target sub-image and the prompt phrase into a pre-trained contrastive learning model so that the contrastive learning model generates a vector feature of the target sub-image to obtain a first modal feature, and generates an initial modal feature for each prompt word in the prompt phrase. For each prompt word, the target sub-feature in the initial modal feature of the prompt word is concatenated with a designated sub-feature corresponding to the target sample prompt word to obtain a second modal feature of the prompt word; the similarity between the first modal feature of the target sub-image and the second modal feature of each prompt word is calculated to obtain the similarity of the target sub-image to each prompt word; and In the embodiment, each prompt word in the prompt word group is an image description text that uses the presentation state of the target object as a description content, and the presentation states described by different prompt words in the same prompt word group are mutually exclusive. The contrastive learning model is trained based on a model training method provided in an embodiment of the present application. The target sample prompt word is a sample prompt word in each sample prompt word whose other non-keywords match the other non-keywords of the prompt word. The designated sub-feature is the feature content of the designated feature part in the current second modal feature of the target sample prompt word when the contrastive learning model is trained. The target sub-feature is a feature that characterizes the target keyword.

[0294] A result determination module 930 is configured to determine a recognition result corresponding to the target sub-image based on the similarity of each prompt word; wherein the recognition result is a result regarding the presentation state of the target object;

[0295] The image determination module 940 is configured to determine a presentation state of the target image with respect to the target object based on a recognition result corresponding to the target sub-image.

[0296] As can be seen from the above, in the image recognition method provided by the embodiment of the present application, after a target sub-image is captured from a target image, the target sub-image and a prompt phrase associated with the target object are input into a pre-trained contrastive learning model. Accordingly, because the contrastive learning model reduces the influence of other non-keywords in the prompt phrase on the accuracy of the recognition result during the training process, when using the contrastive learning model to recognize the target object, for each prompt word in the prompt phrase, the target sub-feature of the prompt word is concatenated with the designated sub-feature corresponding to the target sample prompt word to obtain the second modal feature of the prompt word. That is, the feature representing the target keyword of the prompt word is concatenated with the feature content of the designated feature portion of the current second modal feature of the target sample prompt word when the contrastive learning model is trained, to obtain the second modal feature of the prompt word. Thus, the similarity between the first modal feature of the target sub-image and the second modal feature of each prompt word is calculated to obtain the similarity of the target sub-image with respect to each prompt word. Based on the obtained similarity, the presentation state of the target image with respect to the target object is determined. Through this solution, we can focus on the characteristic content of the keyword part in each prompt word in the prompt phrase group and use the specified characteristic part in the target sample prompt word to reduce the influence of non-keywords in the image recognition process, thereby achieving accurate recognition of the target object.

[0297] Optionally, in a specific implementation, the result determination module 930 includes:

[0298] a confidence determination submodule for determining, for each prompt word, a confidence score corresponding to the prompt word based on the similarity corresponding to the prompt word; wherein the confidence score corresponding to the prompt word is used to represent the probability that the presentation state of the target sub-image with respect to the target object is the presentation state described in the prompt word;

[0299] The result determination submodule is used to determine the recognition result corresponding to the target sub-image based on the confidence score corresponding to each prompt word.

[0300] Optionally, in a specific implementation, the result determination submodule is specifically configured to:

[0301] From the confidence scores corresponding to each prompt word, select the highest confidence score as the target confidence score;

[0302] If the target confidence score is greater than a predetermined threshold, the recognition result corresponding to the target sub-image is determined as: the presentation state described by the prompt word corresponding to the target confidence score.

[0303] Optionally, in a specific implementation, the image determination module 940 is specifically configured to:

[0304] In response to the number of the target sub-image being one, the presentation state represented by the recognition result corresponding to the target sub-image is used as the presentation state of the target image for the target object.

[0305] Optionally, in a specific implementation, the image determination module 940 is specifically configured to:

[0306] In response to the number of the target sub-images being multiple, determining, in accordance with a predetermined analysis method, a presentation state of the target image with respect to the target object based on recognition results corresponding to the respective target sub-images;

[0307] Wherein, the predetermined analysis method includes:

[0308] If the different presentation states of the target object include a first presentation state indicating the presence of the object and a second presentation state indicating the absence of the object, then when, in each of the target sub-images, there is a target sub-image whose recognition result indicates the first presentation state, the presentation state of the target image with respect to the target object is determined to be the first presentation state; otherwise, the presentation state of the target image with respect to the target object is determined to be the second presentation state;

[0309] If the different presentation states of the target object include different presentation states regarding specified attributes, then determine the target recognition result with the highest frequency among the recognition results corresponding to each of the target sub-images, and determine the presentation state of the target image for the target object as: the presentation state represented by the target recognition result.

[0310] Optionally, in a specific implementation, the target object is a tattoo, and the carrier area is a human body area.

[0311] Optionally, in a specific implementation, the device further includes:

[0312] An image filtering module is used to determine whether the presentation state of the target image for the target object complies with predetermined regulations; if not, a predetermined processing operation is performed on the target image; otherwise, the target image is uploaded; wherein the predetermined processing operation includes: using a predetermined image processing method to cover the target object in the target image, or terminating the upload of the target image.

[0313] The present application also provides an electronic device, such as Figure 10 As shown, it includes a processor 1001, a communication interface 1002, a memory 1003 and a communication bus 1004, wherein the processor 1001, the communication interface 1002, and the memory 1003 communicate with each other through the communication bus 1004.

[0314] Memory 1003, used for storing computer programs;

[0315] The processor 1001 is used to implement the model training method and / or image recognition method described in any of the above embodiments when executing the program stored in the memory 1003.

[0316] The communication bus mentioned in the terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used in the figure, but this does not mean that there is only one bus or only one type of bus.

[0317] The communication interface is used for communication between the above terminal and other devices.

[0318] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage. Alternatively, the memory may be at least one storage device located away from the processor.

[0319] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components.

[0320] In another embodiment provided in the present application, a computer-readable storage medium is also provided, in which a computer program is stored. When the computer program is executed by a processor, the model training method and / or the image recognition method described in any one of the above embodiments are implemented.

[0321] In another embodiment provided in the present application, a computer program product comprising instructions is also provided, which, when executed on a computer, enables the computer to execute the model training method and / or the image recognition method described in any one of the above embodiments.

[0322] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0323] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0324] Each embodiment in this specification is described in a related manner. Similar portions between the various embodiments can be referenced to each other. Each embodiment focuses on the differences between the other embodiments. In particular, since the apparatus embodiments, electronic device embodiments, computer-readable storage medium embodiments, and computer program product embodiments are generally similar to the method embodiments, their descriptions are relatively simple. For related portions, reference can be made to the descriptions of the method embodiments.

[0325] The above description is only a preferred embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the scope of protection of the present application.

Claims

1. A model training method, characterized in that: The method comprises: Acquire a sample image; wherein the sample image is provided with a label for characterizing a presentation state of the sample object; The sample image and each predetermined sample prompt word are input into a pre-trained contrastive learning model so that the contrastive learning model generates a vector feature of the sample image, obtains a first modal feature of the sample image, determines a current second modal feature of each sample prompt word, and calculates the similarity between the first modal feature of the sample image and the second modal feature of each sample prompt word, thereby obtaining a similarity of the sample image to each sample prompt word. The pre-trained contrastive learning model is trained on the basis of a pre-trained model using images containing objects other than the sample object. Each sample prompt word is an image description text that describes the presentation state of the sample object. Any sample prompt word belongs to one of two prompt word types, the presentation states described by the two prompt word types are mutually exclusive, and sample prompt words belonging to the same prompt word type have similar semantics. Each sample prompt word is obtained by combining a target keyword and other non-keywords, wherein the target keyword is a keyword used to characterize a presentation state of the sample object. The initial second modal feature of each sample prompt word is a vector feature of the sample prompt word. Determining a loss value corresponding to the contrastive learning model based on the similarity of the sample image to each sample prompt word and the label of the sample image; When it is determined based on the loss value that the contrastive learning model does not meet the model convergence conditions, parameters are adjusted for the specified feature part in the current second modal feature of each sample prompt word, and the process returns to the step of obtaining the sample image; wherein the specified feature part in the current second modal feature of each sample prompt word is the feature part corresponding to other non-keywords in the current second modal feature of the sample prompt word.

2. The method according to claim 1, characterized in that Determining the loss value corresponding to the contrastive learning model based on the similarity of the sample image to each sample prompt word and the label of the sample image includes: Determining the confidence level of the sample image for each sample prompt word based on the similarity of the sample image to each sample prompt word; wherein the confidence level of the sample image for any sample prompt word is used to characterize the presentation state of the sample object in the sample image, and is the predicted probability of the presentation state described in the sample prompt word; Based on the confidence of the sample image for each sample prompt word and the label of the sample image, a loss value corresponding to the contrastive learning model is determined.

3. The method according to claim 1, characterized in that When it is determined based on the loss value that the contrastive learning model does not meet the model convergence condition, adjusting parameters of a specified feature portion of the current second modal feature of each sample prompt word includes: When it is determined based on the loss value that the contrastive learning model does not meet the model convergence condition, calculating, based on the loss value, the gradient value of the feature vector corresponding to the other non-keywords in the current second modal feature of each sample prompt word; The gradient value is used to adjust the feature vector corresponding to the specified feature part in the current second modal feature of the sample prompt word.

4. The method according to claim 2, characterized in that The determining, based on the similarity of the sample image to each sample prompt word, the confidence of the sample image to each sample prompt word includes: The similarity of the sample image with respect to each sample prompt word is normalized to obtain the confidence of the sample image with respect to each sample prompt word.

5. The method according to claim 1, wherein The sample object is a tattoo.

6. An image recognition method, characterized in that: The method comprises: Cutting out a region of interest from the target image to obtain a target sub-image; wherein the region of interest is a carrier region where the target object to be identified is located in the target image when the target object exists in the target image; The target sub-image and the prompt phrase are input into a pre-trained contrastive learning model so that the contrastive learning model generates the vector features of the target sub-image to obtain the first modal features, and generates the initial modal features of each prompt word in the prompt phrase, and, for each prompt word, splices the target sub-feature in the initial modal features of the prompt word with the designated sub-feature corresponding to the target sample prompt word to obtain the second modal features of the prompt word; calculates the similarity between the first modal features of the target sub-image and the second modal features of each prompt word, and obtains the similarity of the target sub-image to each prompt word; wherein, the prompt Each prompt word in the phrase is an image description text that uses the presentation state of the target object as its description content, and the presentation states described by different prompt words in the same prompt word group are mutually exclusive. The contrastive learning model is trained based on the method described in any one of claims 1 to 5. The target sample prompt word is a sample prompt word among the sample prompt words whose other non-keywords match the other non-keywords of the prompt word. The designated sub-feature is the feature content of the designated feature part of the current second modal feature of the target sample prompt word when the contrastive learning model is trained. The target sub-feature is a feature that characterizes the target keyword. Determine a recognition result corresponding to the target sub-image based on the similarity of each prompt word; wherein the recognition result is a result regarding the presentation state of the target object; Based on the recognition result corresponding to the target sub-image, a presentation state of the target image with respect to the target object is determined.

7. The method according to claim 6, characterized in that The determining of the recognition result corresponding to the target sub-image based on the similarity corresponding to each prompt word includes: For each prompt word, a confidence score corresponding to the prompt word is determined based on the similarity corresponding to the prompt word; wherein the confidence score corresponding to the prompt word is used to represent the presentation state of the target object corresponding to the target sub-image, and is the probability of the presentation state described in the prompt word; Based on the confidence score corresponding to each prompt word, a recognition result corresponding to the target sub-image is determined.

8. The method according to claim 7, characterized in that The step of determining the recognition result corresponding to the target sub-image based on the confidence score corresponding to each prompt word includes: From the confidence scores corresponding to each prompt word, select the highest confidence score as the target confidence score; If the target confidence score is greater than a predetermined threshold, the recognition result corresponding to the target sub-image is determined as: the presentation state described by the prompt word corresponding to the target confidence score.

9. The method according to any one of claims 6 to 8, characterized in that: The determining, based on the recognition result corresponding to the target sub-image, a presentation state of the target image with respect to the target object, includes: In response to the number of the target sub-image being one, the presentation state represented by the recognition result corresponding to the target sub-image is used as the presentation state of the target image for the target object.

10. The method according to any one of claims 6 to 8, characterized in that: The determining, based on the recognition result corresponding to the target sub-image, a presentation state of the target image with respect to the target object, includes: In response to the number of the target sub-images being multiple, determining, in accordance with a predetermined analysis method, a presentation state of the target image with respect to the target object based on recognition results corresponding to the respective target sub-images; Wherein, the predetermined analysis method includes: If the different presentation states of the target object include a first presentation state indicating the presence of the object and a second presentation state indicating the absence of the object, then when, in each of the target sub-images, there is a target sub-image whose recognition result indicates the first presentation state, the presentation state of the target image with respect to the target object is determined to be the first presentation state; otherwise, the presentation state of the target image with respect to the target object is determined to be the second presentation state; If the different presentation states of the target object include different presentation states regarding specified attributes, then determine the target recognition result with the highest frequency among the recognition results corresponding to each of the target sub-images, and determine the presentation state of the target image for the target object as: the presentation state represented by the target recognition result.

11. The method according to claim 6, characterized in that The target object is a tattoo, and the carrier area is a human body area.

12. The method according to claim 6, characterized in that The method further comprises: determining whether the presentation state of the target image for the target object complies with a predetermined requirement; if not, performing a predetermined processing operation on the target image; otherwise, uploading the target image; The predetermined processing operation includes: using a predetermined image processing method to cover the target object in the target image, or terminating the uploading of the target image.

13. A model training device, characterized in that: The device comprises: An image acquisition module, configured to acquire a sample image; wherein the sample image is provided with a label for characterizing a presentation state of the sample object; a similarity acquisition module, configured to input the sample image and each predetermined sample prompt word into a pre-trained contrastive learning model, so that the contrastive learning model generates a vector feature of the sample image, obtains a first modal feature of the sample image, determines a current second modal feature of each sample prompt word, and calculates the similarity between the first modal feature of the sample image and the second modal feature of each sample prompt word, thereby obtaining the similarity of the sample image with respect to each sample prompt word; wherein the pre-trained contrastive learning model is trained on the basis of a pre-trained model using images containing objects other than the sample object; each sample prompt word is an image description text that describes the presentation state of the sample object, and any sample prompt word belongs to one of two prompt word types, the presentation states described by the two prompt word types are mutually exclusive, and sample prompt words belonging to the same prompt word type have similar semantics; each sample prompt word is obtained by combining a target keyword and other non-keywords, wherein the target keyword is a keyword used to characterize a presentation state of the sample object; and the initial second modal feature of each sample prompt word is a vector feature of the sample prompt word; a loss value calculation module, configured to determine a loss value corresponding to the contrastive learning model based on the similarity of the sample image to each sample prompt word and the label of the sample image; A parameter adjustment module is used to adjust the parameters of the specified feature part in the current second modal feature of each sample prompt word when it is determined based on the loss value that the contrastive learning model does not meet the model convergence conditions, and return to the step of obtaining the sample image; wherein the specified feature part in the current second modal feature of each sample prompt word is the feature part corresponding to other non-keywords in the current second modal feature of the sample prompt word.

14. An image recognition device, characterized in that: The device comprises: An image capture module is configured to capture a region of interest from a target image to obtain a target sub-image; wherein the region of interest is a carrier region where the target object to be identified is located in the target image when the target object exists in the target image; A similarity determination module is configured to input the target sub-image and the prompt phrase into a pre-trained contrastive learning model, so that the contrastive learning model generates a vector feature of the target sub-image to obtain a first modal feature, and generates an initial modal feature for each prompt word in the prompt phrase, and, for each prompt word, concatenates the target sub-feature in the initial modal feature of the prompt word with a designated sub-feature corresponding to the target sample prompt word to obtain a second modal feature of the prompt word; calculates the similarity between the first modal feature of the target sub-image and the second modal feature of each prompt word, respectively, to obtain the similarity of the target sub-image for each prompt word; wherein Each prompt word in the prompt word group is an image description text that uses the presentation state of the target object as a description content, and the presentation states described by different prompt words in the same prompt word group are mutually exclusive. The contrastive learning model is trained based on the method described in any one of claims 1 to 5. The target sample prompt word is a sample prompt word among each sample prompt word whose other non-keywords match the other non-keywords of the prompt word. The designated sub-feature is the feature content of the designated feature part of the current second modal feature of the target sample prompt word when the contrastive learning model is trained. The target sub-feature is a feature that characterizes the target keyword. A result determination module is configured to determine a recognition result corresponding to the target sub-image based on the similarity of each prompt word; wherein the recognition result is a result regarding the presentation state of the target object; The image determination module is used to determine the presentation state of the target image with respect to the target object based on the recognition result corresponding to the target sub-image.

15. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; Memory for storing computer programs; The processor is configured to implement the method steps described in any one of claims 1 to 5 when executing a program stored in a memory, and / or implement the method steps described in any one of claims 6 to 12.

16. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps of any one of claims 1 to 5 are implemented, and / or the method steps of any one of claims 6 to 12 are implemented.

Images