Model training method and device, image recognition method and device and electronic equipment

Abstract

The embodiment of the invention provides a model training method and device, an image recognition method and device and electronic equipment, and relates to the technical field of machine learning. The model training method comprises the following steps: acquiring a sample image; inputting the sample image and each sample cue word into a pre-trained comparative learning model to enable the comparative learning model to generate a first modal feature of the sample image; determining a current second modal feature of each sample cue word; calculating the similarity of the sample image for each sample cue word; determining a loss value based on each similarity and a label of the sample image; and when it is judged that the model does not meet the model convergence condition based on the loss value, parameter adjustment is performed on a specified feature part in the current second modal feature of each sample prompt word based on the loss value, and the step of obtaining the sample image is returned. According to the scheme, the influence of non-keywords on the accuracy of the recognition result can be reduced, so that the accuracy of image recognition by the contrast learning model is ensured.

Description

Technical Field

[0001] The present application relates to the field of machine learning technology, and in particular to a model training method, image recognition method, device and electronic equipment. Background Art

[0002] In order to maintain the network environment of the network platform, it is necessary to review the images or videos uploaded to the platform to avoid the dissemination of images or videos that may cause bad guidance through the platform, for example, the dissemination of images including tattoos, smoke and other objects.

[0003] Typically, a pre-trained contrastive learning model, such as a pre-trained CLIP (Contrastive Language-Image Pre-Training) model, is used to recognize an image, that is, to recognize the presentation state of an object in an image, such as whether an object exists in the image. In the recognition process, the contrastive learning model can recognize an image based on a pre-set prompt word.

[0004] However, the inventors found in the process of using the above contrastive learning model for image recognition that:

[0005] For the same object, when the keywords in the prompt words used are the same but other non-keywords are different, the recognition results obtained by the contrastive learning model for the same image are different. For example, "a photo of a person with a tattoo" and "a photo of a person with tattoo" are prompt words for "image of a person with tattoo" with the same keyword "tattoo" but different other non-keywords.

[0006] Based on this, how to optimize the pre-trained contrastive learning model to reduce the impact of non-keywords in the prompt words on the accuracy of the recognition results, thereby ensuring the accuracy of image recognition using the contrastive learning model, has become a technical problem that needs to be solved urgently. Summary of the invention

[0007] The purpose of the embodiments of the present application is to provide a model training method, an image recognition method, a device and an electronic device to reduce the influence of non-keywords in the prompt words on the accuracy of the recognition results, thereby ensuring the accuracy of the contrastive learning model for image recognition. The specific technical solution is as follows:

[0008] In a first aspect provided by an embodiment of the present application, first, a model training method is provided, the method comprising:

[0009] Acquire a sample image; wherein the sample image is provided with a label for characterizing a presentation state of the sample object;

[0010] The sample image and each predetermined sample prompt word are input into a pre-trained contrastive learning model, so that the contrastive learning model generates a vector feature of the sample image, obtains the first modal feature of the sample image, determines the current second modal feature of each sample prompt word, and calculates the similarity between the first modal feature of the sample image and the second modal feature of each sample prompt word, so as to obtain the similarity of the sample image to each sample prompt word; wherein the pre-trained contrastive learning model is obtained by training with images containing objects other than the sample object on the basis of the pre-trained model; each sample prompt word is an image description text with the presentation state of the sample object as the description content, and any sample prompt word belongs to one of two prompt word types, the presentation states described by the two prompt word types are mutually exclusive, and the semantics of sample prompt words belonging to the same prompt word type are close, and each sample prompt word is obtained by combining a target keyword and other non-keywords, and the target keyword is a keyword used to characterize a presentation state of the sample object; the initial second modal feature of each sample prompt word is the vector feature of the sample prompt word;

[0011] Determining a loss value corresponding to the contrastive learning model based on the similarity of the sample image to each sample prompt word and the label of the sample image;

[0012] When it is determined based on the loss value that the contrastive learning model does not meet the model convergence conditions, parameters are adjusted for the designated feature parts in the current second modal features of each sample prompt word, and the step of obtaining the sample image is returned; wherein the designated feature parts in the current second modal features of each sample prompt word are feature parts corresponding to other non-keywords in the current second modal features of the sample prompt word.

[0013] In a second aspect provided by an embodiment of the present application, an image recognition method is also provided, the method comprising:

[0014] From the target image, a region of interest is intercepted to obtain a target sub-image; wherein the region of interest is a carrier region where the target object to be identified is located in the target image when the target object to be identified exists in the target image;

[0015] The target sub-image and the prompt phrase are input into a pre-trained contrastive learning model, so that the contrastive learning model generates the vector features of the target sub-image to obtain the first modal features, and generates the initial modal features of each prompt word in the prompt phrase group, and, for each prompt word, concatenates the target sub-features in the initial modal features of the prompt word with the designated sub-features corresponding to the target sample prompt word to obtain the second modal features of the prompt word; calculates the similarity between the first modal features of the target sub-image and the second modal features of each prompt word, respectively, to obtain the similarity of the target sub-image to each prompt word; wherein, the prompt Each prompt word in the phrase is an image description text with the presentation state of the target object as the description content, and the presentation states described by different prompt words in the same prompt word group are mutually exclusive, and the contrastive learning model is obtained by training based on a model training method provided in the first aspect above; the target sample prompt word is a sample prompt word among the sample prompt words, whose other non-keywords match the other non-keywords of the prompt word, the designated sub-feature is the feature content of the designated feature part in the current second modal feature of the target sample prompt word when the contrastive learning model is trained, and the target sub-feature is a feature that characterizes the target keyword;

[0016] Based on the similarity of each prompt word, determining the recognition result corresponding to the target sub-image; wherein the recognition result is a result about the presentation state of the target object;

[0017] Based on the recognition result corresponding to the target sub-image, a presentation state of the target image for the target object is determined.

[0018] In a third aspect provided by an embodiment of the present application, a model training device is provided, the device comprising:

[0019] An image acquisition module, used to acquire a sample image; wherein the sample image is provided with a label for characterizing a presentation state of the sample object;

[0020] A similarity acquisition module is used to input the sample image and each predetermined sample prompt word into a pre-trained contrastive learning model, so that the contrastive learning model generates a vector feature of the sample image, obtains the first modal feature of the sample image, determines the current second modal feature of each sample prompt word, and calculates the similarity between the first modal feature of the sample image and the second modal feature of each sample prompt word, so as to obtain the similarity of the sample image for each sample prompt word; wherein the pre-trained contrastive learning model is obtained by training with images containing objects other than the sample object on the basis of the pre-trained model; each sample prompt word is an image description text with the presentation state of the sample object as the description content, and any sample prompt word belongs to one of two prompt word types, the presentation states described by the two prompt word types are mutually exclusive, and the semantics of sample prompt words belonging to the same prompt word type are similar, and each sample prompt word is obtained by combining a target keyword and other non-keywords, and the target keyword is a keyword used to characterize a presentation state of the sample object; the initial second modal feature of each sample prompt word is the vector feature of the sample prompt word;

[0021] A loss value calculation module, used to determine the loss value corresponding to the contrastive learning model based on the similarity of the sample image to each sample prompt word and the label of the sample image;

[0022] A parameter adjustment module is used to adjust the parameters of the specified feature part in the current second modal feature of each sample prompt word when it is judged that the contrastive learning model does not meet the model convergence condition based on the loss value, and return to the step of obtaining the sample image; wherein the specified feature part in the current second modal feature of each sample prompt word is the feature part corresponding to other non-keywords in the current second modal feature of the sample prompt word.

[0023] In a fourth aspect provided by an embodiment of the present application, an image recognition device is provided, the device comprising:

[0024] An image capture module is used to capture an area of ​​interest from a target image to obtain a target sub-image; wherein the area of ​​interest is a carrier area where the target object to be identified is located in the target image when the target object to be identified exists in the target image;

[0025] A similarity determination module is used to input the target sub-image and the prompt word group into a pre-trained contrastive learning model so that the contrastive learning model generates the vector feature of the target sub-image to obtain the first modal feature, and generates the initial modal feature of each prompt word in the prompt word group, and, for each prompt word, splices the target sub-feature in the initial modal feature of the prompt word with the designated sub-feature corresponding to the target sample prompt word to obtain the second modal feature of the prompt word; calculates the similarity between the first modal feature of the target sub-image and the second modal feature of each prompt word respectively, to obtain the similarity of the target sub-image for each prompt word; wherein , each prompt word in the prompt word group is an image description text with the presentation state of the target object as the description content, and the presentation states described by different prompt words in the same prompt word group are mutually exclusive, and the contrastive learning model is trained based on a model training method provided in the first aspect above; the target sample prompt word is a sample prompt word in each sample prompt word whose other non-keywords match the other non-keywords of the prompt word, the designated sub-feature is the feature content of the designated feature part in the current second modal feature of the target sample prompt word when the contrastive learning model is trained, and the target sub-feature is a feature that characterizes the target keyword;

[0026] A result determination module, used to determine the recognition result corresponding to the target sub-image based on the similarity of each prompt word; wherein the recognition result is a result about the presentation state of the target object;

[0027] The image determination module is used to determine the presentation state of the target image for the target object based on the recognition result corresponding to the target sub-image.

[0028] In the fifth aspect provided by the embodiment of the present application, there is also provided an electronic device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; the memory is used to store computer programs; the processor is used to implement any model training method provided in the first aspect and / or any image recognition method provided in the second aspect when executing the program stored in the memory.

[0029] In another aspect provided by the embodiments of the present application, a computer-readable storage medium is also provided, in which a computer program is stored. When the computer program is executed by a processor, it implements any model training method provided in the first aspect above, and / or any image recognition method provided in the second aspect above.

[0030] In another aspect provided by the embodiments of the present application, a computer program product comprising instructions is also provided, which, when executed on a computer, enables the computer to execute any model training method provided in the first aspect above, and / or any image recognition method provided in the second aspect above.

[0031] In the model training method provided in the embodiment of the present application, before model training, sample prompt words can be set in advance for sample objects, each sample prompt word is an image description text with the presentation state of the sample object as the description content, and any sample prompt word belongs to one of two prompt word types, the presentation states described by the two prompt word types are mutually exclusive, and the semantics of sample prompt words belonging to the same prompt word type are similar, and each sample prompt word is obtained based on a target keyword and other non-keyword combinations, wherein the above-mentioned target keyword is a keyword used to characterize a presentation state of the sample object. Considering that other non-keywords in the prompt word have a greater impact on the second modal features of the entire prompt word, that is, the contrastive learning model is relatively stable in generating vector features corresponding to the target keywords in the second modal features of the prompt word, while for other non-keywords, the other non-keywords are the same but the combination methods are different, or the other non-keywords are different, and the generated vector features may be different. In other words, the contrastive learning model is unstable in generating vector features corresponding to other non-keywords in the second modal features of the prompt word, and the generated vector features are quite different. Therefore, by determining the similarity of the sample image for each sample prompt word and the difference in the pre-set labels of the sample image itself, the loss value of the current second modal feature of each sample prompt word generated by the contrastive learning model is determined. Therefore, when it is judged based on the loss value that the contrastive learning model does not meet the model convergence condition, the parameters of the feature parts corresponding to other non-keywords in the current second modal features of each sample prompt word are adjusted to reduce the influence of other non-keywords in the prompt word on the accuracy of the recognition result, thereby ensuring the recognition accuracy of the image using the contrastive learning model.

[0032] In addition, the combination of a target keyword and other non-keywords to obtain sample prompt words can reduce the tedious process of manually designing prompt words, and at the same time, reduce the possibility of low accuracy of prompt words when manually designing prompt words. Moreover, the accuracy of the sample prompt words obtained by the above combination method is high. Therefore, when training the model, a small number of sample prompt words and sample images can be used to train a comparative learning model with high recognition effect.

[0033] Furthermore, in the image recognition method provided by the embodiment of the present application, after the target sub-image is captured from the target image, the target sub-image and the prompt phrase group related to the target object are input into a pre-trained contrastive learning model; accordingly, since the contrastive learning model reduces the influence of other non-keywords in the prompt words on the accuracy of the recognition result during the training process, when the contrastive learning model is used to recognize the target object, for each prompt word in the prompt phrase group, the target sub-feature of the prompt word is spliced ​​with the designated sub-feature corresponding to the target sample prompt word to obtain the second modal feature of the prompt word, that is, the feature representing the target keyword of the prompt word is spliced ​​with the feature content of the designated feature part in the current second modal feature of the target sample prompt word when the contrastive learning model is trained to obtain the second modal feature of the prompt word; thereby, the similarity between the first modal feature of the target sub-image and the second modal feature of each prompt word is calculated to obtain the similarity of the target sub-image to each prompt word, and based on the obtained similarity, the presentation state of the target image to the target object is determined. Through this solution, we can focus on the feature content of the keyword part in each prompt word in the prompt word group and use the specified feature part in the target sample prompt word to reduce the influence of non-keywords in the image recognition process, thereby achieving accurate recognition of the target object. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0035] Figure 1 A schematic diagram of the classification process of the CLIP model provided in an embodiment of the present application;

[0036] Figure 2 This is a diagram showing the effect of different non-keywords on the final accuracy provided in the embodiment of the present application;

[0037] Figure 3 A flowchart of a model training method provided in an embodiment of the present application;

[0038] Figure 4 A flowchart of an existing training method for a contrastive learning model is provided;

[0039] Figure 5 A schematic diagram of a process for obtaining multimodal features of text provided in an embodiment of the present application;

[0040] Figure 6 A schematic diagram of the process of the first image recognition method provided in the embodiment of the present application;

[0041] Figure 7A schematic diagram of a second image recognition method provided in an embodiment of the present application;

[0042] Figure 8 A schematic diagram of the structure of a model training device provided in an embodiment of the present application;

[0043] Fig. 9 A schematic diagram of the structure of an image recognition device provided in an embodiment of the present application;

[0044] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0046] In order to better understand the embodiments of the present application, the professional terms involved in the embodiments of the present application are first introduced below.

[0047] The text-image pre-training model based on contrastive learning (also referred to as the contrastive learning model) is a model pre-trained on 400 million pairs of image-text matching pairs, which has a strong generalization ability. By giving the contrastive learning model the corresponding text prompt words, the model can obtain the similarity of the corresponding image. Specifically, the contrastive learning model recognizes the image, that is, recognizes the presentation state of the image for the object, such as whether there is an object in the image. Among them, during the recognition process, the contrastive learning model can classify and recognize the image based on pre-set prompt words. The above-mentioned contrastive learning model can be a CLIP model, a BLIP (Bootstrapping Language-Image Pre-training, bootstrap language image pre-training) model or a SigLip (Sigmoid Loss for Language-Image Pre-training, language-image pre-training) model, which is not specifically limited in the embodiments of the present application.

[0048] Taking the above contrastive learning model as a CLIP model as an example, when the CLIP model is used to classify an image, the CLIP model can be given two prompt words, "people with tattoos" and "people without tattoos", and the CLIP model can be allowed to determine which prompt word has a closer similarity to classify the image.

[0049] Combine the following Figure 1 This paper introduces the classification process of the CLIP model. Figure 1As shown in FIG. 1 , plane, car, dog, and bird are all target objects (or sample objects) in the embodiment of the present application, and “a photo of a {object}” is the prompt word in the embodiment of the present application. The byte position of object is the byte position occupied by the target object in the prompt word.

[0050] After receiving the above prompt words, the Text Encoder of the CLIP model can convert the text multimodal features corresponding to the above prompt words, including T 1 , T 2 , T 3 ……T N .

[0051] After receiving the image, the Image Encoder of the CLIP model can convert the image into the multimodal features I corresponding to the above image. 1 .

[0052] The CLIP model can then transform the obtained image multimodal features I 1 The text multimodal features T corresponding to the prompt words 1 , T 2 , T 3 ……T N Perform feature similarity matching to obtain the similarity I corresponding to each prompt word 1 ·T 1 ,I 1 ·T 2 ,I 1 ·T 3 ,I 1 ·T N Among them, I 1 ·T 3 The corresponding similarity is the highest, and I 1 ·T 3 The object in the corresponding prompt word is dog. Therefore, it can be determined that the object in the image input to the Image Encoder is dog, and the output result of the model is "a photo of a dog".

[0053] Among them, the above-mentioned image multimodal feature is the first modal feature in the embodiment of the present application; the above-mentioned text multimodal feature (text multimodal feature) is the second modal feature in the embodiment of the present application.

[0054] However, when giving the CLIP model prompt words, using different prompt words will affect the final classification effect. For example, when the CLIP model uses two different prompt words "a photo of a person with a tattoo" or "aphoto of a person with tattoo" to describe "an image of a person with a tattoo", the final accuracy of the contrastive learning model will be different. In other words, for the same object, when the keywords in the prompt words used are the same but other non-keywords are different, the contrastive learning model will get different recognition results for the same image.

[0055] For example, Figure 2 The following is a graph showing the effect of different non-keywords on the final accuracy. Among them, CLASS (classification) is used to represent the label that needs to be classified, which can also be understood as a keyword; [V] i Used to represent non-keywords; Accuracy is used to represent the accuracy on the test set when using the corresponding prompt word; Prompt is used to represent the prompt word; Photo is the image to be recognized.

[0056] Depend on Figure 2 It can be seen that for the same image to be identified, when other non-keywords are different, or when other non-keywords are the same but the combination of non-keywords is different, the accuracy rates obtained by comparing the learning models are different.

[0057] Based on this, how to optimize the pre-trained contrastive learning model to reduce the impact of non-keywords in the prompt words on the accuracy of the recognition results, thereby ensuring the accuracy of image recognition using the contrastive learning model, has become a technical problem that needs to be solved urgently.

[0058] In order to solve the above technical problems, an embodiment of the present application provides a model training method.

[0059] Among them, a model training method provided in an embodiment of the present application can be applied to an electronic device, and the electronic device is specifically used for training the model. In a specific application, the electronic device can be a smart phone, a tablet computer, etc., which is all reasonable.

[0060] A model training method provided in an embodiment of the present application may include the following steps:

[0061] Acquire a sample image; wherein the sample image is provided with a label for characterizing a presentation state of the sample object;

[0062] The sample image and each predetermined sample prompt word are input into a pre-trained contrastive learning model, so that the contrastive learning model generates a vector feature of the sample image, obtains a first modal feature of the sample image, determines a current second modal feature of each sample prompt word, and calculates the similarity between the first modal feature of the sample image and the second modal feature of each sample prompt word, so as to obtain the similarity of the sample image to each sample prompt word; wherein the pre-trained contrastive learning model is obtained by training with images containing objects other than the sample object on the basis of the pre-trained model; each sample prompt word is an image description text with a presentation state of the sample object as a description content, and any sample prompt word belongs to one of two prompt word types, the presentation states described by the two prompt word types are mutually exclusive, and the semantics of sample prompt words belonging to the same prompt word type are similar, and each sample prompt word is obtained by combining a target keyword and other non-keywords, and the target keyword is a keyword used to characterize a presentation state of the sample object; the initial second modal feature of each sample prompt word is the vector feature of the sample prompt word;

[0063] Determining a loss value corresponding to the contrastive learning model based on the similarity of the sample image to each sample prompt word and the label of the sample image;

[0064] When it is determined based on the loss value that the contrastive learning model does not meet the model convergence conditions, parameters are adjusted for the designated feature parts in the current second modal features of each sample prompt word, and the step of obtaining the sample image is returned; wherein the designated feature parts in the current second modal features of each sample prompt word are feature parts corresponding to other non-keywords in the current second modal features of the sample prompt word.

[0065] As can be seen from the above, in the model training method provided in the embodiment of the present application, before model training, sample prompt words can be set in advance for sample objects, and each sample prompt word is an image description text with the presentation state of the sample object as the description content, and any sample prompt word belongs to one of two prompt word types, the presentation states described by the two prompt word types are mutually exclusive, and the sample prompt words belonging to the same prompt word type have similar semantics, and each sample prompt word is obtained based on a target keyword and other non-keyword combinations, wherein the above-mentioned target keyword is a keyword used to characterize a presentation state of the sample object. Considering that other non-keywords in the prompt word have a greater impact on the second modal features of the entire prompt word, that is, the contrastive learning model is relatively stable in generating vector features corresponding to the target keywords in the second modal features of the prompt word, while for other non-keywords, the other non-keywords are the same but the combination methods are different, or the other non-keywords are different, and the generated vector features may be different. In other words, the contrastive learning model is unstable in generating vector features corresponding to other non-keywords in the second modal features of the prompt word, and the generated vector features are quite different. Therefore, by determining the similarity of the sample image for each sample prompt word and the difference in the pre-set labels of the sample image itself, the loss value of the current second modal feature of each sample prompt word generated by the contrastive learning model is determined. Therefore, when it is judged based on the loss value that the contrastive learning model does not meet the model convergence condition, the parameters of the feature parts corresponding to other non-keywords in the current second modal features of each sample prompt word are adjusted to reduce the influence of other non-keywords in the prompt word on the accuracy of the recognition result, thereby ensuring the recognition accuracy of the image using the contrastive learning model.

[0066] In addition, the combination of a target keyword and other non-keywords to obtain sample prompt words can reduce the tedious process of manually designing prompt words, and at the same time, reduce the possibility of low accuracy of prompt words when manually designing prompt words. Moreover, the accuracy of the sample prompt words obtained by the above combination method is high. Therefore, when training the model, a small number of sample prompt words and sample images can be used to train a comparative learning model with high recognition effect.

[0067] The following is a detailed description of a model training method provided in an embodiment of the present application in conjunction with the accompanying drawings.

[0068] Figure 3 A flow chart of a model training method provided in an embodiment of the present application is shown in FIG. Figure 3 As shown, the method comprises the following steps:

[0069] S301: Acquire a sample image.

[0070] The sample image is provided with a label for characterizing the presentation status of the sample object.

[0071] In the present application, an image provided with a representation for representing a presentation state of a sample object may be acquired as a sample image.

[0072] It is understandable that the number of sample images acquired may be one or more, and this is not specifically limited in the embodiments of the present application.

[0073] The label used to characterize the presentation state of the sample object may be a digital label or a text label, and the present application makes specific limitations on this. For example, if the sample object exists in the sample image, the label of the sample image may be set to 1; if the sample object does not exist in the sample image, the label of the sample image may be set to 0.

[0074] Optionally, in one implementation, the sample object is a tattoo.

[0075] In this implementation, if the sample object is a tattoo, the sample image obtained is an image of the tattoo. Accordingly, the presentation state of the sample object may include: the presence or absence of a tattoo, the presence of a dark tattoo or the presence of a light tattoo, etc.

[0076] S302: Input the sample image and each predetermined sample prompt word into the pre-trained contrastive learning model, so that the contrastive learning model generates the vector features of the sample image, obtains the first modal features of the sample image, determines the current second modal features of each sample prompt word, and calculates the similarity between the first modal features of the sample image and the second modal features of each sample prompt word, respectively, to obtain the similarity of the sample image for each sample prompt word.

[0077] Among them, the pre-trained contrastive learning model is obtained by training on the basis of the pre-trained model using images containing objects other than sample objects; each sample prompt word is an image description text that uses the presentation state of the sample object as the description content, and any sample prompt word belongs to one of two prompt words, the presentation states described by the two prompt word types are mutually exclusive, and the semantics of sample prompt words belonging to the same prompt word type are similar. Each sample prompt word is obtained based on a target keyword and other non-keyword combinations, and the target keyword is a keyword used to characterize a presentation state of the sample object; the initial second modal feature of each sample prompt word is the vector feature of the sample prompt word.

[0078] In the present application, the pre-trained model can be trained in advance using images containing other objects so that a characteristic expression of the image can be formed in the pre-trained model, and a matching mapping relationship is established between the characteristic expression of the image and the prompt word with similar semantic features to obtain a pre-trained contrastive learning model. In other words, the above training of the pre-trained model enables the pre-trained contrastive learning model to generate vector features about the image, and by comparing the vector features about the image with the similarity of the vector features about the prompt word, determine the prompt word that matches the image.

[0079] Therefore, after obtaining the sample image and each predetermined sample prompt word, the obtained sample image and each predetermined sample prompt word can be input into the pre-trained contrastive learning model so that the contrastive learning model can generate the vector features of the sample image and obtain the first modal features of the sample image. At the same time, the current second modal features of each of the above sample prompt words are determined. It should be noted that when the sample prompt word is input into the contrastive learning model for the first time, the generated vector features of the sample prompt word are directly determined as the initial second modal features of the sample prompt word, so that the initial second modal features of the sample prompt word are used as the current second modal features of the sample prompt word. Then, the similarity between the first modal features of the sample image and the second modal features of each sample prompt word is calculated to obtain the similarity of the sample image for each sample prompt word.

[0080] Among them, each sample prompt word is an image description text that uses the presentation state of the sample object as the description content, and any sample prompt word belongs to one of two prompt word types. The presentation states described by the two prompt word types are mutually exclusive, and the sample prompt words belonging to the same prompt word type have similar semantics.

[0081] For example, the sample prompt words include: having a tattoo and not having a tattoo, having a cyan tattoo and not having a cyan tattoo, having a patterned tattoo and not having a patterned tattoo. Among them, having a tattoo, having a cyan tattoo and having a patterned tattoo all belong to the presentation state described by the sample prompt words as having a tattoo, that is, the above three sample prompt words belong to the same prompt word type W, and the semantics of each sample prompt word in the prompt word type W are similar; not having a tattoo, not having a cyan tattoo and not having a patterned tattoo all belong to the presentation state described by the sample prompt words as not having a tattoo, that is, the above three sample prompt words belong to the same prompt word type M, and the semantics of each sample prompt word in the prompt word type M are similar. In summary, the presentation states described by the above prompt word type W and the above prompt word type M are mutually exclusive.

[0082] In this way, when using each of the above-mentioned predetermined sample prompt words to identify the sample image, it is possible to determine which prompt word's presentation state the sample object's presentation state in the sample image is more consistent with based on the prompt words that describe mutually exclusive presentation states. Thus, the presentation state described by the prompt word that is more consistent is used as the presentation state of the sample image for the sample object to achieve recognition of the sample image.

[0083] Optionally, a target keyword and other non-keywords that can be combined with the target keyword can be set in advance for a presentation state of the sample object. In this way, through the combination of a target keyword and other non-keywords, an image description text with the presentation state of the sample object as the description content can be obtained as a sample prompt word, thereby determining to obtain multiple sample prompt words.

[0084] The combination of the above target keywords and other non-keywords may include:

[0085] For different presentation states of the same sample object, corresponding target keywords are set, and for each target keyword, other non-keywords are randomly combined with the target keyword to obtain multiple sample prompt words. For example, when the sample object is a tattoo, the presentation state of the sample object may include: the presence of a tattoo and the absence of a tattoo. At this time, the target keywords that can be set are the presence of a tattoo and the absence of a tattoo.

[0086] For a presentation state of the same sample object, a corresponding target keyword is set, and multiple sample prompt words are obtained by randomly combining the target keyword with other non-keywords. For example, when the sample object is a tattoo, the presentation state of the sample object may include: the presence of a tattoo and the absence of a tattoo. At this time, the target keyword that can be set is the presence of a tattoo or the absence of a tattoo.

[0087] The embodiment of the present application does not limit the number of target keywords in the multiple sample prompt words and the number of other non-keywords.

[0088] Compared with the method of manually designing prompt words in the prior art, the combination method of obtaining sample prompt words by combining the above target keyword and other non-keywords can simplify the cumbersome process of manually designing prompt words, and reduce the possibility of low accuracy of the prompt words involved in manually designing prompt words. In addition, the accuracy of the sample prompt words obtained by the above combination method provided by the embodiment of the present application is high. Therefore, during model training, a comparative learning model with high recognition efficiency can be trained using a small number of sample prompt words and sample images.

[0089] Considering that different data categories, such as text data, image data, etc., have corresponding vector features in different dimensions, in order to facilitate the subsequent determination of the similarity of each prompt word to the sample image, the vector features corresponding to data of different data categories can be unified into the same predetermined dimension. Therefore, after obtaining the sample image, the vector features of the predetermined dimension of the sample image can be generated to obtain the first modal feature.

[0090] Optionally, after acquiring the sample image, the contrastive learning model calls an image encoder to extract image features of the sample image, and converts the extracted image features into vector features of a predetermined dimension to obtain a first modal feature of the sample image.

[0091] At the same time, the text encoder is called to extract the text features of each sample prompt word, thereby converting the text features of each sample prompt word into vector features of a predetermined dimension to obtain the initial second modal features of each sample prompt word. The vector features of the text features and image features are represented by the feature matrix, and this embodiment of the application does not specifically limit this.

[0092] Afterwards, after obtaining the first modal feature and the second modal feature, the similarity between the first modal feature and the second modal feature corresponding to each sample prompt word can be calculated to obtain the similarity of the sample image for each sample prompt word, wherein the feature score can be used to represent the similarity corresponding to each sample prompt word, and the embodiments of the present application do not make specific limitations on this.

[0093] It should be noted that the embodiments of the present application involve a text-image pre-training model based on contrastive learning, and the main ability of this type of model is to align image features and text features in the same feature space (dimension). Therefore, in theory, only the trained model layer in this type of model can be used to convert image or text features to vector features, that is, the image features are converted to the first modality features through the image encoder in this type of model, and the text features are converted to the second modality features through the text encoder in this type of model. Other untrained methods cannot be used, otherwise, the obtained first modality features and second modality features will not be in the same feature space from a mathematical perspective.

[0094] Based on this, optionally, the sample image is input into the image encoder of the pre-trained contrastive learning model so that the image encoder extracts the image features of the sample image and converts the extracted image features to obtain the first modal features of the sample image; the sample prompt word is input into the text encoder of the pre-trained contrastive learning model so that the text encoder extracts the text features of the sample prompt word and converts the extracted text features to obtain the initial second modal features of the sample prompt word.

[0095] In this way, since the generation method of the first modal feature and the second modal feature is pre-trained, the obtained first modal feature and the second modal feature are in the same feature space from a mathematical perspective, that is, the modal features obtained by the image encoder and text encoder using the above-mentioned pre-trained contrastive learning model are in the same predetermined dimension.

[0096] Optionally, cosine similarity is used to calculate the distance between the first modal feature and the second modal feature corresponding to each sample prompt word. If the distance between the first modal feature and the second modal feature corresponding to the sample prompt word is closer, the similarity between the two modal features is higher. Conversely, if the distance between the first modal feature and the second modal feature corresponding to the sample prompt word is farther, the similarity between the two modal features is lower.

[0097] S303: Determine a loss value corresponding to the contrastive learning model based on the similarity of the sample image to each sample prompt word and the label of the sample image.

[0098] In this application, since the sample image is provided with a label for representing the presentation state of the sample object, that is, the presentation state of the sample object represented by the label of the sample image is the actual state presented by the sample object in the sample image. Therefore, after obtaining the similarity of the sample image for each sample prompt word, the loss value corresponding to the contrastive learning model can be determined through the similarity and label.

[0099] S304: when it is determined based on the loss value that the contrastive learning model does not meet the model convergence condition, adjusting the parameters of the designated feature part in the current second modal feature of each sample prompt word, and returning to the above step S301;

[0100] The specified feature part in the current second modal feature of each sample prompt word is the feature part corresponding to other non-keywords in the current second modal feature of the sample prompt word.

[0101] In the present application, since the target keyword is fixed for different sample prompt words, what changes are other non-keywords, that is, the generation of vector features corresponding to the target keyword is relatively stable, while for other non-keywords, due to the different combinations of other non-keywords, the generated vector features are different and have poor stability. Therefore, the loss value of the comparative learning model during training or inference is generated in the process of generating the feature part corresponding to other non-keywords in the second modal features of the prompt word.

[0102] After determining the loss value corresponding to the contrastive learning model, it can be further determined whether the contrastive learning model meets the model convergence condition, that is, whether the determined loss value reaches a predetermined loss value.

[0103] When it is determined based on the loss value that the contrastive learning model does not meet the model convergence conditions, parameters are adjusted for the designated feature parts in the current second modal features of each sample prompt word, that is, the feature parts corresponding to other non-keywords in the current second modal features of the sample prompt word are adjusted, and the process returns to step S301, and based on the next sample image, a model training method provided in an embodiment of the present application is used to adjust parameters for the designated feature parts in the current second modal features of each sample prompt word, so as to reduce the influence of other non-keywords in the prompt word on the accuracy of the recognition result, thereby optimizing the training of the contrastive learning model.

[0104] Optionally, when it is determined based on the loss value that the contrastive learning model does not meet the model convergence condition, the parameter of the designated feature part in the current second modal feature of each sample prompt word is adjusted according to a predetermined amplitude. That is, when it is determined that the contrastive learning model does not meet the model convergence condition, the user can adjust the parameter according to a predetermined parameter adjustment amplitude, for example, according to an amplitude represented by a value such as 0.5, 1, 2, etc.

[0105] Optionally, when it is determined based on the loss value that the contrastive learning model does not meet the model convergence conditions, the parameters of the specified feature part in the current second modal feature of each sample prompt word are adjusted in a predetermined direction. That is to say, when it is determined that the contrastive learning model does not meet the model convergence conditions, the user can adjust the parameters in a predetermined parameter adjustment direction, for example, in a direction of increasing the parameter value, or in a direction of reducing the parameter value. Among them, the specific adjusted parameter value can be any positive number, and the embodiments of the present application do not make specific limitations on this.

[0106] Optionally, when it is determined based on the loss value that the contrastive learning model does not meet the model convergence condition, the parameters of the designated feature part in the current second modal feature of each sample prompt word are adjusted according to a predetermined direction and a predetermined amplitude. That is, when it is determined that the contrastive learning model does not meet the convergence condition, the user can adjust the parameters according to a predetermined parameter adjustment direction and with a predetermined parameter adjustment amplitude. For example, based on the designated feature part in the current second modal feature of each sample prompt word, the parameters are adjusted with an amplitude represented by 0.5 in the direction of increasing the parameter value.

[0107] When it is determined based on the loss value that the contrastive learning model meets the model convergence condition, the training of the model is terminated.

[0108] Optionally, in order to improve the training efficiency of the model training, the model training end condition can be set to the model training times reaching a predetermined number of times. That is, when the model training times reach the predetermined number of times, the model training ends.

[0109] As can be seen from the above, in the model training method provided in the embodiment of the present application, before model training, sample prompt words can be set in advance for sample objects, and each sample prompt word is an image description text with the presentation state of the sample object as the description content, and any sample prompt word belongs to one of two prompt word types, the presentation states described by the two prompt word types are mutually exclusive, and the sample prompt words belonging to the same prompt word type have similar semantics, and each sample prompt word is obtained based on a target keyword and other non-keyword combinations, wherein the above-mentioned target keyword is a keyword used to characterize a presentation state of the sample object. Considering that other non-keywords in the prompt word have a greater impact on the second modal features of the entire prompt word, that is, the contrastive learning model is relatively stable in generating vector features corresponding to the target keywords in the second modal features of the prompt word, while for other non-keywords, the other non-keywords are the same but the combination methods are different, or the other non-keywords are different, and the generated vector features may be different. In other words, the contrastive learning model is unstable in generating vector features corresponding to other non-keywords in the second modal features of the prompt word, and the generated vector features are quite different. Therefore, by determining the similarity of the sample image for each sample prompt word and the difference in the pre-set labels of the sample image itself, the loss value of the current second modal feature of each sample prompt word generated by the contrastive learning model is determined. Therefore, when it is judged based on the loss value that the contrastive learning model does not meet the model convergence condition, the parameters of the feature parts corresponding to other non-keywords in the current second modal features of each sample prompt word are adjusted to reduce the influence of other non-keywords in the prompt word on the accuracy of the recognition result, thereby ensuring the recognition accuracy of the image using the contrastive learning model.

[0110] In addition, the combination of a target keyword and other non-keywords to obtain sample prompt words can reduce the tedious process of manually designing prompt words, and at the same time, reduce the possibility of low accuracy of prompt words when manually designing prompt words. Moreover, the accuracy of the sample prompt words obtained by the above combination method is high. Therefore, when training the model, a small number of sample prompt words and sample images can be used to train a comparative learning model with high recognition effect.

[0111] Optionally, in one implementation, the above step S303, based on the similarity of the sample image to each sample prompt word and the label of the sample image, determines the loss value corresponding to the contrastive learning model, which may include the following steps:

[0112] Step A1: determining the confidence of the sample image for each sample prompt word based on the similarity of the sample image for each sample prompt word;

[0113] The confidence of the sample image for any sample prompt word is used to characterize the presentation state of the sample object in the sample image, which is the predicted probability of the presentation state described in the sample prompt word;

[0114] Step A2: Based on the confidence of the sample image for each sample prompt word and the label of the sample image, determine the loss value corresponding to the contrastive learning model.

[0115] In this implementation, for each sample prompt word, after determining the similarity of the sample image to the sample prompt word, the confidence of the sample prompt word can be further determined to determine the predicted probability that the presentation state of the sample object in the sample image is the presentation state described in the sample prompt word. In other words, the higher the confidence score corresponding to the sample prompt word, the higher the predicted probability that the presentation state of the sample object in the sample image is the presentation state described in the sample prompt word.

[0116] In this way, based on the confidence of the sample image for each sample prompt word and the label of the sample image, the loss value corresponding to the contrastive learning model can be determined. The smaller the determined loss value is, the closer the presentation state of the sample object in the sample image is to the presentation state described in the sample prompt word.

[0117] Among them, the label set for the sample image to characterize the presentation state of the sample object can be understood as that the label of the sample image represents the actual state presented by the sample object in the sample image. Therefore, the corresponding confidence level can be set directly based on the actual state presented by the sample object. For example, if the sample object is a tattoo, one presentation state represented by the label is the presence of a tattoo, and the corresponding confidence level of the presentation state represented by the label is set to 1.

[0118] Optionally, in one implementation, the confidence of the sample image for each sample prompt word and the confidence corresponding to the label of the sample image are input into a predetermined loss function to calculate the loss value corresponding to the contrastive learning model.

[0119] Among them, the above-mentioned predetermined loss function may include a cross entropy loss function, etc., to which the embodiment of the present application does not make any specific limitation.

[0120] In this implementation, the confidence score is used to digitize the probability that the presentation state of the sample object corresponding to the sample image is the presentation state described in each sample prompt word, so as to match the sample image with the corresponding sample prompt word.

[0121] Optionally, in one implementation, the above step A1, determining the confidence of the sample image for each sample prompt word based on the similarity of the sample image for each sample prompt word, may include the following steps:

[0122] Step A11: normalize the similarity of the sample image for each sample prompt word to obtain the confidence of the sample image for each sample prompt word.

[0123] In this implementation, after the similarity of each sample prompt word is obtained, the similarity corresponding to each sample prompt word may be normalized to obtain the confidence corresponding to the sample prompt word.

[0124] Optionally, for each sample prompt word, a confidence level corresponding to the sample prompt word is calculated using a predetermined formula; wherein the predetermined formula includes:

[0125]

[0126] σ(z i ) represents the confidence corresponding to the i-th sample prompt word of the sample object, K represents the total number of sample prompt words of the sample object, z i Characterize the feature score corresponding to the i-th sample prompt word, z j Characterize the feature score corresponding to the j-th sample prompt word.

[0127] The feature score corresponding to each sample prompt word is used to characterize the similarity between the sample prompt word and the sample image, that is, the similarity corresponding to each prompt word in the embodiment of the present application.

[0128] This implementation method, by normalizing the similarity corresponding to each sample prompt word to a specific interval, for example, [0, 1] or [-1, 1], etc., makes the data distribution statistically significant and the probability distribution of similarity more statistically intuitive, thereby improving the comparability of the similarity corresponding to each sample prompt word.

[0129] Optionally, in one implementation, the above step S304, when it is determined based on the loss value that the contrastive learning model does not meet the model convergence condition, performs parameter adjustment on a specified feature part of the current second modal feature of each sample prompt word, may include the following steps:

[0130] Step B: When it is determined based on the loss value that the contrastive learning model does not meet the model convergence conditions, the gradient value of the feature vector corresponding to other non-keywords in the current second modal feature of each sample prompt word is calculated based on the loss value; and the gradient value is used to adjust the feature vector corresponding to the specified feature part in the current second modal feature of the sample prompt word.

[0131] In this implementation, it is considered that for different sample prompt words, the target keywords are fixed, and what changes are other non-keywords, that is, the loss value of the contrastive learning model during the training and inference process is caused by other non-keywords that change. Therefore, when it is judged based on the loss value that the contrastive learning model does not meet the model convergence conditions, the parameters of the specified feature part in the current second modal feature of each sample prompt word can be adjusted based on the loss value. Specifically: First, based on the loss value, the gradient value of the vector features corresponding to other non-keywords in the current second modal feature of each sample prompt word is calculated, and then the calculated gradient value is used to adjust the feature vector corresponding to the specified feature part in the current second modal feature of the sample prompt word.

[0132] In this way, the determined loss value can provide a specific adjustment range and adjustment direction for subsequent parameter adjustments. When adjusting the parameters, there is no need to adjust according to a predetermined direction and / or predetermined range, so as to improve the flexibility of parameter adjustment, thereby reducing the number and frequency of parameter adjustments, and further improving the efficiency of model training.

[0133] Optionally, in one implementation, the current second modal feature of each sample prompt word carries a feature tag, wherein the feature tag includes: other non-keyword tags and target keyword tags. In this way, by identifying whether the carried feature tag is other non-keyword tags, it is determined whether the feature part is the designated feature part corresponding to other non-keywords in the current second modal feature.

[0134] Optionally, in one implementation, the representation form of the current second modal feature of each sample prompt word converted by calling the text encoder is a feature matrix group, which includes a first matrix of feature vectors corresponding to the target keywords of the sample prompt word and a second matrix of feature vectors corresponding to other non-keywords of the sample prompt word. Since the matrix positions of the first matrix and the second matrix are different, the feature vectors corresponding to other non-keywords in the generated current second modal feature can be determined by determining the position of the matrix, that is, the feature vector corresponding to the specified feature part of the current second modal feature of the sample prompt word that needs parameter adjustment.

[0135] In this implementation, the feature vector corresponding to the specified feature part in the current second modal feature of the sample prompt word is adjusted to reduce the impact of non-keywords in the prompt word on the accuracy of the recognition result, so as to ensure the recognition accuracy of the image using the contrastive learning model.

[0136] Optionally, a back propagation algorithm is used to calculate the gradient value of the feature vector corresponding to the non-keyword in each sample prompt word based on the loss value.

[0137] Below, in conjunction with a specific embodiment, a model training method provided in an embodiment of the present application is specifically introduced.

[0138] Conventional deep learning models need to label large amounts of data, and in actual business, the data that needs to be identified has a long-tail distribution, that is, the data that needs to be identified accounts for a small proportion of the total data and is difficult to collect, resulting in high data labeling costs and slow model iteration cycles. Therefore, considering that contrastive learning models such as CLIP do not need to collect data to complete image recognition, CLIP and other contrastive learning models can be used to perform zero-sample predictions and achieve automatic image recognition.

[0139] In the related art, when training the above-mentioned contrastive learning model, it is necessary to train the text encoder and image encoder contained in the contrastive learning model at the same time. During this period, the number of training samples and the training time required are greatly increased, resulting in a decrease in the training efficiency of the model.

[0140] For example, Figure 4 The figure shows a flowchart of an existing contrastive learning model training method. In the existing contrastive learning model training process, it is necessary to obtain key prompt words such as airplane, butterfly and pizza respectively, combine the key prompt words with other non-keywords, and then output each prompt word obtained after the above combination to the Text Encoder of the CLIP model to obtain the Text features corresponding to each prompt word; then, input the image into the Image Encoder of the CLIP model to obtain the Image features corresponding to the image; then, calculate the Similarity scores of the text features and image features as the similarity between the two. Figure 4 It can be seen that when training the existing contrastive learning model, the text encoder and the image encoder need to be trained at the same time, which makes the training efficiency of the model low.

[0141] Based on this, an embodiment of the present application provides a model training method to achieve optimized training of the entire model by optimizing the feature vectors of other non-keywords in the prompt words. In this way, there is no need to optimize the text encoder and the image encoder, so that the model can obtain a higher recognition accuracy, thereby improving the training efficiency of the model.

[0142] The following is a specific example of a model training method provided by the present application. The sample object in this specific example is a tattoo, the pre-trained contrastive learning model is a CLIP model, and the specific process includes the following steps:

[0143] Step 1: Set the initial keywords, combine the keywords with the learnable vectors, and convert them into text multimodal features through a text encoder; wherein the above-mentioned learnable vectors are the non-keywords in the embodiment of the present application, and the above-mentioned initial keywords are the target keywords in the embodiment of the present application.

[0144] Step 2: Input the image and its corresponding label, and convert the image into image multimodal features through an image encoder.

[0145] Step three: Use the back propagation algorithm to tune the parameters corresponding to the learnable vectors in the prompt word; wherein the parameters corresponding to the learnable variables in the prompt word are the specified feature parts in the current second modal features of the sample prompt word in the embodiment of the present application.

[0146] In this embodiment, feature engineering is used to tune the parameters corresponding to the learnable vectors in the prompt words. That is to say, when the keywords "tattoo" and "not tattoo" are given in the prompt words, other non-keywords are filled with learnable vector parameters. In this way, the filling vector parameters can be tuned and learned by using a small number of samples to obtain the best model effect.

[0147] The following is a detailed introduction to the above steps 1 to 3:

[0148] Wherein, obtaining text multimodal features in step 1 may include the following steps:

[0149] Step D1: setting a plurality of keywords for tattoo classification, for example, when the image has a tattoo, setting the prompt word to "tattoo"; when the image does not have a tattoo, setting the prompt word to "not tattoo".

[0150] Step D2: Add multiple learnable vectors before and after the prompt word, and combine them with the keyword in step D1 to form a complete prompt word;

[0151] That is, through the combination of a keyword and multiple other non-keywords, multiple complete prompt words are obtained as multiple prompt words.

[0152] Step D3: The combined multiple prompt words are respectively passed through the text encoder of the CLIP model to obtain text features, and then the text features are passed through the text multimodal network to convert the text features into text multimodal features. Here, the text multimodal features corresponding to each text are feature matrices of size [1, 512].

[0153] For example, Figure 5 It is a flowchart of obtaining multimodal features of text. Figure 5 The learnable context in is the learnable vector mentioned above, and airplane, butterfly, and pizza can all be sample objects. After combining the learnable vector with any sample object, a complete prompt word is obtained, and the obtained complete prompt word is input into the text encoder of the CLIP model to obtain the text multimodal features corresponding to the input prompt word.

[0154] The process of converting the image into the multimodal features of the image in step 2 may include the following steps:

[0155] Step E1: Randomly select multiple images from the dataset to form a batch.

[0156] The above data set is a plurality of sample images in the embodiment of the present application, each image in the data set is about tattoos, and each image is provided with a label about tattoos. Therefore, a plurality of images can be randomly selected from the data set to form a batch as a plurality of images for model training, so that the model can be trained using the plurality of images and the plurality of combined prompt words.

[0157] Step E2: scaling the image to a fixed size; the fixed size may be 224×224 or 336×336, etc., which is not specifically limited in the embodiment of the present application.

[0158] Step E3: Use the image encoder of the CLIP model to extract the image features, and then pass the image features through the image multimodal network to convert the image features into image multimodal features. Here, the image multimodal features corresponding to each image are feature matrices of size [1, 512].

[0159] In this way, multiple text multimodal features and image multimodal features are dot-multiplied and added in sequence, and divided by the vector length of each text multimodal feature and image multimodal feature to calculate the similarity between the text multimodal features and the image multimodal features, and obtain the feature scores corresponding to the features of the image and each text feature; wherein the above feature scores are used to characterize the similarity between the image and each text feature.

[0160] Then, define the feature score as z and perform softmax (normalization) operation on all feature scores:

[0161]

[0162] σ(z i ) represents the confidence of the i-th prompt word about tattoos, K represents the total number of sample prompt words about tattoos, and z i Characterize the feature score corresponding to the i-th prompt word, z j Represents the feature score corresponding to the j-th prompt word.

[0163] Through the above formula, the feature score of the image for each text can be mapped to a confidence score between 0 and 1 as the predicted confidence score.

[0164] Among them, for the above step three, using the back propagation algorithm to tune the parameters corresponding to the learnable vector in the prompt word may include the following steps:

[0165] The cross-entropy loss function is used to calculate the loss value between the confidence score predicted by the model and the label of the image itself. The back-propagation algorithm is used to calculate the gradient value of the learnable vector in the prompt word based on the loss value, and the parameter value of the learnable vector is updated.

[0166] That is to say, the predicted confidence score and the confidence score corresponding to the label of the image itself are input into the cross-entropy loss function calculation model, and the loss value between the predicted confidence score and the label of the image itself is calculated. Then, the back propagation algorithm is used to calculate the gradient value of the learnable vector in the prompt word based on the above loss value, and the obtained gradient value is used to tune the text features of the learnable vector.

[0167] After tuning the parameters of the text features of the learnable vector, repeat the above steps 2 and 3 until all images in the data set are extracted, and perform gradient descent calculations to optimize the contrastive learning model.

[0168] The above steps can be regarded as one round of model training. Repeat multiple rounds of training until the preset number of training rounds or the loss value reaches a preset value.

[0169] Compared with the use of traditional models and traditional training methods, this embodiment only requires a small number of samples to obtain a higher model accuracy rate, which greatly reduces the difficulty of data collection in scenarios with long-tail distribution data characteristics such as audit business. Moreover, compared with the use of manually designed prompt words, the prompt words obtained by feature engineering can obtain a higher accuracy rate, and it also greatly reduces the cumbersome process of manually designing prompt words.

[0170] The following is a detailed introduction to an image recognition method provided in an embodiment of the present application.

[0171] Among them, an image recognition method provided in an embodiment of the present application can be applied to an electronic device, and the electronic device is specifically used for image recognition. In a specific application, the electronic device can be a smart phone, a tablet computer, etc., which is all reasonable.

[0172] An image recognition method provided in an embodiment of the present application may include the following steps:

[0173] From the target image, a region of interest is intercepted to obtain a target sub-image; wherein the region of interest is a carrier region where the target object to be identified is located in the target image when the target object to be identified exists in the target image;

[0174] The target sub-image and the prompt phrase are input into a pre-trained contrastive learning model, so that the contrastive learning model generates the vector features of the target sub-image to obtain the first modal features, and generates the initial modal features of each prompt word in the prompt phrase group, and, for each prompt word, concatenates the target sub-features in the initial modal features of the prompt word with the designated sub-features corresponding to the target sample prompt word to obtain the second modal features of the prompt word; calculates the similarity between the first modal features of the target sub-image and the second modal features of each prompt word, respectively, to obtain the similarity of the target sub-image to each prompt word; wherein, the prompt Each prompt word in the phrase is an image description text with the presentation state of the target object as the description content, and the presentation states described by different prompt words in the same prompt word group are mutually exclusive, and the contrastive learning model is obtained by training based on a model training method provided in an embodiment of the present application; the target sample prompt word is a sample prompt word among the sample prompt words, whose other non-keywords match the other non-keywords of the prompt word, the designated sub-feature is the feature content of the designated feature part in the current second modal feature of the target sample prompt word when the contrastive learning model is trained, and the target sub-feature is a feature that characterizes the target keyword;

[0175] Based on the similarity of each prompt word, determining the recognition result corresponding to the target sub-image; wherein the recognition result is a result about the presentation state of the target object;

[0176] Based on the recognition result corresponding to the target sub-image, a presentation state of the target image with respect to the target object is determined.

[0177] As can be seen from the above, in the image recognition method provided by the embodiment of the present application, after the target sub-image is captured from the target image, the target sub-image and the prompt phrase group related to the target object are input into the pre-trained contrastive learning model; accordingly, since the contrastive learning model reduces the influence of other non-keywords in the prompt words on the accuracy of the recognition result during the training process, when the contrastive learning model is used to identify the target object, for each prompt word in the prompt phrase group, the target sub-feature of the prompt word is spliced ​​with the designated sub-feature corresponding to the target sample prompt word to obtain the second modal feature of the prompt word, that is, the feature representing the target keyword of the prompt word is spliced ​​with the feature content of the designated feature part in the current second modal feature of the target sample prompt word when the contrastive learning model is trained to obtain the second modal feature of the prompt word; thereby, the similarity between the first modal feature of the target sub-image and the second modal feature of each prompt word is calculated, the similarity of the target sub-image to each prompt word is obtained, and based on the obtained similarity, the presentation state of the target image to the target object is determined. Through this solution, we can focus on the feature content of the keyword part in each prompt word in the prompt word group and use the specified feature part in the target sample prompt word to reduce the influence of non-keywords in the image recognition process, thereby achieving accurate recognition of the target object.

[0178] An image recognition method provided by an embodiment of the present application is described below in conjunction with the accompanying drawings.

[0179] Figure 6 A flowchart of an image recognition method provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, the method may include the following steps:

[0180] S601: Cut out a region of interest from a target image to obtain a target sub-image.

[0181] The region of interest is the carrier region where the target object to be identified is located in the target image when the target object to be identified exists in the target image.

[0182] In the present application, it is taken into consideration that when the target object to be identified exists in the target image, the carrier area where the target object is located in the target image will not change much. For example, the carrier area corresponding to the trademark is the edge area of ​​the overall image, the carrier area corresponding to the watch is the arm area, and the carrier area corresponding to the car is the ground area.

[0183] Therefore, when performing image recognition, the carrier region can be determined as the region of interest from the target image, and the region of interest can be intercepted to obtain a target sub-image. In other words, the image region included in the target sub-image is the region of interest.

[0184] In addition, for a target image, there may be one or more regions of interest contained therein. When there is one region of interest, the number of intercepted target sub-images is one; correspondingly, when there are multiple regions of interest, the number of intercepted target sub-images is multiple. The same method can be used to process each target sub-image to obtain the recognition result corresponding to each target sub-image.

[0185] Optionally, in one embodiment, the target object is a tattoo, and the carrier area is a human body area.

[0186] In this embodiment, when the target object is a tattoo, the carrier region where the tattoo exists is a human body region, and therefore, the captured target sub-image is an image containing a human body.

[0187] It should be noted that in the technical solution of this application, the operations of obtaining, storing, using, processing, transmitting, providing and disclosing the user's personal information are all carried out with the authorization of the user. Moreover, the image of the human body in this embodiment is not a human body image of a specific user and cannot reflect the personal information of a specific user. In addition, the two-dimensional human body image in this embodiment comes from a public data set.

[0188] S602: Input the target sub-image and the prompt word group into a pre-trained contrastive learning model, so that the contrastive learning model generates a vector feature of the target sub-image to obtain a first modal feature, and generates an initial modal feature of each prompt word in the prompt word group, and, for each prompt word, concatenates the target sub-feature in the initial modal feature of the prompt word with the designated sub-feature corresponding to the target sample prompt word to obtain a second modal feature of the prompt word; calculates the similarity between the first modal feature of the target sub-image and the second modal feature of each prompt word, respectively, to obtain the similarity of the target sub-image for each prompt word;

[0189] Among them, each prompt word in the prompt word group is an image description text with the presentation state of the target object as the description content, and the presentation states described by different prompt words in the same prompt word group are mutually exclusive. The contrastive learning model is trained based on a model training method provided in an embodiment of the present application. The target sample prompt word is a sample prompt word among each sample prompt word, whose other non-keywords match the other non-keywords of the prompt word. The designated sub-feature is the feature content of the designated feature part in the current second modal feature of the target sample prompt word when the contrastive learning model is trained. The target sub-feature is the feature that characterizes the target keyword.

[0190] In the present application, a prompt phrase group describing the target object is obtained in advance, and each prompt word in the prompt phrase group is an image description text with the presentation state of the target object as the description content, and the presentation states described by different prompt words in the same prompt phrase group are mutually exclusive. For example, when the target object is a tattoo, the presentation state of the target object can include the presence of a tattoo and the absence of a tattoo, and the corresponding prompt words can be "the image has a person with a tattoo" and "the image does not have a person with a tattoo"; for another example, when the target object is a tattoo, the presentation state of the target object can include the presence of a dark tattoo and the absence of a dark tattoo, and the corresponding prompt words can be "the image has a person with a dark tattoo" and "the image does not have a person with a dark tattoo"; for another example, when the target object is a cat, the presentation state of the target object can include the presence of a cat and the absence of a cat, and the corresponding prompt words can be "the image has a cat" and "the image does not have a cat", etc.

[0191] In this way, by using different prompt words in the prompt word group to describe the mutually exclusive presentation states of the target object, it is possible to determine the presentation state of the target object in the target sub-image, and determine which prompt word in the prompt words better fits the presentation state described. Thus, the presentation state described by the more appropriate prompt word is used as the presentation state of the target sub-image for the target object, and the recognition result corresponding to the target sub-image is obtained. Furthermore, based on the presentation state of the target sub-image for the target object, the presentation state of the overall target image for the target object can be further determined to achieve automatic image recognition.

[0192] Therefore, the acquired target sub-image and prompt phrase can be input into a contrastive learning model pre-trained using a model training method provided in an embodiment of the present application, so that the contrastive learning model generates vector features of the target sub-image, obtains the first modal features, and generates initial modal features for each prompt word in the prompt phrase.

[0193] Since the trained contrastive learning model stores various designated sub-features, that is, stores the designated feature content of the current second modal features of each sample prompt word when the contrastive learning model is trained, for each prompt word, a sample prompt word having other non-keywords matching the other non-keywords of the prompt word can be selected from each sample prompt word as the target sample prompt word, thereby splicing the target sub-feature in the initial modal feature of the prompt word with the designated sub-feature corresponding to the selected target sample prompt word to obtain the second modal feature of the prompt word. In this way, the trained designated sub-feature is used to replace the vector features corresponding to other non-keywords in the initial modal feature to reduce the influence of the non-keywords in the prompt word on the accuracy of the recognition result.

[0194] Afterwards, the similarity between the first modality feature of the target sub-image and the second modality feature of each prompt word is calculated to obtain the similarity of the target sub-image to each prompt word in the prompt word group.

[0195] In this way, by focusing on the characteristic content of the keyword part in each prompt word in the prompt word group and using the specified characteristic part in the target sample prompt word, the influence of non-keywords in the image recognition process is reduced, thereby achieving accurate recognition of the target object.

[0196] The specific implementation of step S602 generating the vector feature of the target sub-image, obtaining the first modal feature, and generating the initial modal feature of each prompt word in the prompt word group is similar to the specific implementation of step S302, which will not be repeated here.

[0197] Considering that the target sub-images captured from different target images, or the target sub-images captured from the same target image, may have different image sizes. For images with the same image size, the dimensions of the vector features that can be converted are also the same. Therefore, in order to ensure that any target sub-image obtained can be converted into image features of the same dimension, and then further converted into vector features of a predetermined dimension, the target sub-image can be scaled to a predetermined size before generating the first modal feature for the target sub-image, so as to be used in the vector feature generation process. Based on this, optionally, after acquiring the target sub-image, the target sub-image is scaled to a predetermined size, and after image feature extraction is performed on the target sub-image of the predetermined size, the extracted image features are converted into vector features of a predetermined dimension to obtain the first modal feature.

[0198] Exemplarily, after acquiring the target sub-image, the image size of the target sub-image is scaled to a fixed size, for example, 224×224, or 336×336, and then the scaled target sub-image is input into a module for generating the first modal feature to generate the first modal feature of the target sub-image.

[0199] In this implementation, by converting the dimensions of the image features of the target sub-image to obtain vector features of the target sub-image in a predetermined dimension, it is convenient to determine the similarity of each prompt word with respect to the target sub-image.

[0200] S603: Determine the recognition result corresponding to the target sub-image based on the similarity of each prompt word.

[0201] The recognition result is a result about the presentation status of the target object.

[0202] In the present application, after obtaining the similarity corresponding to each prompt word, the degree of similarity between the presentation state of the target object corresponding to the target sub-image and the presentation state described by the prompt word can be determined. The higher the similarity, the higher the possibility that the presentation state of the target object in the target sub-image is the presentation state described by the prompt word. Therefore, after determining the similarity corresponding to each prompt word, the presentation state of the target object described by the prompt word with high similarity can be determined as the presentation state of the target object corresponding to the target sub-image, thereby determining the recognition result corresponding to the target sub-image.

[0203] S604: Determine the presentation state of the target image for the target object based on the recognition result corresponding to the target sub-image.

[0204] In the present application, it is considered that there may be multiple regions of interest in a target image, that is, there may be at least one target sub-image in a target image. Therefore, after obtaining the recognition result for the target sub-image, the presentation state of at least one target sub-image for the target object can be determined based on the above recognition result, thereby determining the presentation state of the entire target image for the target object, so as to obtain the recognition result corresponding to the target image.

[0205] As can be seen from the above, in the image recognition method provided by the embodiment of the present application, after the target sub-image is captured from the target image, the target sub-image and the prompt phrase group related to the target object are input into the pre-trained contrastive learning model; accordingly, since the contrastive learning model reduces the influence of other non-keywords in the prompt words on the accuracy of the recognition result during the training process, when the contrastive learning model is used to identify the target object, for each prompt word in the prompt phrase group, the target sub-feature of the prompt word is spliced ​​with the designated sub-feature corresponding to the target sample prompt word to obtain the second modal feature of the prompt word, that is, the feature representing the target keyword of the prompt word is spliced ​​with the feature content of the designated feature part in the current second modal feature of the target sample prompt word when the contrastive learning model is trained to obtain the second modal feature of the prompt word; thereby, the similarity between the first modal feature of the target sub-image and the second modal feature of each prompt word is calculated, the similarity of the target sub-image to each prompt word is obtained, and based on the obtained similarity, the presentation state of the target image to the target object is determined. Through this solution, we can focus on the feature content of the keyword part in each prompt word in the prompt word group and use the specified feature part in the target sample prompt word to reduce the influence of non-keywords in the image recognition process, thereby achieving accurate recognition of the target object.

[0206] Optionally, in one implementation, the above step S604, determining the presentation state of the target image for the target object based on the recognition result corresponding to the target sub-image, may include the following steps:

[0207] Step F1: In response to the number of the target sub-image being one, the presentation state represented by the recognition result corresponding to the target sub-image is used as the presentation state of the target image for the target object.

[0208] In this implementation, if the number of captured target sub-images is one, it means that there is only one area in the target image where the target object exists. At this time, the presentation state represented by the recognition result corresponding to the target sub-image can be directly used as the presentation state of the overall target image for the target object to achieve automatic image recognition.

[0209] In this implementation, when the number of the target sub-image is one, the recognition result corresponding to the obtained target sub-image can be directly used as the recognition result corresponding to the entire target image, thereby improving the efficiency of automatic image recognition.

[0210] Optionally, in one implementation, the above step S604, determining the presentation state of the target image for the target object based on the recognition result corresponding to the target sub-image, may include the following steps:

[0211] Step F2: in response to: there are multiple target sub-images, determining the presentation state of the target image for the target object based on the recognition results corresponding to each target sub-image in a predetermined analysis manner;

[0212] Among them, the predetermined analysis methods include:

[0213] If the different presentation states of the target object include a first presentation state indicating the existence of the object and a second presentation state indicating the non-existence of the object, then when there is a target sub-image in each target sub-image whose recognition result indicates the first presentation state, the presentation state of the target image with respect to the target object is determined to be the first presentation state; otherwise, the presentation state of the target image with respect to the target object is determined to be the second presentation state;

[0214] If the different presentation states of the target object include different presentation states regarding specified attributes, then determine the target recognition result with the highest frequency among the recognition results corresponding to each target sub-image, and determine the presentation state of the target image for the target object as: the presentation state represented by the target recognition result.

[0215] In this implementation, if the number of captured target sub-images is multiple, it indicates that the target image contains multiple areas where target objects exist. At this time, it is necessary to determine the corresponding recognition results of each captured target sub-image, and thus, determine the presentation status of the target object for the target object based on the recognition results of each target sub-image.

[0216] Considering that different presentation states of the target object may include presentation states for whether the target object exists, for example, the presence of a tattoo and the absence of a tattoo; it may also include presentation states for specified attributes of the target object, for example, the presence of a black cat and the presence of a white cat.

[0217] Therefore, a predetermined analysis method is set in advance for different presentation states of the target object, so as to determine the presentation state of the target image for the target object based on the recognition results corresponding to each target sub-image. Specifically:

[0218] If the different presentation states of the target object include a first presentation state indicating the existence of the object and a second presentation state indicating the non-existence of the object, it is necessary to determine that in each target sub-image, when there is a target sub-image whose recognition result indicates the first presentation state, the presentation state of the target image for the target object can be determined as the first presentation state. Otherwise, the presentation state of the target image for the target object is determined as the second presentation state.

[0219] If the different presentation states of the target object include different presentation states regarding specified attributes, it is necessary to determine the target recognition result with the highest frequency among the corresponding recognition results in each target sub-image, thereby determining the presentation state of the target image for the target object as the presentation state represented by the target recognition result.

[0220] For example, different presentation states of the target object include different presentation states regarding tattoo colors. The target recognition result with the highest frequency indicates that the tattoo color in the image is dark. In this case, the presentation state of the target image for the target object is determined as the tattoo color in the image is dark, and the target image is determined to be an image including a dark tattoo.

[0221] In the present implementation, considering that when there are multiple target sub-images, the determination of the presentation state of the target object of the overall target image needs to be combined with the recognition results of each target sub-image; and considering that the presentation state of the target object includes multiple presentation methods, therefore, when there are multiple target sub-images, based on the different presentation states of the target object, the presentation state of the target object of the overall target image is determined using a corresponding analysis method, thereby improving the accuracy of automatic image recognition.

[0222] Optionally, in one implementation, the above step S603, determining the recognition result corresponding to the target sub-image based on the similarity of each prompt word, may include the following steps:

[0223] Step G1: for each prompt word, based on the similarity corresponding to the prompt word, determine the confidence score corresponding to the prompt word;

[0224] The confidence score corresponding to the prompt word is used to characterize the presentation state of the target object corresponding to the target sub-image, which is the probability of the presentation state described in the prompt word;

[0225] Step G2: Based on the confidence score corresponding to each prompt word, determine the recognition result corresponding to the target sub-image.

[0226] In this implementation, for each prompt word, after determining the similarity corresponding to the prompt word, the confidence score corresponding to the prompt word can be further determined, thereby determining the probability that the presentation state of the target object corresponding to the target sub-image is the presentation state described in the prompt word. In other words, the higher the confidence score corresponding to the prompt word, the higher the probability that the presentation state of the target object corresponding to the target sub-image is the presentation state described in the prompt word; thus, based on the confidence score corresponding to each prompt word, the recognition result corresponding to the target sub-image is determined.

[0227] In this implementation, the confidence score is used to digitize the probability that the presentation state of the target object corresponding to the target sub-image is the presentation state described in each prompt word, so as to match the target sub-image with the corresponding prompt word, thereby improving the accuracy of the recognition result corresponding to the target sub-image.

[0228] Optionally, in one implementation, step G2, determining the recognition result corresponding to the target sub-image based on the confidence score corresponding to each prompt word, may include the following steps:

[0229] Step G21: Select the highest confidence score from the confidence scores corresponding to each prompt word as the target confidence score;

[0230] Step G22: If the target confidence score is greater than a predetermined threshold, the recognition result corresponding to the target sub-image is determined as: the presentation state described by the prompt word corresponding to the target confidence score.

[0231] In this implementation, since the higher the confidence score corresponding to the determined prompt word, the higher the probability that the presentation state of the target object corresponding to the target sub-image is the presentation state described in the prompt word, after determining the confidence score corresponding to each prompt word, the highest confidence score is selected as the target confidence score.

[0232] However, although the determined confidence score is the highest confidence score, the probability that the presentation state of the target object represented by the confidence score is the presentation state described in the prompt word is still low. For example, generally speaking, the two can only be considered as a match when the confidence score is above 50%. At this time, if the highest confidence score determined is 35%, although the confidence score is the highest confidence score among the confidence scores of the target image for each prompt word, in the case where the confidence score is 50% to be considered as a match, the presentation state described by the prompt word corresponding to the confidence score is actually not matched with the presentation state of the target object corresponding to the target object.

[0233] Therefore, in order to improve the accuracy of the recognition result corresponding to the determined target sub-image, a predetermined threshold can be set in advance for the confidence score, that is, when the determined confidence score is greater than the predetermined threshold, it is characterized that the presentation state described by the prompt word is the presentation state of the target object corresponding to the target sub-image. Therefore, by comparing the relationship between the determined target confidence score and the predetermined threshold, it can be determined whether the presentation state described by the prompt word is the presentation state of the target object corresponding to the target sub-image.

[0234] If the target confidence score is greater than the above-mentioned predetermined threshold, the presentation state described by the prompt word corresponding to the target confidence score is the presentation state of the target object corresponding to the target sub-image, and the recognition result corresponding to the target sub-image can be determined as: the presentation state described by the prompt word corresponding to the target confidence score.

[0235] If the target confidence score is not greater than the predetermined threshold, the presentation state described by the prompt word corresponding to the target confidence score is not the presentation state of the target object corresponding to the target sub-image.

[0236] In this implementation, the accuracy of the recognition result corresponding to the determined target sub-image is further improved by presetting a predetermined threshold value regarding the confidence score.

[0237] Optionally, in one implementation, the above step G1, for each prompt word, based on the similarity corresponding to the prompt word, determines the confidence score corresponding to the prompt word, and may include the following steps:

[0238] Step G11: For each prompt word, normalize the similarity corresponding to the prompt word to obtain the confidence score corresponding to the prompt word.

[0239] In this implementation, after the similarity of each prompt word is obtained, the similarity corresponding to each prompt word may be normalized to obtain a confidence score corresponding to the prompt word.

[0240] Optionally, for each prompt word, a confidence score corresponding to the prompt word is calculated using a predetermined formula; wherein the predetermined formula includes:

[0241]

[0242] σ(z i ) represents the confidence score corresponding to the i-th prompt word about the target object, K represents the total number of prompt words in the prompt word group about the target object, z i Characterize the feature score corresponding to the i-th prompt word, z j Represents the feature score corresponding to the j-th prompt word.

[0243] The feature score corresponding to each prompt word is used to characterize the similarity between the prompt word and the target sub-image, that is, the similarity corresponding to each prompt word in the embodiment of the present application.

[0244] In this implementation, by standardizing the similarity corresponding to each prompt word to a specific interval, for example, [0, 1] or [-1, 1], etc., the data distribution has statistical significance, and the probability distribution of similarity is more statistically intuitive, thereby improving the comparability of the similarity corresponding to each prompt word.

[0245] Optionally, in one implementation, the above step S601, capturing the region of interest from the target image to obtain the target sub-image, may include the following steps:

[0246] Step H1: using a region detection algorithm for the carrier region, performing region detection on the target image to obtain a region of interest;

[0247] Step H2: intercept the region of interest in the target image to obtain a target sub-image.

[0248] In this implementation, after acquiring the target image, the region detection algorithm for the carrier region can be used to perform region detection on the target image to obtain the region of interest, and the region of interest in the target image can be intercepted to obtain the target sub-image.

[0249] Considering that the types of objects to be identified are different, the corresponding carrier areas are different, and accordingly, the region detection algorithms for the carrier areas are also different. For example, if the object to be identified is a tattoo, the corresponding carrier area is the human body area, and accordingly, the region detection algorithm for the carrier area can be a human body detection algorithm; if the object to be identified is a trademark, the corresponding carrier area is the edge area of ​​the target image, and accordingly, the region detection algorithm for the carrier area can be a trademark detection algorithm. Therefore, optionally, based on the type of object to be identified, the corresponding region detection algorithm is selected, and the target image is subjected to region detection to obtain the region of interest.

[0250] Among them, the above-mentioned human body detection algorithm can be an anchor-base (based on anchor frame) human body detection method, which is not specifically limited in the embodiments of the present application.

[0251] This implementation method can reduce the interference of other image elements in the image on image recognition by intercepting the region of interest in the target image, thereby improving the accuracy of automatic image recognition.

[0252] Next, in conjunction with a specific embodiment, the above step H1 is specifically introduced, using a region detection algorithm for the carrier region to perform region detection on the target image to obtain the region of interest. In this specific embodiment, the target object is a tattoo, and the region of interest is a human body region.

[0253] For the target image to be identified, the size information of the target image is determined. If the size information is not equal to the first size adapted by the convolutional neural network, the target image is scaled to the first size to obtain a first image, and the convolutional neural network is used to extract features of the obtained first image, thereby obtaining a first feature image corresponding to the above-mentioned first image.

[0254] In this way, for each pixel point in the obtained first feature image, multiple first anchor frames about the pixel point can be generated, so that for each first anchor frame, the confidence level of the existence of a human body in the first anchor frame can be determined.

[0255] Among them, the so-called anchor frame can be understood as: when identifying the target existing in the feature image obtained after processing by the convolutional neural network, if it is determined that there is a target belonging to the detected human body in a certain area in the feature image, the area can be identified using a graphic frame, and then, the graphic frame used to identify the area where the human body target exists in the feature image is the anchor frame.

[0256] Since there may be no human body in some areas of the feature image, or only part of the human body exists in the area framed by the first anchor frame, in order to ensure the accuracy of the determined image position information of the human body, a first preset score can be set in advance, so that the number of anchor frames that need to be processed later can be greatly reduced. Therefore, for each pixel point in the first feature image, after determining the confidence corresponding to each first anchor frame about the pixel point, each anchor frame with a confidence greater than the first preset score is retained as each first target anchor frame.

[0257] In this way, based on the position information between each first target anchor frame, the IOU (Intersection over Union) value between every two first target anchor frames in all the first target anchor frames can be calculated, where the so-called intersection over union ratio refers to the ratio of the area of ​​the overlapping area of ​​two first target anchor frames divided by the area of ​​the union area of ​​the two first target anchor frames.

[0258] For example, the area of ​​the first target anchor box J1 is S J1 , the area of ​​the first target anchor box J2 is S J2 , the overlapping area of ​​the first target anchor box J1 and the first target anchor box J2 is S J1J2 , then the intersection-and-union ratio between the first target anchor box J1 and the first target anchor box J2 can be calculated as

[0259] Thus, in each group of first target anchor frames whose calculated IOU values ​​are greater than the first preset IOU value, the position information corresponding to the first target anchor frame with the confidence score is selected and determined as the region of interest in the target image.

[0260] For example, for a target image, if the calculated IOU between any two first target anchor frames is greater than a first preset IOU value, it can be determined that the human bodies framed by the two first target anchor frames belong to the same human body. In this way, the position information corresponding to the first target anchor frame with the highest confidence among the two first target anchor frames can be determined as the position information of the region of interest in the target image.

[0261] Optionally, since there is an offset between the position of the preset first anchor frame and the actual position of the human body, a first relative offset can be preset, so that the first offset of the first anchor frame is determined while the confidence score of each first anchor frame is determined. In this way, each first target anchor frame whose first offset is not greater than the first relative offset can be retained, so that the first target anchor frame obtained can be filtered twice using the first relative offset. Then, through two filterings, each first anchor frame whose confidence score is greater than the first preset score and whose first offset is not greater than the first relative offset is determined as each first target anchor frame, further improving the accuracy of the image position information of the determined region of interest in the target image.

[0262] Optionally, in one embodiment, as Figure 7 As shown, an image recognition method provided in an embodiment of the present application may also include the following steps:

[0263] S605: Determine whether the presentation state of the target image for the target object meets the predetermined requirements; if not, execute step S606; otherwise, execute step S607;

[0264] S606: performing a predetermined processing operation on the target image;

[0265] The predetermined processing operation includes: using a predetermined image processing method to cover the target object in the target image, or terminating the uploading of the target image;

[0266] S607: Upload the target image.

[0267] In this embodiment, in order to maintain the network environment of the network platform, it is necessary to set a predetermined rule for the presentation state of the target object in advance, that is, an image that meets the predetermined rule is a normal image, and an image that does not meet the predetermined rule is an illegal image. For example, a human image without tattoos is a normal image, and a human image with tattoos is an illegal image.

[0268] Therefore, after determining the presentation status of the target image for the target object, it is possible to further determine whether the presentation status complies with the above-mentioned predetermined regulations, thereby determining whether the target image can be uploaded.

[0269] If the presentation state of the target image for the target object meets the predetermined requirements, the target image is characterized as a normal image, and the dissemination of the target image will not cause bad guidance, and the target image can be uploaded.

[0270] If the presentation state of the target image to the target object does not meet the predetermined regulations, the target image is characterized as an illegal image, and the spread of the target image may cause bad guidance. Therefore, in order to maintain the network environment of the network platform, the target image can be subjected to a predetermined processing operation.

[0271] Among them, in order to improve the image upload rate of the platform, the above-mentioned predetermined processing operation may include using a predetermined image processing method to cover the target object in the target image; in this way, when it is determined that the presentation state of the target image for the target object does not meet the predetermined regulations, the predetermined image processing method can be used to cover the target object in the target image, and thus, after covering the target object in the target image, an image recognition method provided by an embodiment of the present application is executed again on the covered target image until it is determined that the presentation state of the target image for the target object meets the predetermined regulations and the target image is a normal image, and the target image is uploaded.

[0272] In addition, considering that the number of images to be identified is too large, the above-mentioned predetermined processing operation may include terminating the uploading of the target image; in this way, when it is determined that the presentation state of the target image for the target object does not meet the predetermined regulations, the uploading of the target image is directly terminated.

[0273] Optionally, after terminating the upload of the target image, a reminder message indicating that the review has not been passed is sent to the corresponding client.

[0274] In this embodiment, target images uploaded to the network platform are classified according to predetermined regulations to ensure that the images uploaded to the network platform will not cause bad guidance and protect the network environment of the network platform.

[0275] Corresponding to the model training method provided in the above embodiment of the present application, the embodiment of the present application also provides a model training device, such as Figure 8 As shown, the device comprises:

[0276] An image acquisition module 810 is used to acquire a sample image; wherein the sample image is provided with a label for characterizing a presentation state of the sample object;

[0277] The similarity acquisition module 820 is used to input the sample image and each predetermined sample prompt word into a pre-trained contrastive learning model, so that the contrastive learning model generates the vector feature of the sample image, obtains the first modal feature of the sample image, determines the current second modal feature of each sample prompt word, and calculates the similarity between the first modal feature of the sample image and the second modal feature of each sample prompt word, so as to obtain the similarity of the sample image for each sample prompt word; wherein the pre-trained contrastive learning model is obtained by training with images containing objects other than the sample object on the basis of the pre-trained model; each sample prompt word is an image description text with the presentation state of the sample object as the description content, and any sample prompt word belongs to one of two prompt word types, the presentation states described by the two prompt word types are mutually exclusive, and the semantics of sample prompt words belonging to the same prompt word type are similar, and each sample prompt word is obtained by combining a target keyword and other non-keywords, and the target keyword is a keyword used to characterize a presentation state of the sample object; the initial second modal feature of each sample prompt word is the vector feature of the sample prompt word;

[0278] A loss value calculation module 830, configured to determine a loss value corresponding to the contrastive learning model based on the similarity of the sample image to each sample prompt word and the label of the sample image;

[0279] The parameter adjustment module 840 is used to adjust the parameters of the specified feature part in the current second modal feature of each sample prompt word when it is judged that the contrastive learning model does not meet the model convergence condition based on the loss value, and return to the step of obtaining the sample image; wherein the specified feature part in the current second modal feature of each sample prompt word is the feature part corresponding to other non-keywords in the current second modal feature of the sample prompt word.

[0280] As can be seen from the above, in the model training method provided in the embodiment of the present application, before model training, sample prompt words can be set in advance for sample objects, and each sample prompt word is an image description text with the presentation state of the sample object as the description content, and any sample prompt word belongs to one of two prompt word types, the presentation states described by the two prompt word types are mutually exclusive, and the sample prompt words belonging to the same prompt word type have similar semantics, and each sample prompt word is obtained based on a target keyword and other non-keyword combinations, wherein the above-mentioned target keyword is a keyword used to characterize a presentation state of the sample object. Considering that other non-keywords in the prompt word have a greater impact on the second modal features of the entire prompt word, that is, the contrastive learning model is relatively stable in generating vector features corresponding to the target keywords in the second modal features of the prompt word, while for other non-keywords, the other non-keywords are the same but the combination methods are different, or the other non-keywords are different, and the generated vector features may be different. In other words, the contrastive learning model is unstable in generating vector features corresponding to other non-keywords in the second modal features of the prompt word, and the generated vector features are quite different. Therefore, by determining the similarity of the sample image for each sample prompt word and the difference in the pre-set labels of the sample image itself, the loss value of the current second modal feature of each sample prompt word generated by the contrastive learning model is determined. Therefore, when it is judged based on the loss value that the contrastive learning model does not meet the model convergence condition, the parameters of the feature parts corresponding to other non-keywords in the current second modal features of each sample prompt word are adjusted to reduce the influence of other non-keywords in the prompt word on the accuracy of the recognition result, thereby ensuring the recognition accuracy of the image using the contrastive learning model.

[0281] In addition, the combination of a target keyword and other non-keywords to obtain sample prompt words can reduce the tedious process of manually designing prompt words, and at the same time, reduce the possibility of low accuracy of prompt words when manually designing prompt words. Moreover, the accuracy of the sample prompt words obtained by the above combination method is high. Therefore, when training the model, a small number of sample prompt words and sample images can be used to train a comparative learning model with high recognition effect.

[0282] Optionally, in a specific implementation, the loss value calculation module 830 includes:

[0283] A confidence acquisition submodule, used to determine the confidence of the sample image for each sample prompt word based on the similarity of the sample image for each sample prompt word; wherein the confidence of the sample image for any sample prompt word is used to characterize the presentation state of the sample object in the sample image, which is the predicted probability of the presentation state described in the sample prompt word;

[0284] The loss value determination submodule is used to determine the loss value corresponding to the contrastive learning model based on the confidence of the sample image for each sample prompt word and the label of the sample image.

[0285] Optionally, in a specific implementation, the parameter adjustment module 840 is specifically used to:

[0286] When it is determined based on the loss value that the contrastive learning model does not meet the model convergence condition, the gradient value of the feature vector corresponding to the other non-keywords in the current second modal feature of each sample prompt word is calculated based on the loss value;

[0287] The gradient value is used to adjust the feature vector corresponding to the specified feature part in the current second modal feature of the sample prompt word.

[0288] Optionally, in a specific implementation, the confidence acquisition submodule is specifically used to:

[0289] The similarity of the sample image for each sample prompt word is normalized to obtain the confidence of the sample image for each sample prompt word.

[0290] Optionally, in a specific implementation, the sample object is a tattoo.

[0291] Corresponding to the image recognition method provided in the above embodiment of the present application, the embodiment of the present application also provides an image recognition device, such as Fig. 9 As shown, the device comprises:

[0292] The image capture module 910 is used to capture an area of ​​interest from a target image to obtain a target sub-image; wherein the area of ​​interest is a carrier area where the target object to be identified is located in the target image when the target object to be identified exists in the target image;

[0293] The similarity determination module 920 is used to input the target sub-image and the prompt word group into a pre-trained contrastive learning model so that the contrastive learning model generates the vector feature of the target sub-image to obtain the first modal feature, and generates the initial modal feature of each prompt word in the prompt word group, and, for each prompt word, concatenates the target sub-feature in the initial modal feature of the prompt word with the designated sub-feature corresponding to the target sample prompt word to obtain the second modal feature of the prompt word; calculates the similarity between the first modal feature of the target sub-image and the second modal feature of each prompt word respectively, to obtain the similarity of the target sub-image for each prompt word; and In the embodiment, each prompt word in the prompt word group is an image description text with the presentation state of the target object as the description content, and the presentation states described by different prompt words in the same prompt word group are mutually exclusive, and the contrastive learning model is obtained by training based on a model training method provided in an embodiment of the present application; the target sample prompt word is a sample prompt word in each sample prompt word whose other non-keywords match the other non-keywords of the prompt word, the designated sub-feature is the feature content of the designated feature part in the current second modal feature of the target sample prompt word when the contrastive learning model is trained, and the target sub-feature is a feature that characterizes the target keyword;

[0294] A result determination module 930 is used to determine the recognition result corresponding to the target sub-image based on the similarity of each prompt word; wherein the recognition result is a result about the presentation state of the target object;

[0295] The image determination module 940 is used to determine the presentation state of the target image for the target object based on the recognition result corresponding to the target sub-image.

[0296] As can be seen from the above, in the image recognition method provided by the embodiment of the present application, after the target sub-image is captured from the target image, the target sub-image and the prompt phrase group related to the target object are input into the pre-trained contrastive learning model; accordingly, since the contrastive learning model reduces the influence of other non-keywords in the prompt words on the accuracy of the recognition result during the training process, when the contrastive learning model is used to identify the target object, for each prompt word in the prompt phrase group, the target sub-feature of the prompt word is spliced ​​with the designated sub-feature corresponding to the target sample prompt word to obtain the second modal feature of the prompt word, that is, the feature representing the target keyword of the prompt word is spliced ​​with the feature content of the designated feature part in the current second modal feature of the target sample prompt word when the contrastive learning model is trained to obtain the second modal feature of the prompt word; thereby, the similarity between the first modal feature of the target sub-image and the second modal feature of each prompt word is calculated, the similarity of the target sub-image to each prompt word is obtained, and based on the obtained similarity, the presentation state of the target image to the target object is determined. Through this solution, we can focus on the feature content of the keyword part in each prompt word in the prompt word group and use the specified feature part in the target sample prompt word to reduce the influence of non-keywords in the image recognition process, thereby achieving accurate recognition of the target object.

[0297] Optionally, in a specific implementation, the result determination module 930 includes:

[0298] A confidence determination submodule is used to determine, for each prompt word, a confidence score corresponding to the prompt word based on the similarity corresponding to the prompt word; wherein the confidence score corresponding to the prompt word is used to characterize the presentation state of the target sub-image corresponding to the target object, which is the probability of the presentation state described in the prompt word;

[0299] The result determination submodule is used to determine the recognition result corresponding to the target sub-image based on the confidence score corresponding to each prompt word.

[0300] Optionally, in a specific implementation, the result determination submodule is specifically used to:

[0301] From the confidence scores corresponding to each prompt word, select the highest confidence score as the target confidence score;

[0302] If the target confidence score is greater than a predetermined threshold, the recognition result corresponding to the target sub-image is determined as: the presentation state described by the prompt word corresponding to the target confidence score.

[0303] Optionally, in a specific implementation, the image determination module 940 is specifically configured to:

[0304] In response to the number of the target sub-image being one, the presentation state represented by the recognition result corresponding to the target sub-image is used as the presentation state of the target image for the target object.

[0305] Optionally, in a specific implementation, the image determination module 940 is specifically configured to:

[0306] In response to the number of the target sub-images being multiple, determining, in a predetermined analysis manner, a presentation state of the target image for the target object based on recognition results corresponding to each of the target sub-images;

[0307] Wherein, the predetermined analysis method includes:

[0308] If the different presentation states of the target object include a first presentation state indicating the existence of the object and a second presentation state indicating the non-existence of the object, then when there is a target sub-image in each of the target sub-images whose recognition result indicates the first presentation state, the presentation state of the target image with respect to the target object is determined to be the first presentation state; otherwise, the presentation state of the target image with respect to the target object is determined to be the second presentation state;

[0309] If the different presentation states of the target object include different presentation states regarding specified attributes, then determine the target recognition result with the highest frequency among the recognition results corresponding to each of the target sub-images, and determine the presentation state of the target image for the target object as: the presentation state represented by the target recognition result.

[0310] Optionally, in a specific implementation, the target object is a tattoo, and the carrier area is a human body area.

[0311] Optionally, in a specific implementation, the device further includes:

[0312] The image filtering module is used to determine whether the presentation state of the target image for the target object meets the predetermined regulations; if not, a predetermined processing operation is performed on the target image, otherwise, the target image is uploaded; wherein the predetermined processing operation includes: using a predetermined image processing method to cover the target object in the target image, or terminating the upload of the target image.

[0313] The present application also provides an electronic device, such as Fig.10 As shown, it includes a processor 1001, a communication interface 1002, a memory 1003 and a communication bus 1004, wherein the processor 1001, the communication interface 1002, and the memory 1003 communicate with each other through the communication bus 1004.

[0314] Memory 1003, used for storing computer programs;

[0315] The processor 1001 is used to implement the model training method and / or the image recognition method described in any of the above embodiments when executing the program stored in the memory 1003.

[0316] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0317] The communication interface is used for communication between the above terminal and other devices.

[0318] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0319] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0320] In another embodiment provided in the present application, a computer-readable storage medium is also provided, in which a computer program is stored. When the computer program is executed by a processor, the model training method and / or the image recognition method described in any one of the above embodiments are implemented.

[0321] In another embodiment provided in the present application, a computer program product comprising instructions is also provided, which, when executed on a computer, enables the computer to execute the model training method and / or the image recognition method described in any one of the above embodiments.

[0322] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive Solid State Disk (SSD)), etc.

[0323] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0324] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, electronic device embodiment, computer-readable storage medium embodiment, and computer program product embodiment, since they are basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0325] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A model training method, characterized in that: The method comprises: Acquire a sample image; wherein the sample image is provided with a label for characterizing a presentation state of the sample object; The sample image and each predetermined sample prompt word are input into a pre-trained contrastive learning model, so that the contrastive learning model generates a vector feature of the sample image, obtains a first modal feature of the sample image, determines a current second modal feature of each sample prompt word, and calculates the similarity between the first modal feature of the sample image and the second modal feature of each sample prompt word, so as to obtain the similarity of the sample image to each sample prompt word; wherein the pre-trained contrastive learning model is obtained by training with images containing objects other than the sample object on the basis of the pre-trained model; each sample prompt word is an image description text with a presentation state of the sample object as a description content, and any sample prompt word belongs to one of two prompt word types, the presentation states described by the two prompt word types are mutually exclusive, and the semantics of sample prompt words belonging to the same prompt word type are similar, and each sample prompt word is obtained by combining a target keyword and other non-keywords, and the target keyword is a keyword used to characterize a presentation state of the sample object; the initial second modal feature of each sample prompt word is the vector feature of the sample prompt word; Determining a loss value corresponding to the contrastive learning model based on the similarity of the sample image to each sample prompt word and the label of the sample image; When it is determined based on the loss value that the contrastive learning model does not meet the model convergence conditions, parameters are adjusted for the designated feature parts in the current second modal features of each sample prompt word, and the step of obtaining the sample image is returned; wherein the designated feature parts in the current second modal features of each sample prompt word are feature parts corresponding to other non-keywords in the current second modal features of the sample prompt word.

2. The method according to claim 1, characterized in that The determining, based on the similarity of the sample image to each sample prompt word and the label of the sample image, a loss value corresponding to the contrastive learning model includes: Based on the similarity of the sample image to each sample prompt word, determine the confidence of the sample image to each sample prompt word; wherein the confidence of the sample image to any sample prompt word is used to characterize the presentation state of the sample object in the sample image, which is the predicted probability of the presentation state described in the sample prompt word; Based on the confidence of the sample image for each sample prompt word and the label of the sample image, a loss value corresponding to the contrastive learning model is determined.

3. The method according to claim 1, characterized in that When it is determined based on the loss value that the contrastive learning model does not meet the model convergence condition, adjusting parameters of a designated feature part in the current second modal feature of each sample prompt word includes: When it is determined based on the loss value that the contrastive learning model does not meet the model convergence condition, the gradient value of the feature vector corresponding to the other non-keywords in the current second modal feature of each sample prompt word is calculated based on the loss value; The gradient value is used to adjust the feature vector corresponding to the specified feature part in the current second modal feature of the sample prompt word.

4. The method according to claim 2, characterized in that: The determining, based on the similarity of the sample image to each sample prompt word, the confidence of the sample image to each sample prompt word comprises: The similarity of the sample image for each sample prompt word is normalized to obtain the confidence of the sample image for each sample prompt word.

5. The method according to claim 1, characterized in that The sample object is a tattoo.

6. An image recognition method, characterized in that: The method comprises: From the target image, a region of interest is intercepted to obtain a target sub-image; wherein the region of interest is a carrier region where the target object to be identified is located in the target image when the target object to be identified exists in the target image; The target sub-image and the prompt phrase are input into a pre-trained contrastive learning model, so that the contrastive learning model generates the vector features of the target sub-image to obtain the first modal features, and generates the initial modal features of each prompt word in the prompt phrase group, and, for each prompt word, concatenates the target sub-features in the initial modal features of the prompt word with the designated sub-features corresponding to the target sample prompt word to obtain the second modal features of the prompt word; calculates the similarity between the first modal features of the target sub-image and the second modal features of each prompt word, respectively, to obtain the similarity of the target sub-image to each prompt word; wherein, the prompt Each prompt word in the phrase is an image description text with the presentation state of the target object as the description content, and the presentation states described by different prompt words in the same prompt word group are mutually exclusive, and the contrastive learning model is obtained by training based on the method described in any one of claims 1 to 5; the target sample prompt word is a sample prompt word among the sample prompt words, whose other non-keywords match the other non-keywords of the prompt word, the designated sub-feature is the feature content of the designated feature part in the current second modal feature of the target sample prompt word when the contrastive learning model is trained, and the target sub-feature is the feature characterizing the target keyword; Based on the similarity of each prompt word, determining the recognition result corresponding to the target sub-image; wherein the recognition result is a result about the presentation state of the target object; Based on the recognition result corresponding to the target sub-image, a presentation state of the target image for the target object is determined.

7. The method according to claim 6, characterized in that The determining the recognition result corresponding to the target sub-image based on the similarity corresponding to each prompt word includes: For each prompt word, based on the similarity corresponding to the prompt word, determine the confidence score corresponding to the prompt word; wherein the confidence score corresponding to the prompt word is used to characterize the presentation state of the target object corresponding to the target sub-image, which is the probability of the presentation state described in the prompt word; Based on the confidence score corresponding to each prompt word, a recognition result corresponding to the target sub-image is determined.

8. The method according to claim 7, characterized in that The step of determining the recognition result corresponding to the target sub-image based on the confidence score corresponding to each prompt word includes: From the confidence scores corresponding to each prompt word, select the highest confidence score as the target confidence score; If the target confidence score is greater than a predetermined threshold, the recognition result corresponding to the target sub-image is determined as: the presentation state described by the prompt word corresponding to the target confidence score.

9. The method according to any one of claims 6 to 8, characterized in that: The determining, based on the recognition result corresponding to the target sub-image, a presentation state of the target image for the target object includes: In response to the number of the target sub-image being one, the presentation state represented by the recognition result corresponding to the target sub-image is used as the presentation state of the target image for the target object.

10. The method according to any one of claims 6 to 8, characterized in that: The determining, based on the recognition result corresponding to the target sub-image, a presentation state of the target image for the target object includes: In response to the number of the target sub-images being multiple, determining, in a predetermined analysis manner, a presentation state of the target image for the target object based on recognition results corresponding to each of the target sub-images; Wherein, the predetermined analysis method includes: If the different presentation states of the target object include a first presentation state indicating the existence of the object and a second presentation state indicating the non-existence of the object, then when there is a target sub-image in each of the target sub-images whose recognition result indicates the first presentation state, the presentation state of the target image with respect to the target object is determined to be the first presentation state; otherwise, the presentation state of the target image with respect to the target object is determined to be the second presentation state; If the different presentation states of the target object include different presentation states regarding specified attributes, then determine the target recognition result with the highest frequency among the recognition results corresponding to each of the target sub-images, and determine the presentation state of the target image for the target object as: the presentation state represented by the target recognition result.

11. The method according to claim 6, characterized in that The target object is a tattoo, and the carrier area is a human body area.

12. The method according to claim 6, characterized in that The method further comprises: Determine whether the presentation state of the target image for the target object meets the predetermined requirement; if not, perform a predetermined processing operation on the target image; otherwise, upload the target image; The predetermined processing operation includes: using a predetermined image processing method to cover the target object in the target image, or terminating the uploading of the target image.

13. A model training device, characterized in that: The device comprises: An image acquisition module, used to acquire a sample image; wherein the sample image is provided with a label for characterizing a presentation state of the sample object; A similarity acquisition module is used to input the sample image and each predetermined sample prompt word into a pre-trained contrastive learning model, so that the contrastive learning model generates a vector feature of the sample image, obtains the first modal feature of the sample image, determines the current second modal feature of each sample prompt word, and calculates the similarity between the first modal feature of the sample image and the second modal feature of each sample prompt word, so as to obtain the similarity of the sample image for each sample prompt word; wherein the pre-trained contrastive learning model is obtained by training with images containing objects other than the sample object on the basis of the pre-trained model; each sample prompt word is an image description text with the presentation state of the sample object as the description content, and any sample prompt word belongs to one of two prompt word types, the presentation states described by the two prompt word types are mutually exclusive, and the semantics of sample prompt words belonging to the same prompt word type are similar, and each sample prompt word is obtained by combining a target keyword and other non-keywords, and the target keyword is a keyword used to characterize a presentation state of the sample object; the initial second modal feature of each sample prompt word is the vector feature of the sample prompt word; A loss value calculation module, used to determine the loss value corresponding to the contrastive learning model based on the similarity of the sample image to each sample prompt word and the label of the sample image; A parameter adjustment module is used to adjust the parameters of the specified feature part in the current second modal feature of each sample prompt word when it is judged that the contrastive learning model does not meet the model convergence condition based on the loss value, and return to the step of obtaining the sample image; wherein the specified feature part in the current second modal feature of each sample prompt word is the feature part corresponding to other non-keywords in the current second modal feature of the sample prompt word.

14. An image recognition device, characterized in that: The device comprises: An image capture module is used to capture an area of ​​interest from a target image to obtain a target sub-image; wherein the area of ​​interest is a carrier area where the target object to be identified is located in the target image when the target object to be identified exists in the target image; A similarity determination module is used to input the target sub-image and the prompt word group into a pre-trained contrastive learning model so that the contrastive learning model generates the vector feature of the target sub-image to obtain the first modal feature, and generates the initial modal feature of each prompt word in the prompt word group, and, for each prompt word, splices the target sub-feature in the initial modal feature of the prompt word with the designated sub-feature corresponding to the target sample prompt word to obtain the second modal feature of the prompt word; calculates the similarity between the first modal feature of the target sub-image and the second modal feature of each prompt word respectively, to obtain the similarity of the target sub-image for each prompt word; wherein , each prompt word in the prompt word group is an image description text with the presentation state of the target object as the description content, and the presentation states described by different prompt words in the same prompt word group are mutually exclusive, and the contrastive learning model is trained based on the method described in any one of claims 1 to 5; the target sample prompt word is a sample prompt word in each sample prompt word whose other non-keywords match the other non-keywords of the prompt word, the designated sub-feature is the feature content of the designated feature part in the current second modal feature of the target sample prompt word when the contrastive learning model is trained, and the target sub-feature is a feature that characterizes the target keyword; A result determination module, used to determine the recognition result corresponding to the target sub-image based on the similarity of each prompt word; wherein the recognition result is a result about the presentation state of the target object; The image determination module is used to determine the presentation state of the target image for the target object based on the recognition result corresponding to the target sub-image.

15. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1-5 when executing a program stored in a memory, and / or implementing the method steps described in any one of claims 6-12.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps of any one of claims 1-5 are implemented, and / or the method steps of any one of claims 6-12 are implemented.

Images