Image processing method, apparatus, computing device, and medium

Abstract

The application provides an image processing method and device, a computing device and a medium, and relates to the technical field of deep learning. After obtaining a target image to be processed, the application determines the first similarity of the target image and the first images of each category based on the first feature vector corresponding to the target image and the second feature vectors respectively corresponding to the plurality of first images, and determines the second similarity of the target image and the plurality of first images based on the first color distribution information of the target region in the target image and the second color distribution information of the target region in the plurality of first images, so that the image category to which the target image belongs can be determined based on the first similarity and the second similarity. By introducing the second similarity determined based on the color distribution information as a supplementary basis for determining the image category, the accuracy of the determined image category is higher, and the accuracy of image classification is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning, and in particular to an image processing method and device, a computing device and a medium. BACKGROUND

[0002] With the continuous development of deep learning, image classification, image segmentation, image recognition and the like are carried out through deep learning, which has become an important technical means for image processing. However, due to the high cost of image sample collection, and even the fact that some image samples cannot be collected at all, the training effect of the image classification model is poor, thereby resulting in low accuracy of image classification.

[0003] Based on the above situation, the small sample learning scheme emerges as the times require. Small sample learning is to use a sample data amount much smaller than that required by big data deep learning to achieve a processing effect close to or even exceeding that of big data deep learning. Through small sample learning, an image classification model with high image classification accuracy can be obtained in the case of limited image samples.

[0004] In the related art, a public image sample with low collection difficulty is mainly used to train a convolutional neural network (CNN) to obtain a pre-training model, and then the pre-training model is trained by using an image sample corresponding to an actual image classification requirement to obtain an image classification model that can meet the actual image classification requirement.

[0005] In the above implementation process, due to the large number of sample image categories and image quantities, and the large scale variation of sample images of the same category, and the presence of a large amount of noise in the sample images, the training effect of the model is poor, thereby resulting in poor classification accuracy of the image classification model obtained by training. SUMMARY

[0006] The present application provides an image processing method, device, computing device and medium to solve the deficiencies in the related art.

[0007] According to a first aspect of an embodiment of the present application, an image processing method is provided, which comprises:

[0008] obtaining a target image to be processed;

[0009] determining a first similarity between the target image and first images of each category based on a first feature vector corresponding to the target image and second feature vectors respectively corresponding to the first images, the first images being labeled with image categories, the first images corresponding to a plurality of image categories;

[0010] determine a second similarity between the target image and the first images based on the first color distribution information of the target region in the target image and the second color distribution information of the target region in the first images;

[0011] determine an image category to which the target image belongs from the image categories of the first images based on the first similarity and the second similarity.

[0012] In an embodiment of the present application, the color distribution information is a color distribution spectrum; the second similarity between the target image and the first images is determined based on the first color distribution information of the target region in the target image and the second color distribution information of the target region in the first images, and the determination includes:

[0013] determine a color distribution similarity between the first color distribution spectrum and each second color distribution spectrum, respectively;

[0014] for at least one first image belonging to any image category, determine a color distribution similarity with a maximum value in the color distribution similarities corresponding to the at least one first image as the second similarity between the target image and the at least one first image.

[0015] In an embodiment of the present application, before the second similarity between the target image and the first images is determined based on the first color distribution information of the target region in the target image and the second color distribution information of the target region in the first images, the method further includes:

[0016] determine the target region from the target image and the first images based on an attention matrix of the image classification model;

[0017] obtain the first color distribution information of the target region in the target image and the second color distribution information of the target region in each first image.

[0018] In an embodiment of the present application, the first similarity is a cosine similarity, and the cosine similarity is used to indicate a cosine distance between a feature vector of the target image and a feature vector of each first image of the category;

[0019] Before the first similarity between the target image and the first images of each category is determined based on the first feature vector corresponding to the target image and the second feature vectors corresponding to the first images, the method further includes:

[0020] determine a first vector sequence for representing the target image and a plurality of second vector sequences for representing the first images based on the target image and the first images through an embedding layer of the image classification model;

[0021] Based on the first vector sequence and the plurality of second vector sequences, the first feature vector and the plurality of second feature vectors are obtained through an encoder of the image classification model, and based on the first feature vector and the plurality of second feature vectors, the cosine similarity between the target image and the plurality of first images is determined.

[0022] In an embodiment of the present application, based on the first vector sequence and the plurality of second vector sequences, the first feature vector and the plurality of second feature vectors are obtained through an encoder of the image classification model, and based on the first feature vector and the plurality of second feature vectors, the cosine similarity between the target image and the plurality of first images is determined, including:

[0023] The first vector sequence and the plurality of second vector sequences are input into the encoder, and through the encoder, the first feature vector corresponding to the first vector sequence and the plurality of second feature vectors corresponding to the plurality of second vector sequences are determined.

[0024] For at least one first image belonging to any image category, a mean vector of the second feature vector corresponding to the at least one first image is determined.

[0025] The cosine distance between the first feature vector and the mean vector is determined, and based on the cosine distance, the cosine similarity between the target image and the at least one first image is determined.

[0026] In an embodiment of the present application, based on the cosine distance, the cosine similarity between the target image and the plurality of first images is determined, including any of the following:

[0027] In the case where the cosine distance is greater than a preset distance threshold, the cosine similarity is determined as a first value;

[0028] In the case where the cosine distance is less than or equal to a preset distance threshold, the cosine similarity is determined as a second value.

[0029] In an embodiment of the present application, based on the first similarity and the second similarity, the image category to which the target image belongs is determined from the image categories of the plurality of first images, including:

[0030] The first similarity and the second similarity are weighted and summed according to a first weight corresponding to the first similarity and a second weight corresponding to the second similarity, to obtain an image similarity between the target image and the first images of each category;

[0031] The image category corresponding to the target image similarity with the maximum value in each image similarity is determined as the image category to which the target image belongs.

[0032] In an embodiment of the present application, the image classification model is obtained by pre-training;

[0033] The training process of the image classification model comprises:

[0034] Obtaining a plurality of first sample images labeled with sample image categories;

[0035] Inputting the plurality of first sample images into an initial visual converter model, and determining predicted image categories of the plurality of first sample images through the initial visual converter model;

[0036] Training the initial visual converter model based on a first loss function indicating the difference between the predicted image categories of the plurality of first sample images and the sample image categories of the plurality of first sample images until a preset training completion condition is met, to obtain a trained visual converter model;

[0037] Obtaining the image classification model based on the trained visual converter model.

[0038] In an embodiment of the present application, the trained visual converter model comprises an embedding layer, a converter encoder and a multi-head perceiver;

[0039] Obtaining the image classification model based on the trained visual converter model comprises:

[0040] Obtaining an embedding layer and a converter encoder from the trained visual converter model to form an initial image classification model;

[0041] Obtaining a plurality of second sample images labeled with similarity true values;

[0042] For any two second sample images in the plurality of second sample images, inputting the two second sample images into the initial image classification model, and outputting first similarity prediction values of the two second sample images through the initial image classification model;

[0043] Training the initial image classification model based on a second loss function indicating the difference between the first similarity prediction values and the similarity true values.

[0044] In an embodiment of the present application, the method further comprises:

[0045] Determining a second similarity prediction result between color distribution information of target regions in the two second sample images based on an attention matrix of the initial image classification model;

[0046] Determining a second similarity prediction value based on the second similarity prediction result and a preset similarity threshold;

[0047] The initial image classification model is trained based on a second loss function indicating a difference between the first similarity prediction value and the similarity ground truth, and a third loss function indicating a difference between the second similarity prediction value and the similarity ground truth.

[0048] In an embodiment of the present application, the second similarity prediction value is determined based on the second similarity prediction result and a preset similarity threshold, including any one of the following:

[0049] In a case where the second similarity prediction result is greater than the preset similarity threshold, the second similarity prediction value is determined as a first numerical value;

[0050] In a case where the second similarity prediction result is less than or equal to the preset similarity threshold, the second similarity prediction value is determined as a second numerical value.

[0051] In an embodiment of the present application, the initial image classification model is trained based on a second loss function indicating a difference between the first similarity prediction value and the similarity ground truth, and a third loss function indicating a difference between the second similarity prediction value and the similarity ground truth, including:

[0052] The second loss function and the third loss function are weighted and summed based on a first initial weight corresponding to the second loss function and a second initial weight corresponding to the third loss function, to obtain a target loss function;

[0053] The initial image classification model is trained based on the target loss function until a training completion condition is met, to obtain the image classification model.

[0054] According to a second aspect of an embodiment of the present application, an image processing apparatus is provided, which includes:

[0055] An image acquisition module is configured to acquire a target image to be processed;

[0056] A similarity determination module is configured to determine a first similarity between the target image and first images of each category based on a first feature vector corresponding to the target image and second feature vectors respectively corresponding to the first images, wherein the first images have been labeled with image categories, and the first images correspond to a plurality of image categories.

[0057] The similarity determination module is further configured to determine a second similarity between the target image and the first images based on first color distribution information of a target region in the target image and second color distribution information of the target region in the first images.

[0058] The category determining module is configured to determine, based on the first similarity and the second similarity, an image category to which the target image belongs from among image categories of the plurality of first images.

[0059] In an embodiment of the present application, the color distribution information is a color distribution spectrum.

[0060] The similarity determining module is configured to determine, based on the first color distribution information of the target region in the target image and the second color distribution information of the target region in the plurality of first images, a second similarity between the target image and the plurality of first images.

[0061] respectively determine a color distribution similarity between the first color distribution spectrum and each second color distribution spectrum.

[0062] For at least one first image belonging to any image category, a color distribution similarity with a maximum value in color distribution similarities corresponding to the at least one first image is determined as the second similarity between the target image and the at least one first image.

[0063] In an embodiment of the present application, the device further comprises:

[0064] The region determining module is configured to determine, based on an attention matrix of the image classification model, the target region from the target image and the plurality of first images, respectively.

[0065] The information obtaining module is configured to obtain first color distribution information of a target region in the target image and second color distribution information of the target region in each first image.

[0066] In an embodiment of the present application, the first similarity is a cosine similarity, and the cosine similarity is used to indicate a cosine distance between the first feature vector corresponding to the target image and a second feature vector corresponding to each first image of the category.

[0067] The similarity determining module, when determining the first similarity between the target image and each first image of the category based on the plurality of first feature vectors corresponding to the target image and the plurality of second feature vectors corresponding to the plurality of first images, comprises a sequence determining submodule and a similarity determining submodule.

[0068] The sequence determining submodule is configured to determine, based on the target image and the plurality of first images, a first vector sequence for representing the target image and a plurality of second vector sequences for representing the plurality of first images through an embedding layer of the image classification model.

[0069] The similarity determination submodule is configured to: acquire the first feature vector and the plurality of second feature vectors by an encoder of the image classification model based on the first vector sequence and the plurality of second vector sequences; and determine the cosine similarity between the target image and the plurality of first images based on the first feature vector and the plurality of second feature vectors.

[0070] In an embodiment of the present application, the similarity determination submodule, when determining the cosine similarity between the target image and the plurality of first images based on the first vector sequence and the plurality of second vector sequences, comprises a vector determination unit, a distance determination unit and a similarity determination unit.

[0071] The vector determination unit is configured to input the first vector sequence and the plurality of second vector sequences into the encoder, and determine the first feature vector corresponding to the first vector sequence and the plurality of second feature vectors corresponding to the plurality of second vector sequences by the encoder.

[0072] The vector determination unit is further configured to determine a mean vector of the second feature vector corresponding to at least one first image belonging to any image category.

[0073] The distance determination unit is configured to determine the cosine distance between the first feature vector and the mean vector.

[0074] The similarity determination unit is configured to determine the cosine similarity between the target image and the at least one first image based on the cosine distance.

[0075] In an embodiment of the present application, when determining the cosine similarity between the target image and the plurality of first images based on the cosine distance, the similarity determination unit is configured to perform any one of the following:

[0076] In a case where the cosine distance is greater than a preset distance threshold, the cosine similarity is determined as a first value.

[0077] In a case where the cosine distance is less than or equal to the preset distance threshold, the cosine similarity is determined as a second value.

[0078] In an embodiment of the present application, when determining the image category to which the target image belongs from the image categories of the plurality of first images based on the first similarity and the second similarity, the category determination module is configured to:

[0079] weighting and summing the first similarity and the second similarity according to a first weight corresponding to the first similarity and a second weight corresponding to the second similarity, to obtain an image similarity between the target image and the first image of each category;

[0080] determining an image category corresponding to a target image similarity with the maximum value in the image similarities as the image category to which the target image belongs.

[0081] In an embodiment of the present application, the image classification model is obtained in advance through training;

[0082] The device further comprises:

[0083] The sample acquisition module is configured to acquire a plurality of first sample images that have been labeled with sample image categories.

[0084] The sample category determination module is configured to input the plurality of first sample images into an initial visual converter model, and determine predicted image categories of the plurality of first sample images through the initial visual converter model.

[0085] The training module is configured to train the initial visual converter model based on a first loss function indicating a difference between the predicted image categories of the plurality of first sample images and the sample image categories of the plurality of first sample images, until a preset training completion condition is met, to obtain a trained visual converter model.

[0086] The model acquisition module is configured to acquire the image classification model based on the trained visual converter model.

[0087] In an embodiment of the present application, the trained visual converter model comprises an embedding layer, a converter encoder, and a multi-head perceiver.

[0088] When the model acquisition module is configured to acquire the image classification model based on the trained visual converter model, the model acquisition module comprises a model acquisition submodule, a sample acquisition submodule, a first prediction submodule, and a first training submodule.

[0089] The model acquisition submodule is configured to acquire an embedding layer and a converter encoder from the trained visual converter model to form an initial image classification model.

[0090] The sample acquisition submodule is configured to acquire a plurality of second sample images that have been labeled with similarity true values.

[0091] The first prediction submodule is configured to input any two second sample images in the plurality of second sample images into the initial image classification model, and output first similarity prediction values of the two second sample images through the initial image classification model.

[0092] The first training submodule is configured to train the initial image classification model based on a second loss function indicating a difference between the first similarity prediction value and the similarity true value.

[0093] In an embodiment of the present application, the model obtaining module further comprises a result determining submodule, a second prediction submodule and a second training submodule.

[0094] The result determining submodule is configured to determine a second similarity prediction result between color distribution information of target regions in the two second sample images based on the attention matrix of the initial image classification model.

[0095] The second prediction submodule is configured to determine a second similarity prediction value based on the second similarity prediction result and a preset similarity threshold.

[0096] The second training submodule is configured to train the initial image classification model based on a second loss function indicating a difference between the first similarity prediction value and the similarity true value, and a third loss function indicating a difference between the second similarity prediction value and the similarity true value.

[0097] In an embodiment of the present application, when the second prediction submodule is configured to determine a second similarity prediction value based on the second similarity prediction result and a preset similarity threshold, the second prediction submodule is configured to perform any one of the following:

[0098] In a case where the second similarity prediction result is greater than the preset similarity threshold, the second similarity prediction value is determined as a first numerical value.

[0099] In a case where the second similarity prediction result is less than or equal to the preset similarity threshold, the second similarity prediction value is determined as a second numerical value.

[0100] In an embodiment of the present application, when the second training submodule is configured to train the initial image classification model based on a second loss function indicating a difference between the first similarity prediction value and the similarity true value, and a third loss function indicating a difference between the second similarity prediction value and the similarity true value, the second training submodule is configured to:

[0101] The second training submodule is configured to train the initial image classification model based on a second loss function indicating a difference between the first similarity prediction value and the similarity true value, and a third loss function indicating a difference between the second similarity prediction value and the similarity true value.

[0102] The second training submodule is configured to train the initial image classification model based on a second loss function indicating a difference between the first similarity prediction value and the similarity true value, and a third loss function indicating a difference between the second similarity prediction value and the similarity true value.

[0103] According to a third aspect of the embodiments of the present application, a computing device is provided, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the operations performed by the image processing method provided by the first aspect and any one of the embodiments of the first aspect when executing the computer program.

[0104] According to a fourth aspect of the embodiments of the present application, a computer readable storage medium is provided, which stores a program, and the program, when executed by a processor, implements the operations performed by the image processing method provided by the first aspect and any one of the embodiments of the first aspect.

[0105] According to a fifth aspect of the embodiments of the present application, a computer program product is provided, which comprises a computer program, and the computer program, when executed by a processor, implements the operations performed by the image processing method provided by the first aspect and any one of the embodiments of the first aspect.

[0106] According to the above embodiments, after obtaining the target image to be processed, the first similarity of the target image and the first images of each category is determined based on the first feature vector corresponding to the target image and the second feature vectors respectively corresponding to the plurality of first images, and the second similarity of the target image and the plurality of first images is determined based on the first color distribution information of the target region in the target image and the second color distribution information of the target region in the plurality of first images, so that the image category to which the target image belongs can be determined based on the first similarity and the second similarity. By introducing the second similarity determined based on the color distribution information as a supplementary basis for determining the image category, the accuracy of the determined image category is higher, and the accuracy of image classification is improved.

[0107] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF DRAWINGS

[0108] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present application and, together with the specification, serve to explain the principles of the present application.

[0109] Figure 1 is a flowchart of an image processing method according to an embodiment of the present application;

[0110] Figure 2 is a flowchart of an image processing method according to an embodiment of the present application;

[0111] Figure 3 is a flowchart of a training image classification model according to an embodiment of the present application;

[0112] Figure 4 is a block diagram of an image processing apparatus according to an embodiment of the present application;

[0113] Figure 5 is a structural schematic diagram of a computing device according to an embodiment of the present application. DETAILED DESCRIPTION

[0114] The exemplary embodiments will be described in detail herein with reference to the attached drawings. In the following description, the same numbers are used to denote the same elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not meant to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the present application as detailed in the appended claims.

[0115] The present application provides an image processing method for determining an image category to which a target image to be processed belongs. The target image can be of various types, such as an article image, an animal image, etc., and the present application does not limit the target image. Taking the target image as an article image as an example, the image category to which the target image belongs is determined, i.e., the category to which the article contained in the target image belongs is determined. For example, if the article contained in the target image is a thermos cup, the image category to which the target image belongs is a thermos cup image. Taking the target image as an animal image as an example, the image category to which the target image belongs is determined, i.e., the category to which the animal contained in the target image belongs is determined. For example, if the animal contained in the target image is a Siberian tiger, the image category to which the target image belongs is a Siberian tiger image.

[0116] The image processing method described above can be executed by a computing device, which can be a server, such as a server, multiple servers, a server cluster, a cloud computing platform, etc. Alternatively, the computing device can also be a terminal device, such as a mobile phone, a tablet computer, a game console, a portable computer, a desktop computer, an advertising machine, an all-in-one machine, etc. The present application does not limit the type and number of the computing device.

[0117] The above is only an exemplary description of the application scenario of the present application, and does not constitute a limitation on the application scenario of the present application. In more possible implementations, the present application can be applied in the image processing process of various types of target images.

[0118] After introducing the application scenario of the present application, the image processing method provided by the present application will be described in detail in combination with the multiple optional embodiments of the present application.

[0119] Figure 1is a flow chart of an image processing method according to an embodiment of the present application, referring to Figure 1 The method comprises:

[0120] Step 101, obtaining a target image to be processed.

[0121] In a possible implementation, the computing device can receive a user-triggered image processing instruction, and then in response to the image processing instruction, obtain the target image to be processed, and then trigger the subsequent image processing process.

[0122] The computing device can obtain the target image to be processed from the associated image database, and can also receive images sent by other devices, and take the received images as the target images to be processed.

[0123] In another possible implementation, the computing device can, after receiving the images sent by other devices, take the received images as the target images to be processed by itself, and then trigger the subsequent image processing process.

[0124] The above are only two exemplary ways of triggering the image processing process, and in more possible implementations, other ways can also be used to trigger the image processing process, which are not limited by the present application.

[0125] Step 102, determining a first similarity between the target image and each first image of a plurality of first images based on a first feature vector corresponding to the target image and a plurality of second feature vectors respectively corresponding to the plurality of first images, the plurality of first images having been labeled with image categories, and the plurality of first images corresponding to a plurality of image categories.

[0126] Step 103, determining a second similarity between the target image and the plurality of first images based on first color distribution information of a target region in the target image and second color distribution information of the target region in the plurality of first images.

[0127] The target image and the first image both include a plurality of types of information, but only part of the plurality of types of information can assist in judging the image category, and other irrelevant information cannot assist in judging the image category, the target region in the target image is a region in the target image that contains information that can assist in judging the image category, and the target region in the first image is a region in the first image that contains information that can assist in judging the image category. Optionally, the target region can also be referred to as a region of interest.

[0128] Step 104, determining an image category to which the target image belongs from the image categories of the plurality of first images based on the first similarity and the second similarity.

[0129] The application determines the first similarity of the target image and the first images of each category based on the first feature vector corresponding to the target image and the second feature vectors respectively corresponding to the first images, and determines the second similarity of the target image and the first images based on the first color distribution information of the target region in the target image and the second color distribution information of the target region in the first images, so that the image category to which the target image belongs can be determined based on the first similarity and the second similarity. By introducing the second similarity determined based on the color distribution information as a supplementary basis for determining the image category, the accuracy of the determined image category is higher, and the accuracy of image classification is improved.

[0130] After introducing the basic implementation process of the application, several non-limiting embodiments of the application will be introduced.

[0131] In an embodiment of the application, the first similarity involved in the step 102 is a cosine similarity, which is used to indicate the cosine distance between the first feature vector corresponding to the target image and the second feature vectors respectively corresponding to the first images of each category.

[0132] In the case where the first similarity is a cosine similarity, the cosine similarity of the target image and the first images of each category can be determined by the image classification model to obtain the first similarity of the target image and the first images of each category. That is, the step 102 can be implemented by the image classification model.

[0133] The image classification model can be a Vision Transformer model, and optionally, the image classification model can also be other types of models, which are not limited by the application. Taking the image classification model as a Vision Transformer model as an example, the image classification model can include an Embedding layer and a Transformer Encoder. For convenience of description, the Transformer Encoder is referred to as an encoder hereinafter.

[0134] When the step 102 is implemented by the image classification model, the process of determining the first similarity of the target image and the first images of each category based on the first feature vector corresponding to the target image and the second feature vectors respectively corresponding to the first images can include the following steps:

[0135] In step 1021, the first vector sequence for representing the target image and the second vector sequences for representing the first images are determined based on the target image and the first images by the Embedding layer of the image classification model.

[0136] In the process of determining the first vector sequence for representing the target image and the plurality of second vector sequences for representing the plurality of first images through the embedding layer of the image classification model, the following can be implemented:

[0137] Taking the process of determining the first vector sequence for representing the target image as an example, in the process of determining the first vector sequence for representing the target image through the embedding layer of the image classification model, the target image is divided into a plurality of image blocks conforming to the set image block size according to the set image block size, and then each image block is mapped into a one-dimensional vector in a linear mapping manner, so as to obtain the one-dimensional vector corresponding to each image block, and then the plurality of one-dimensional vectors are spliced to obtain the first vector sequence.

[0138] It should be noted that the determination process of each second vector sequence is the same as that of the first vector sequence, which will not be described here.

[0139] In step 1022, the first feature vector and the plurality of second feature vectors are obtained based on the first vector sequence and the plurality of second vector sequences through the encoder of the image classification model, and the cosine similarity between the target image and the plurality of first images is determined based on the first feature vector and the plurality of second feature vectors.

[0140] In a possible implementation, the above step 1022 can be implemented by the following steps:

[0141] Step one, input the first vector sequence and the plurality of second vector sequences into the encoder, and determine the first feature vector corresponding to the first vector sequence and the plurality of second feature vectors corresponding to the plurality of second vector sequences through the encoder.

[0142] In a possible implementation, the first vector sequence and the plurality of second vector sequences are input into the encoder, and the first vector sequence and the plurality of second vector sequences are respectively subjected to convolution processing through the encoder to obtain the first feature vector corresponding to the first vector sequence and the plurality of second feature vectors corresponding to the plurality of second vector sequences.

[0143] Step two, for at least one first image belonging to any image category, determine the mean vector of the second feature vector corresponding to the at least one first image.

[0144] It should be noted that the plurality of first images correspond to a plurality of image categories, and each category can include at least one first image. When determining the mean vector, the mean vector of the second feature vector corresponding to the first image of each category is determined respectively, so as to obtain the mean vector corresponding to the first image of each category.

[0145] In a possible implementation, for at least one first image belonging to any image category, at least one second feature vector corresponding to the at least one first image is averaged to obtain a mean vector of the at least one second feature vector.

[0146] Step three, determining a cosine distance between the first feature vector and the mean vector, and determining a cosine similarity between the target image and the at least one first image based on the cosine distance.

[0147] In the determination of the cosine distance between the first feature vector and the mean vector, the cosine value of the included angle between the first feature vector and the mean vector can be determined first, and then the cosine value determined is subtracted by 1 to obtain the cosine distance between the first feature vector and the mean vector. That is, the cosine distance between the first feature vector and the mean vector can be determined by the following formula (1):

[0148] dist(A, B) = 1 - cos(A, B) (1)

[0149] Wherein, A represents the first feature vector, B represents the mean vector, dist(A, B) represents the cosine distance between the first feature vector and the mean vector, and cos(A, B) represents the cosine value of the included angle between the first feature vector and the mean vector.

[0150] In the determination of the cosine similarity between the target image and the at least one first image based on the cosine distance, the cosine distance can be compared with a preset distance threshold, so as to determine the cosine similarity according to the comparison result.

[0151] In a possible implementation, when the cosine distance is greater than the preset distance threshold, the cosine similarity is determined as a first value.

[0152] In another possible implementation, when the cosine distance is less than or equal to the preset distance threshold, the cosine similarity is determined as a second value.

[0153] Wherein, the preset distance threshold can be any positive value, and the present application does not limit the specific value of the preset distance threshold. The first value can be 0, and the second value can be 1.

[0154] By comparing the cosine distance with the preset distance threshold, the cosine similarity is mapped to 0 or 1 according to the comparison result, so as to realize the binary processing of the cosine similarity, highlight the cosine similarity between two images, and improve the accuracy of image classification.

[0155] The steps two and three are only described by taking the processing of the first image of any image category as an example. The processing of the first image of different image categories is the same, and the processing of the first image of other image categories is not described here.

[0156] In the case of using the Vision Transformer model as the image classification model, an attention mechanism is used in the encoder of the image classification model, so as to determine the target region in the target image and the target region in the first image based on the attention mechanism, so that only the color distribution information of the target region needs to be determined subsequently, and then the color distribution similarity is determined based on the color distribution information of the target region, reducing the processing pressure of the computing device, thereby improving the processing speed of the computing device, and further improving the image classification speed.

[0157] In a possible implementation, the encoder of the image classification model includes an attention matrix, and the target region can be determined from the target image and the first image through the attention matrix. That is, before step 103 determines the second similarity of the target image and the plurality of first images based on the first color distribution information of the target region in the target image and the second color distribution information of the target region in the plurality of first images, the method further includes the following steps:

[0158] Step one, based on the attention matrix of the image classification model, the target region is determined from the target image and the plurality of first images, respectively.

[0159] Taking the process of determining the target region from the target image as an example, the pixel value of the target image can be multiplied by the attention matrix to obtain the target region in the target image. In the attention matrix, the value of the matrix element corresponding to the pixel included in the target region is 1, and the value of the matrix element corresponding to the pixel included in the non-target region is 0, so that the determination of the target region can be realized by multiplying the pixel value of the target image by the attention matrix.

[0160] The above only takes the determination of the target region in the target image as an example, and the determination of the target region in each first image is the same, and the determination of the target region in each first image is not described here.

[0161] Step two, obtaining the first color distribution information of the target region in the target image, and the second color distribution information of the target region in each first image.

[0162] The color distribution information can be a color distribution spectrum.

[0163] In a possible implementation, the RGB colors included in the target region in the target image and the positions where the colors are distributed can be counted to obtain a first color distribution spectrum of the target region in the target image, and the RGB colors included in the target region in each first image and the positions where the colors are distributed can be counted to obtain a second color distribution spectrum of the target region in the plurality of first images.

[0164] After the first color distribution spectrum of the target region in the target image and the second color distribution spectrum of the target region in each first image are obtained, the second similarity can be determined based on the obtained first color distribution spectrum and second color distribution spectrum.

[0165] In some embodiments, for the step 103, when the color distribution similarity between the target image and the plurality of first images is determined based on the first color distribution information of the target region in the target image and the second color distribution information of the target region in the plurality of first images, the following steps can be used:

[0166] The step 1031 comprises determining the color distribution similarity between the first color distribution spectrum and each second color distribution spectrum.

[0167] In a possible implementation, for any second color distribution spectrum in the plurality of second color distribution spectra, the color distribution similarity between the first color distribution spectrum and the second color distribution spectrum can be determined by comparing the RGB colors and the positions where the colors are distributed indicated by the first color distribution spectrum and the RGB colors and the positions where the colors are distributed indicated by the second color distribution spectrum.

[0168] The above only takes the determination of the color distribution similarity between the first color distribution spectrum and any second color distribution spectrum as an example, and the determination of the color distribution similarity between each second color distribution spectrum and the first color distribution spectrum is the same, which will not be described here.

[0169] Optionally, after the color distribution similarity is determined, the color distribution similarity can be compared with a preset similarity threshold, so as to realize the binary processing of the color distribution similarity.

[0170] In a possible implementation, when the color distribution similarity is greater than the preset similarity threshold, the color distribution similarity is determined as a first numerical value.

[0171] In another possible implementation, when the color distribution similarity is less than or equal to the preset similarity threshold, the color distribution similarity is determined as a second numerical value.

[0172] The preset similarity threshold can be any positive value, and the present application does not limit the specific value of the preset similarity threshold. The first value can be 0, and the second value can be 1.

[0173] The color distribution similarity is compared with the preset similarity threshold, so that the color distribution similarity is mapped to 0 or 1 according to the comparison result, thereby realizing the binarization processing of the color distribution similarity, highlighting the color distribution similarity between the two images, and further improving the accuracy of image classification.

[0174] In step 1032, for at least one first image belonging to any image category, the color distribution similarity with the maximum value in the color distribution similarity corresponding to the at least one first image is determined as the second similarity between the target image and the at least one first image.

[0175] It should be noted that the plurality of first images correspond to a plurality of image categories, and each category can include at least one first image. When determining the second similarity between the target image and the plurality of first images, the second similarity between the target image and the first image of each category is determined respectively.

[0176] In a possible implementation, for at least one first image belonging to any category, after determining the color distribution similarity between the first color distribution spectrum and the at least one second color distribution spectrum corresponding to the at least one first image, the determined color distribution similarity is sorted in descending order of value, so that the color distribution similarity in the first position (i.e., the color distribution similarity with the maximum value) is determined as the second similarity between the target image and the first image of this category.

[0177] By introducing the attention mechanism in the image classification process, the attention matrix is separated from the Transformer encoder of the image classification model, so that the attention matrix is applied to the unprocessed target image and the first image. More mining can be performed on the image information contained in the target region of the target image and the target region of the first image, and then the color distribution information of the target region of the target image and the color distribution information of the target region of the first image are obtained. The color distribution information is used as a supplementary basis for judging similarity, which is intuitive, fast, and consistent with human visual habits.

[0178] In an embodiment of the present application, for the above-mentioned step 104, when determining the image category to which the target image belongs from the image categories of the plurality of first images based on the first similarity and the second similarity, the following steps can be used to achieve this:

[0179] In step 1041, the first similarity and the second similarity are weighted and summed according to the first weight corresponding to the first similarity and the second weight corresponding to the second similarity, to obtain an image similarity between the target image and the first image of each category.

[0180] In a possible implementation, the image similarity between the target image and the first image of each category can be determined by the following formula (2):

[0181] S = aS1 + bS2 (2)

[0182] In the above formula (2), S represents the image similarity, S1 represents the first similarity, S2 represents the second similarity, a represents the first weight, and b represents the second weight.

[0183] The first weight and the second weight are weight values obtained through model training, and the specific obtaining process is described in the following model training process, which will not be repeated here.

[0184] In step 1042, the image category corresponding to the target image similarity with the maximum value in the image similarity is determined as the image category to which the target image belongs.

[0185] The image classification process described in each of the above embodiments can refer to Figure 2 , Figure 2 is a flowchart of an image processing method according to an embodiment of the present application. The target image can be an image in the query set, and the first image can be an image in the support set. Through the image processing method provided by the present application, the Vision Transformer as an image classification model can be used to process the images in the query set and the support set respectively, to obtain the feature vector corresponding to the target image in the query set and the feature vector corresponding to each first image in the support set, and then perform mean value processing on the feature vector corresponding to each first image in the support set to obtain the mean vector corresponding to each first image in the support set, so as to determine the first similarity between the target image and the first image of each category based on the mean vector and the feature vector corresponding to the target image. In the Vision Transformer, the attention matrix is included, based on which the color distribution spectrum of the region of interest can be obtained from the target image in the query set and the plurality of first images in the support set respectively, so as to determine the second similarity between the target image and the first image of each category based on the obtained color distribution spectrum, to weight and sum the first similarity and the second similarity of the first image corresponding to the same image category to obtain the image similarity, and then to determine the image category based on the image similarity with the maximum value by sorting the image similarity.

[0186] Wherein, the support set includes N classification labels, each classification label includes K first images, that is, the support set includes N x K first images, when the color distribution spectrum of the first image is obtained, N x K color distribution spectra can be obtained, and then the color distribution similarity is determined based on the N x K color distribution spectra, N x K-1 color distribution similarities can be obtained, and then the second similarity between the target image and the first image of different categories is determined based on the N x K-1 color distribution similarities. N and K are both arbitrary positive integer values.

[0187] It should be noted that when the image is processed by the image classification model, the image is first processed by the embedding layer of the image classification model to obtain a one-dimensional vector corresponding to each image block, and then the one-dimensional vectors corresponding to the plurality of image blocks are spliced to flatten the one-dimensional vectors corresponding to the plurality of image blocks to obtain a vector sequence corresponding to the image, and then the first similarity is determined by the Transformer encoder. The Transformer encoder includes an attention matrix.

[0188] The above Figure 2 The above

[0189] Optionally, after the image category to which the target image belongs is determined by the above-mentioned embodiments, the image recognition, image segmentation, image detection and the like can be further performed based on the determined image category, and the subsequent processing process is not limited by the present application.

[0190] It should be noted that the above-mentioned embodiments are based on the trained image classification model to classify the image. The image classification model is obtained by pre-training, and the training process of the image classification model is described below.

[0191] During the training of the image classification model, the transfer learning method can be used to pre-train the initial visual converter model by using public sample images to obtain a visual converter model with preliminary image classification capability, and then continue the model training based on specific small sample images and the trained visual converter model to obtain an image classification model that can be used to identify the image category to which the small sample image belongs. The specific model training process is described below in combination with several non-limiting embodiments.

[0192] In one embodiment, the training process of the image classification model is described with reference to Figure 3 , Figure 3 is a flowchart for training an image classification model according to an embodiment of the present application, as shown in Figure 3As shown, the process of training the image classification model includes the following steps:

[0193] Step 301, obtaining a plurality of first sample images labeled with sample image categories.

[0194] Among them, the first sample image is a public sample image with small acquisition difficulty. Optionally, the images in the large data set provided by the prior art can be used as the first sample image. For example, the images in large-scale data sets such as ImageNet, ImageNet-21k and JFT300M can be used as the first sample image.

[0195] Step 302, inputting the plurality of first sample images into an initial visual converter model, and determining the predicted image categories of the plurality of first sample images through the initial visual converter model.

[0196] Among them, the initial visual converter (Vision Transformer) model includes an Embedding layer, a Transformer Encoder and a Multilayer Perceptron Head (MLP Head).

[0197] Next, taking the processing process of any two first sample images as an example, the processing process inside the initial Vision Transformer will be described:

[0198] Inputting any two first sample images into the initial Vision Transformer, through the Embedding layer of the initial Vision Transformer, dividing the two first sample images into a plurality of image blocks according to the set image block size, and then respectively mapping each image block to a one-dimensional vector by using a linear mapping manner, thereby obtaining the one-dimensional vector corresponding to each image block, and then respectively splicing the plurality of one-dimensional vectors to obtain the sample vector sequence corresponding to the two first sample images.

[0199] Inputting the two sample vector sequences into the Transformer Encoder, determining the cosine similarity of the two first sample images through the Transformer Encoder, and the specific process is the same as the processing process of step 1022, which will not be repeated here.

[0200] Based on the determined cosine similarity and the sample image category of one of the first sample images, the predicted image categories of the two first sample images are determined through the MLP Head.

[0201] Step 303, training the initial visual converter model based on the first loss function indicating the difference between the predicted image category of the plurality of first sample images and the sample image category of the plurality of first sample images until a preset training completion condition is met, to obtain a trained visual converter model.

[0202] Wherein, the trained visual converter model also includes an embedding layer, a converter encoder and a multi-head perceiver, which are the same as the structure of the initial visual converter model.

[0203] It should be noted that the process of training the model in steps 302 and 303 above is an iterative process, that is, after determining the predicted image category of the first first sample image, the model parameters of the initial visual converter model are updated based on the first loss function indicating the difference between the predicted image category of the first first sample image and the sample image category of the first first sample image to obtain a visual converter model after the first parameter update; continue to process the second first sample image, and after determining the predicted image category of the second first sample image, continue to update the model parameters based on the first loss function indicating the difference between the predicted image category of the second first sample image and the sample image category of the second first sample image to obtain a visual converter model after the second parameter update. In this way, the training is continued until the preset training completion condition is met, and the visual converter model that meets the training completion condition is taken as the trained visual converter model.

[0204] Wherein, the training completion condition is that the first loss function meets a set condition, or the number of iterations reaches a set number, etc., which is not limited by the present application.

[0205] Step 304, obtaining an image classification model based on the trained visual converter model.

[0206] By first pre-training the initial visual converter model using a large data set with a large number of samples, a visual converter model with image classification capability is obtained. However, the classification accuracy of the visual converter model for specific small sample data is poor. Then, based on the small sample data, the initial image classification model obtained based on the visual converter model is further trained to obtain an image classification model with high classification accuracy for specific small sample data, thereby improving the classification accuracy of the image classification model trained by small sample learning and improving the model training effect.

[0207] In an embodiment of the present application, the above step 304 can include the following steps:

[0208] Step 3041, obtaining an embedding layer and a converter encoder from the trained visual converter model to form an initial image classification model.

[0209] That is, the initial image classification model includes an embedding layer and a transformer encoder.

[0210] Step 3042, obtaining a plurality of second sample images labeled with similarity true values.

[0211] Among them, the first sample image is a small sample image with greater difficulty to obtain. Optionally, the images in the small sample dataset provided by the prior art can be used as the second sample images. For example, the images in the miniImageNet dataset can be used as the second sample images.

[0212] Optionally, the plurality of second sample images can be directly used as the first images when subsequently performing image classification.

[0213] Step 3043, for any two second sample images in the plurality of second sample images, inputting the two second sample images into the initial image classification model, and outputting first similarity prediction values of the two second sample images through the initial image classification model.

[0214] The process of this step 3043 is the same as the process of determining the first similarity through the Embedding layer and the Transformer Encoder in the above-mentioned step 302. For details, please refer to the above-mentioned step 302, which will not be repeated here.

[0215] Step 3044, training the initial image classification model based on a second loss function indicating the difference between the first similarity prediction value and the similarity true value.

[0216] It should be noted that the process of training the model in the above-mentioned steps 3043 and 3044 is an iterative process, that is, after the first similarity prediction value of the first pair of second sample images is determined, the model parameters of the initial image classification model are updated based on the second loss function indicating the difference between the first similarity prediction value of the first pair of second sample images and the similarity true value of the first pair of first sample images, to obtain the image classification model after the first parameter update; continue to process the second pair of second sample images, after the first similarity prediction value of the second pair of second sample images is determined, continue to update the model parameters based on the second loss function indicating the difference between the first similarity prediction value of the second pair of second sample images and the similarity true value of the second pair of second sample images, to obtain the image classification model after the second parameter update. In this way, the process of training the model based on the plurality of second sample images is realized.

[0217] In an embodiment of the present application, when training the initial image classification model, the second similarity prediction value determined based on the attention matrix of the initial image classification model can also be combined to train the initial image classification model. That is, after step 3033 is performed, the model training process can also be completed through steps 3045 to 3047.

[0218] Step 3045, determining a second similarity prediction result between the color distribution information of the target region in the two second sample images based on the attention matrix of the initial image classification model.

[0219] The specific process involved in this step 3045 can refer to steps 1031 and 1032 described above, and will not be repeated here.

[0220] Step 3046, determining a second similarity prediction value based on the second similarity prediction result and a preset similarity threshold.

[0221] In a possible implementation, when the second similarity prediction result is greater than the preset similarity threshold, the second similarity prediction value is determined as a first numerical value.

[0222] In another possible implementation, when the second similarity prediction result is less than or equal to the preset similarity threshold, the second similarity prediction value is determined as a second numerical value.

[0223] The specific process involved in this step 3046 can refer to step 1031 described above, and will not be repeated here.

[0224] Step 3047, training the initial image classification model based on a second loss function indicating the difference between the second similarity prediction value and the similarity true value, and a third loss function indicating the difference between the second similarity prediction value and the similarity true value.

[0225] In a possible implementation, step 3047 can include the following steps:

[0226] Step one, weighting and summing the second loss function and the third loss function based on a first initial weight corresponding to the second loss function and a second initial weight corresponding to the third loss function, to obtain a target loss function.

[0227] In a possible implementation, the target loss function can be determined by the following formula (3):

[0228] Loss=α′Loss1+β′Loss2 (3)

[0229] In the above formula (3), Loss represents the target loss function, Loss1 represents the second loss function, Loss2 represents the third loss function, a' represents the first initial weight, and b' represents the second initial weight.

[0230] Step two, based on the target loss function, the initial image classification model is trained until the training completion condition is met, and an image classification model is obtained.

[0231] It should be noted that the process of training the model in the above step one and step two is an iterative process, and the specific process of iterative processing is the same as the above content, which will not be repeated here.

[0232] It should be noted that when the model parameters of the initial image classification model are adjusted based on the target loss function, the values of the first initial weight and the second initial weight can also be adjusted until the training completion condition is met, that is, the first weight and the second weight used in the image classification process can be obtained. The process of adjusting the values of the first initial weight and the second initial weight is also an iterative process.

[0233] It should be noted that the above model training process can be performed by a server, and after the model training is completed, the server deploys the trained image classification model to a terminal device used for image classification. It should be noted that if the computing device used for image classification is a server, the server can directly perform image classification through the trained image classification model after the model training is completed.

[0234] Embodiments of the present application also propose an image processing device, see Figure 4 , Figure 4 is a block diagram of an image processing device according to an embodiment of the present application, which comprises:

[0235] The image acquisition module 401 is configured to acquire a target image to be processed.

[0236] The similarity determination module 402 is configured to determine the first similarity between the target image and the first images of each category based on the first feature vector corresponding to the target image and the second feature vectors respectively corresponding to the plurality of first images, wherein the plurality of first images have been labeled with image categories, and the plurality of first images correspond to a plurality of image categories.

[0237] The similarity determination module 402 is further configured to determine the second similarity between the target image and the plurality of first images based on the first color distribution information of the target region in the target image and the second color distribution information of the target region in the plurality of first images.

[0238] The category determining module 403 is configured to determine, from the image categories of the plurality of first images, an image category to which the target image belongs based on the first similarity and the second similarity.

[0239] In an embodiment of the present application, the color distribution information is a color distribution spectrum.

[0240] The similarity determining module 402 is configured to:

[0241] determine a color distribution similarity between the first color distribution spectrum and each second color distribution spectrum, respectively.

[0242] For at least one first image belonging to any image category, the color distribution similarity with the maximum value in the color distribution similarities corresponding to the at least one first image is determined as the second similarity between the target image and the at least one first image.

[0243] In an embodiment of the present application, the device further comprises:

[0244] The region determining module is configured to determine the target region from the target image and the plurality of first images based on the attention matrix of the image classification model.

[0245] The information obtaining module is configured to obtain the first color distribution information of the target region in the target image and the second color distribution information of the target region in each first image.

[0246] In an embodiment of the present application, the first similarity is a cosine similarity, and the cosine similarity is used to indicate a cosine distance between the first feature vector corresponding to the target image and the second feature vector corresponding to each first image of the category.

[0247] The similarity determining module 402 comprises a sequence determining submodule and a similarity determining submodule when determining the first similarity between the target image and each first image of the category based on the plurality of first feature vectors corresponding to the target image and the second feature vectors corresponding to the plurality of first images.

[0248] The sequence determining submodule is configured to determine, based on the target image and the plurality of first images, a first vector sequence for representing the target image and a plurality of second vector sequences for representing the plurality of first images through an embedding layer of the image classification model.

[0249] The similarity determination submodule is configured to: acquire a first feature vector and a plurality of second feature vectors by an encoder of the image classification model based on the first vector sequence and the plurality of second vector sequences; and determine a cosine similarity between the target image and the plurality of first images based on the first feature vector and the plurality of second feature vectors.

[0250] In an embodiment of the present application, the similarity determination submodule, when determining the cosine similarity between the target image and the plurality of first images based on the first vector sequence and the plurality of second vector sequences by the encoder of the image classification model, includes a vector determination unit, a distance determination unit, and a similarity determination unit.

[0251] The vector determination unit is configured to: input the first vector sequence and the plurality of second vector sequences into the encoder; and determine the first feature vector corresponding to the first vector sequence and the plurality of second feature vectors corresponding to the plurality of second vector sequences by the encoder.

[0252] The vector determination unit is further configured to determine a mean vector of the plurality of second feature vectors.

[0253] The distance determination unit is configured to determine a cosine distance between the first feature vector and the mean vector.

[0254] The similarity determination unit is configured to determine the cosine similarity between the target image and the plurality of first images based on the cosine distance.

[0255] In an embodiment of the present application, the similarity determination unit, when determining the cosine similarity between the target image and the plurality of first images based on the cosine distance, is configured to perform any one of the following:

[0256] In a case where the cosine distance is greater than a preset distance threshold, the cosine similarity is determined as a first numerical value.

[0257] In a case where the cosine distance is less than or equal to the preset distance threshold, the cosine similarity is determined as a second numerical value.

[0258] In an embodiment of the present application, the category determination module 403, when determining the image category to which the target image belongs from the image categories of the plurality of first images based on the first similarity and the second similarity, is configured to:

[0259] weight and sum the first similarity and the second similarity according to a first weight corresponding to the first similarity and a second weight corresponding to the second similarity to obtain an image similarity between the target image and the first images of each category.

[0260] The image category corresponding to the target image similarity with the maximum value in the respective image similarities is determined as the image category to which the target image belongs.

[0261] In an embodiment of the present application, the image classification model is obtained in advance through training.

[0262] The apparatus further includes:

[0263] The sample obtaining module is configured to obtain a plurality of first sample images that have been labeled with sample image categories.

[0264] The sample category determining module is configured to input the plurality of first sample images into the initial visual transformer model, and determine predicted image categories of the plurality of first sample images through the initial visual transformer model.

[0265] The training module is configured to train the initial visual transformer model based on a first loss function indicating a difference between the predicted image categories of the plurality of first sample images and the sample image categories of the plurality of first sample images, until a preset training completion condition is met, to obtain a trained visual transformer model.

[0266] The model obtaining module is configured to obtain the image classification model based on the trained visual transformer model.

[0267] In an embodiment of the present application, the trained visual transformer model includes an embedding layer, a transformer encoder, and a multi-head perceiver.

[0268] When the model obtaining module is configured to obtain the image classification model based on the trained visual transformer model, the model obtaining module includes a model obtaining submodule, a sample obtaining submodule, a first prediction submodule, and a first training submodule.

[0269] The model obtaining submodule is configured to obtain the embedding layer and the transformer encoder from the trained visual transformer model to form an initial image classification model.

[0270] The sample obtaining submodule is configured to obtain a plurality of second sample images that have been labeled with similarity true values.

[0271] The first prediction submodule is configured to, for any two second sample images in the plurality of second sample images, input the two second sample images into the initial image classification model, and output a first similarity prediction value of the two second sample images through the initial image classification model.

[0272] The first training submodule is configured to train the initial image classification model based on a second loss function indicating a difference between the first similarity prediction value and the similarity true value.

[0273] In an embodiment of the present application, the model acquisition module further comprises a result determination submodule, a second prediction submodule and a second training submodule.

[0274] The result determination submodule is configured to determine a second similarity prediction result between the color distribution information of the target region in the two second sample images based on the attention matrix of the initial image classification model.

[0275] The second prediction submodule is configured to determine a second similarity prediction value based on the second similarity prediction result and a preset similarity threshold.

[0276] The second training submodule is configured to train the initial image classification model based on a second loss function indicating a difference between the first similarity prediction value and the similarity ground truth, and a third loss function indicating a difference between the second similarity prediction value and the similarity ground truth.

[0277] In an embodiment of the present application, when the second prediction submodule is configured to determine the second similarity prediction value based on the second similarity prediction result and the preset similarity threshold, the second prediction submodule is configured to perform any one of the following:

[0278] In a case where the second similarity prediction result is greater than the preset similarity threshold, the second similarity prediction value is determined as a first numerical value.

[0279] In a case where the second similarity prediction result is less than or equal to the preset similarity threshold, the second similarity prediction value is determined as a second numerical value.

[0280] In an embodiment of the present application, when the second training submodule is configured to train the initial image classification model based on the second loss function indicating the difference between the first similarity prediction value and the similarity ground truth, and the third loss function indicating the difference between the second similarity prediction value and the similarity ground truth, the second training submodule is configured to:

[0281] The second training submodule is configured to perform weighted summation on the second loss function and the third loss function based on a first initial weight corresponding to the second loss function and a second initial weight corresponding to the third loss function to obtain a target loss function.

[0282] The second training submodule is configured to train the initial image classification model based on the target loss function until a training completion condition is met to obtain the image classification model.

[0283] The functions and roles of the modules, submodules and units in the above device are realized in the implementation process of the corresponding steps in the above method, which will not be described here.

[0284] For the device embodiment, since it basically corresponds to the method embodiment, the relevant part can be seen from the part of the method embodiment. The device embodiment described above is only illustrative, wherein the modules, sub-modules and units described as separate components can or can not be physically separated, and the components displayed as modules, sub-modules and units can or can not be physical modules, i.e., can be located in one place or distributed on multiple network modules. Part or all of the modules can be selected to achieve the purpose of the scheme of the present specification according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0285] The present application also provides a computing device, referring to Figure 5 , Figure 5 is a structural schematic diagram of a computing device according to an embodiment of the present application. As shown in Figure 5 , the computing device comprises a processor 510, a memory 520 and a network interface 530, the memory 520 is used to store computer program codes executable on the processor 510, the processor 510 is used to implement the image processing method provided by any embodiment of the present application when executing the computer program codes, and the network interface 530 is used to implement input and output functions. In more possible implementation manners, the computing device can also comprise other hardware, which is not limited by the present application.

[0286] The present application also provides a computer readable storage medium, which can be in various forms, such as, in different examples, the computer readable storage medium can be: RAM (Radom Access Memory, Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drive (such as hard disk drive), solid state disk, any type of storage disk (such as optical disk, DVD, etc.), or similar storage medium, or a combination thereof. In particular, the computer readable medium can also be paper or other suitable medium capable of printing programs. The computer readable storage medium stores a computer program, which is executed by the processor to implement the image processing method provided by any embodiment of the present application.

[0287] The present application also provides a computer program product comprising a computer program, which is executed by the processor to implement the image processing method provided by any embodiment of the present application.

[0288] In the present application, the terms "first", "second" are only for descriptive purposes, and cannot be understood as indicating or implying relative importance. The term "multiple" means two or more, unless otherwise explicitly limited.

[0289] Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. It is intended that the application cover any and all variations of the application that come within the scope of the

[0290] It is understood that the application is not limited to the precise construction herein disclosed and shown in the drawings, and that various modifications and changes can be made by those skilled in the art without departing from the scope of the application. The scope of the application is limited only by the claims that follow.

Claims

1. An image processing method, characterized by, The method comprises: acquiring a target image to be processed; determining a first similarity between the target image and first images of each category based on a first feature vector corresponding to the target image and second feature vectors respectively corresponding to the first images, wherein the first images have been labeled with image categories and correspond to a plurality of image categories; determining a target region from the target image and the first images based on an attention matrix of an image classification model; acquiring first color distribution information of the target region in the target image and second color distribution information of the target region in each first image; determining a second similarity between the target image and the first images based on the first color distribution information of the target region in the target image and the second color distribution information of the target region in the first images; determining an image category to which the target image belongs from the image categories of the first images based on the first similarity and the second similarity; wherein the color distribution information is a color distribution spectrum, and the color distribution spectrum is used to indicate RGB colors and positions of each color; the determination of the second similarity between the target image and the first images based on the first color distribution information of the target region in the target image and the second color distribution information of the target region in the first images comprises: determining color distribution similarities of the first color distribution spectrum and each second color distribution spectrum by comparing RGB colors and positions of each color indicated by the first color spectrum and RGB colors and positions of each color indicated by each second color distribution spectrum; determining the second similarity between the target image and the first images based on the color distribution similarities of the first color distribution spectrum and each second color distribution spectrum.

2. The method of claim 1, wherein, the determination of the second similarity between the target image and the first images based on the color distribution similarities of the first color distribution spectrum and each second color distribution spectrum comprises: for at least one first image belonging to any image category, determining a color distribution similarity with the maximum value in the color distribution similarities corresponding to the at least one first image as the second similarity between the target image and the at least one first image.

3. The method of claim 1, wherein, the first similarity is a cosine similarity, and the cosine similarity is used to indicate a cosine distance between the first feature vector corresponding to the target image and the second feature vectors respectively corresponding to the first images of each category; the determination of the first similarity between the target image and the first images of each category based on the first feature vector corresponding to the target image and the second feature vectors respectively corresponding to the first images comprises: determining a first vector sequence for representing the target image and a plurality of second vector sequences for representing the first images based on the target image and the first images through an embedding layer of an image classification model. The first feature vector and the plurality of second feature vectors are obtained by an encoder of the image classification model based on the first vector sequence and the plurality of second vector sequences, and the cosine similarity between the target image and the plurality of first images is determined based on the first feature vector and the plurality of second feature vectors.

4. The method of claim 3, wherein, The first feature vector and the plurality of second feature vectors are obtained by an encoder of the image classification model based on the first vector sequence and the plurality of second vector sequences, and the cosine similarity between the target image and the plurality of first images is determined based on the first feature vector and the plurality of second feature vectors. The first feature vector and the plurality of second feature vectors are obtained by an encoder of the image classification model based on the first vector sequence and the plurality of second vector sequences, and the cosine similarity between the target image and the plurality of first images is determined based on the first feature vector and the plurality of second feature vectors. The first feature vector and the plurality of second feature vectors are obtained by an encoder of the image classification model based on the first vector sequence and the plurality of second vector sequences, and the cosine similarity between the target image and the plurality of first images is determined based on the first feature vector and the plurality of second feature vectors. The first feature vector and the plurality of second feature vectors are obtained by an encoder of the image classification model based on the first vector sequence and the plurality of second vector sequences, and the cosine similarity between the target image and the plurality of first images is determined based on the first feature vector and the plurality of second feature vectors.

5. The method of claim 4, wherein, The first feature vector and the plurality of second feature vectors are obtained by an encoder of the image classification model based on the first vector sequence and the plurality of second vector sequences, and the cosine similarity between the target image and the plurality of first images is determined based on the first feature vector and the plurality of second feature vectors. The first feature vector and the plurality of second feature vectors are obtained by an encoder of the image classification model based on the first vector sequence and the plurality of second vector sequences, and the cosine similarity between the target image and the plurality of first images is determined based on the first feature vector and the plurality of second feature vectors. The first feature vector and the plurality of second feature vectors are obtained by an encoder of the image classification model based on the first vector sequence and the plurality of second vector sequences, and the cosine similarity between the target image and the plurality of first images is determined based on the first feature vector and the plurality of second feature vectors.

6. The method of claim 1, wherein, The first feature vector and the plurality of second feature vectors are obtained by an encoder of the image classification model based on the first vector sequence and the plurality of second vector sequences, and the cosine similarity between the target image and the plurality of first images is determined based on the first feature vector and the plurality of second feature vectors. The first feature vector and the plurality of second feature vectors are obtained by an encoder of the image classification model based on the first vector sequence and the plurality of second vector sequences, and the cosine similarity between the target image and the plurality of first images is determined based on the first feature vector and the plurality of second feature vectors. The image classification model is obtained by pre-training; 7. The method of claim 1, wherein, The training process of the image classification model includes: a plurality of first sample images are obtained, which have been labeled with sample image categories; the plurality of first sample images are input into an initial visual converter model, and the predicted image categories of the plurality of first sample images are determined by the initial visual converter model; the initial visual converter model is trained based on a first loss function indicating the difference between the predicted image categories of the plurality of first sample images and the sample image categories of the plurality of first sample images until a preset training completion condition is met, and a trained visual converter model is obtained; the image classification model is obtained based on the trained visual converter model. The trained visual converter model includes an embedding layer, a converter encoder, and a multi-head perceiver; 8. The method of claim 7, wherein, ​ The method comprises the following steps: obtaining an embedding layer and a transformer encoder from the trained visual converter model to form an initial image classification model; obtaining a plurality of second sample images labeled with similarity true values; inputting any two second sample images in the plurality of second sample images into the initial image classification model, and outputting first similarity prediction values of the two second sample images through the initial image classification model; training the initial image classification model based on a second loss function indicating the difference between the first similarity prediction values and the similarity true values.

9. The method of claim 8, wherein, The method further comprises the following steps: determining a second similarity prediction result between the color distribution information of the target regions in the two second sample images based on the attention matrix of the initial image classification model; determining a second similarity prediction value based on the second similarity prediction result and a preset similarity threshold value; training the initial image classification model based on a second loss function indicating the difference between the first similarity prediction values and the similarity true values, and a third loss function indicating the difference between the second similarity prediction value and the similarity true value.

10. The method of claim 9, wherein, The method further comprises the following steps: determining a second similarity prediction value based on the second similarity prediction result and a preset similarity threshold value, comprising any one of the following: in a case where the second similarity prediction result is greater than the preset similarity threshold value, determining the second similarity prediction value as a first numerical value; 11. The method of claim 9, wherein, in a case where the second similarity prediction result is less than or equal to the preset similarity threshold value, determining the second similarity prediction value as a second numerical value. The method further comprises the following steps: training the initial image classification model based on a second loss function indicating the difference between the first similarity prediction values and the similarity true values, and a third loss function indicating the difference between the second similarity prediction value and the similarity true value, comprising:

12. An image processing apparatus characterized by comprising: weighting and summing the second loss function and the third loss function based on a first initial weight corresponding to the second loss function and a second initial weight corresponding to the third loss function to obtain a target loss function; training the initial image classification model based on the target loss function until a training completion condition is met to obtain the image classification model. The device comprises: an image acquisition module configured to acquire a target image to be processed; a similarity determination module configured to determine first similarities between the target image and first images of each category based on the target image and a plurality of first images, wherein the plurality of first images are labeled with image categories, and the plurality of first images correspond to a plurality of image categories; a region determination module configured to determine target regions from the target image and the plurality of first images based on an attention matrix of an image classification model; an information acquisition module configured to acquire first color distribution information of the target region in the target image and second color distribution information of the target region in each first image; The similarity determination module is further configured to determine a second similarity between the target image and the plurality of first images based on the first color distribution information of the target region in the target image and second color distribution information of the target region in the plurality of first images. The category determination module is configured to determine, based on the first similarity and the second similarity, an image category to which the target image belongs from among image categories of the plurality of first images. The color distribution information is a color distribution spectrum, and the color distribution spectrum is used to indicate RGB colors and positions of each color. When determining the second similarity between the target image and the plurality of first images based on the first color distribution information of the target region in the target image and the second color distribution information of the target region in the plurality of first images, the similarity determination module is configured to: determine a color distribution similarity between the first color distribution spectrum and each second color distribution spectrum by comparing the RGB colors and the positions of each color indicated by the first color spectrum and the RGB colors and the positions of each color indicated by each second color distribution spectrum; and determine the second similarity between the target image and the plurality of first images based on the color distribution similarity between the first color distribution spectrum and each second color distribution spectrum.

13. A computing device, comprising: The computing device includes a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements operations performed by the image processing method according to any one of claims 1 to 11 when executing the computer program.

14. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a program, and the program is executed by the processor to implement operations performed by the image processing method according to any one of claims 1 to 11.

Images