Model training method, image classification method, device, equipment and storage medium

Abstract

The application relates to a model training method and device, an image classification method and device, equipment and a storage medium, and relates to the technical field of image processing. The method comprises the following steps: obtaining training samples; performing first random transformation on sample images in the training samples to obtain a first set of transformed images; performing second random transformation on first sample images and the same category images corresponding to second sample images in the training samples to obtain a second set of transformed images; inputting the transformed images in the first set of transformed images and the second set of transformed images into a feature extraction model respectively to obtain a first feature vector group and a second feature vector group; updating model parameters of the feature extraction model based on the first feature vector group and the second feature vector group; and repeating the above process to iteratively train the feature extraction model to obtain a trained feature extraction model. Through the above method, the accuracy of feature extraction by the feature extraction model and the accuracy of image classification can be improved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of image processing, in particular to a model training method, an image classification method, an apparatus, a device and a storage medium. BACKGROUND

[0002] The task of image classification requires identifying the category to which a given image belongs.

[0003] In the related art, image classification is usually based on a trained neural network, that is, a to-be-classified image is input into the trained neural network, and the probability of the to-be-classified image belonging to each category output by the neural network is obtained. The one with the highest probability value is determined as the classification result of the to-be-classified image.

[0004] However, the category to which the above-mentioned classified image belongs is usually a predefined category set, that is, a large number of sample images with known categories are needed to train the neural network during the training of the neural network, so that a large amount of manual annotation is required in the process of the neural network. If the category of the to-be-classified image is outside the predefined category set during the training of the neural network, the to-be-classified image cannot be accurately classified, thereby reducing the accuracy of image classification. SUMMARY

[0005] Embodiments of the present application provide a model training method, an image classification method, an apparatus, a device and a storage medium, which can improve the accuracy of feature extraction by a feature extraction model, and further improve the accuracy of image classification. The technical solution is as follows:

[0006] On the one hand, a model training method is provided, and the method comprises:

[0007] obtaining training samples, wherein the training samples contain n first sample images and m second sample images; the second sample images have image category labels; m and n are positive integers;

[0008] performing first random transformation on the m+n sample images in the training samples to obtain a first set of transformed images;

[0009] performing second random transformation on the n first sample images and m second sample images in the training samples to obtain a second set of transformed images; the second sample images correspond to the same category images, that is, sample images with the same image category labels as the second sample images;

[0010] inputting a first transformed image in the first set of transformed images and a second transformed image in the second set of transformed images into a feature extraction model to be trained respectively, to obtain a first feature vector group and a second feature vector group output by the feature extraction model to be trained respectively; the first feature vector group contains feature vectors of the first transformed images respectively, and the second feature vector group contains feature vectors of the second transformed images respectively;

[0011] updating model parameters of the feature extraction model to be trained based on the first feature vector group and the second feature vector group;

[0012] repeating the above process to iteratively train the feature extraction model to be trained, to obtain the trained feature extraction model.

[0013] In another aspect, an image classification method is provided, the method comprising:

[0014] obtaining an image to be classified;

[0015] inputting the image to be classified into a feature extraction model to obtain a first feature vector of the image to be classified output by the feature extraction model; the feature extraction model is trained according to the model training method of any one of claims 1 to 4;

[0016] performing similarity matching between the first feature vector of the image to be classified and each second feature vector corresponding to each image category label in a feature library respectively, to obtain a similarity score between the first feature vector and each second feature vector corresponding to each image category label; the second feature vector is a feature vector of a sample image group corresponding to the image category label at any clustering center;

[0017] calculating an average similarity score between the first feature vector and each image category label;

[0018] determining an image category of the image to be classified according to an image category indicated by a target image category label; the target image category label is one of the image category labels corresponding to the highest average similarity score.

[0019] In another aspect, a model training device is provided, the device comprising:

[0020] a sample obtaining module configured to obtain training samples, the training samples containing n first sample images and m second sample images; the second sample images have image category labels; m and n are positive integers;

[0021] a first transformation module, configured to perform first random transformation on m+n sample images in the training samples respectively to obtain a first transformed image set;

[0022] a second transformation module, configured to perform second random transformation on n first sample images and m second sample images in the training samples respectively to obtain a second transformed image set; the second sample images correspond to images of the same image category label as the second sample images;

[0023] a feature extraction module, configured to input first transformed images in the first transformed image set and second transformed images in the second transformed image set into a feature extraction model to be trained respectively to obtain first feature vector groups and second feature vector groups output by the feature extraction model to be trained respectively; the first feature vector groups contain feature vectors of the first transformed images respectively, and the second feature vector groups contain feature vectors of the second transformed images respectively;

[0024] a parameter updating module, configured to update model parameters of the feature extraction model based on the first feature vector groups and the second feature vector groups;

[0025] an iterative training module, configured to repeat the above process to perform iterative training on the feature extraction model to obtain a trained feature extraction model.

[0026] In a possible implementation, the parameter updating module comprises:

[0027] a function value calculation submodule, configured to calculate a function value of a contrast loss function based on the first feature vector groups and the second feature vector groups;

[0028] a parameter updating submodule, configured to update model parameters of the feature extraction model to be trained based on the function value of the contrast loss function;

[0029] wherein the function value of the contrast loss function is a ratio between a first similarity sum and a second similarity sum; the first similarity refers to a similarity between feature vectors of the first transformed image and the second transformed image corresponding to the same sample image; the second similarity refers to a similarity between feature vectors of the first transformed image and the second transformed image corresponding to different sample images respectively; the second transformed image corresponding to the second sample image is an image obtained by performing second random transformation on the same category image corresponding to the second sample image.

[0030] In a possible implementation, the apparatus further comprises:

[0031] An image set obtaining module is configured to obtain a target sample image set, the target sample image set including at least two third sample images; the third sample images have image category labels;

[0032] A vector obtaining module is configured to input the at least two third sample images in the target sample image set into the trained feature extraction model respectively, and obtain feature vectors of the respective third sample images;

[0033] A feature library constructing module is configured to construct a feature library based on the feature vectors of the respective third sample images and the image category labels respectively possessed by the respective third sample images.

[0034] In a possible implementation, the feature library constructing module includes:

[0035] A grouping submodule is configured to group the at least two third sample images based on the image category labels, and obtain T sample image groups, T being a positive integer; each sample image group corresponds to a same image category label;

[0036] A clustering submodule is configured to cluster the feature vectors of the third sample images in each sample image group respectively, and obtain K clustering centers respectively corresponding to each sample image group; K is a positive integer;

[0037] A feature library constructing submodule is configured to construct the feature library based on a correspondence between the feature vectors at the K clustering centers respectively corresponding to each sample image group and the image category labels; the feature library includes each image category label and K feature vectors respectively corresponding to the image category label.

[0038] On the other hand, an image classification device is provided, and the device includes:

[0039] An image obtaining module is configured to obtain an image to be classified;

[0040] A vector obtaining module is configured to input the image to be classified into a feature extraction model, and obtain a first feature vector of the image to be classified output by the feature extraction model; the feature extraction model is trained according to the model training method provided in the above embodiment;

[0041] A similarity matching module is configured to perform similarity matching between the first feature vector of the image to be classified and each second feature vector corresponding to each image category label in a feature library respectively, and obtain similarity scores of the first feature vector and each second feature vector corresponding to each image category label; the second feature vector is a feature vector at any clustering center of a sample image group corresponding to the image category label;

[0042] a score calculation module configured to calculate an average similarity score between the first feature vector and each of the image class labels;

[0043] a class determination module configured to determine an image class of the image to be classified according to an image class indicated by a target image class label, the target image class label being one of the image class labels with the highest average similarity score.

[0044] In a possible implementation, the class determination module is configured to, in a case where the average similarity score corresponding to the target image class label is higher than a score threshold, determine an image class indicated by the image class label corresponding to the target sample image group as the image class of the image to be classified.

[0045] In a possible implementation, the class determination module is configured to, in a case where the average similarity score corresponding to the target image class label is not higher than the score threshold, determine the image class of the image to be classified as another class.

[0046] In another aspect, a computer device is provided, which includes a processor and a memory, the memory storing at least one computer program, the at least one computer program being loaded and executed by the processor to implement the model training method and / or the image classification method described above.

[0047] In another aspect, a computer readable storage medium is provided, which stores at least one computer program, the computer program being loaded and executed by a processor to implement the model training method and / or the image classification method described above.

[0048] In another aspect, a computer program product is provided, which includes at least one computer program, the computer program being loaded and executed by a processor to implement the model training method and / or the image classification method provided in various optional implementations described above.

[0049] The technical solutions provided in the present application can have the following beneficial effects:

[0050] The model training method provided in the embodiments of the present application uses training samples containing both labeled sample images and unlabeled sample images, performs first random transformation on the sample images in the training samples to obtain a first set of transformed images, performs second random transformation on the unlabeled sample images in the training samples, obtains images of the same category based on the image classification labels of the labeled sample images, and performs second random transformation on the images of the same category to obtain a second set of transformed images; the feature extraction model to be trained is used to extract feature vectors of each transformed image in the first set of transformed images and each transformed image in the second set of transformed images, respectively obtaining a first feature vector group and a second feature vector group, and the feature extraction model is trained based on the first feature vector group and the second feature vector group, the above process is repeated, and the feature extraction model is iteratively trained to obtain a trained feature extraction model; through the method of semi-supervised contrast learning, attention can be paid to the similarity in image vision and the similarity in image semantic information in the training process of the feature extraction model, thereby improving the accuracy of the feature vectors extracted by the feature extraction model, and further improving the accuracy of image classification; meanwhile, only part of the sample images need to be set with image classification labels in the training process, which reduces the cost of manual labeling and makes the feature vectors extracted by the feature extraction model obtained through training more suitable for classification scenarios where the image to be classified is not in the pre-defined category set when the model is trained.

[0051] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF DRAWINGS

[0052] The accompanying drawings incorporated in and forming a part of the specification illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the application.

[0053] Figure 1 A flowchart of a model training method according to an exemplary embodiment of the present application is shown;

[0054] Figure 2 A flowchart of an image classification method according to an exemplary embodiment of the present application is shown;

[0055] Figure 3 A framework diagram of model training and image classification according to an exemplary embodiment is shown;

[0056] Figure 4 A flowchart of an image classification method according to an exemplary embodiment of the present application is shown;

[0057] Figure 5 A structure diagram of AlexNet according to an exemplary embodiment of the present application is shown;

[0058] Figure 6 A block diagram of a model training apparatus is shown according to an example embodiment of the present application;

[0059] Figure 7 A block diagram of an image classification apparatus is shown according to an example embodiment of the present application;

[0060] Figure 8 is a structural block diagram of a computer device according to an example embodiment;

[0061] Figure 9 is a structural block diagram of a computer device according to an example embodiment. DETAILED DESCRIPTION

[0062] The example embodiments will be described in detail herein with reference to the accompanying drawings. When the following description refers to accompanying drawings, unless otherwise specified, the same numbers in different drawings represent the same or similar elements. The implementations described in the following example embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of apparatuses and methods consistent with some aspects of the present application as detailed in the appended claims.

[0063] Figure 1 A flowchart of a model training method is shown according to an example embodiment of the present application, which can be performed by a computer device, which can be implemented as a terminal or a server, as shown in Figure 1 The model training method can include the following steps:

[0064] At step 110, a training sample is obtained, which contains n first sample images and m second sample images; the second sample images have image class labels; m and n are positive integers.

[0065] That is, in the training sample in the embodiments of the present application, both labeled sample images and unlabeled sample images are included.

[0066] The number of labeled sample images can be the same as or different from the number of unlabeled sample images; optionally, the number of labeled sample images is much smaller than the number of unlabeled sample images, i.e., m << n, the values of m and n can be set by relevant personnel, or the values of m and n can also be randomly determined in the process of obtaining the training sample.

[0067] At step 120, first random transformation is performed on the m+n sample images in the training sample to obtain a first set of transformed images.

[0068] The first random transformation can include at least one of rotation, cropping, adding noise, etc.

[0069] In step 130, the same class images corresponding to the n first sample images and the m second sample images in the training samples are respectively subjected to second random transformation to obtain a second transformed image set; the same class image corresponding to the second sample image is a sample image having the same image class label as the second sample image.

[0070] The second random transformation can include at least one of rotation, cropping, adding noise, etc.

[0071] In the embodiments of the present application, the second random transformation performed on the first sample image can be different from the first random transformation; the second random transformation performed on the same class image corresponding to the second sample image can be the same as or different from the first random transformation performed on the second sample image.

[0072] The same class image corresponding to the second sample image can be a sample image obtained from the second sample image set based on the image type label of the second sample image, and the number of second sample images is the same as that of the same class images; by performing second random transformation on the same class images having the same image class label as the second sample image, the number of sample images under the image class label can be increased to increase the image features under the image class label.

[0073] In step 140, the first transformed images in the first transformed image set and the second transformed images in the second transformed image set are respectively input into the feature extraction model to be trained to obtain the first feature vector group and the second feature vector group output by the feature extraction model to be trained; the first feature vector group contains the feature vectors of each first transformed image, and the second feature vector group contains the feature vectors of each second transformed image.

[0074] The feature vectors of each first transformed image in the first transformed image set and the feature vectors of each second transformed image in the second transformed image set are obtained through the feature extraction model.

[0075] In step 150, the model parameters of the feature extraction model to be trained are updated based on the first feature vector group and the second feature vector group.

[0076] In the embodiment of the present application, according to the same image, even if different random transformations are performed, the semantic information contained in the image does not change, and the semantic information contained in two images of the same category is similar or the same after random transformation. The corresponding relationship between the first feature vector group corresponding to the first transformed image set and the second feature vector group corresponding to the second transformed image set is used to train the feature extraction model to be trained, so that the feature extraction model obtained by training can not only judge whether the images are visually similar, but also judge whether the semantic information of the images is consistent within a certain range.

[0077] In step 160, the above process is repeated to iteratively train the feature extraction model to be trained to obtain a trained feature extraction model.

[0078] The process of steps 110 to 150 is repeated, and the model parameters of the feature extraction model are updated based on the first feature vector group and the second feature vector group corresponding to each training sample to train the feature extraction model and obtain a trained feature extraction model.

[0079] As described above, the model training method provided in the embodiments of the present application uses training samples containing both labeled sample images and unlabeled sample images, performs first random transformation on the sample images in the training samples to obtain a first transformed image set, performs second random transformation on the unlabeled sample images in the training samples, obtains images of the same category based on the image classification labels of the labeled sample images, and performs second random transformation on the images of the same category to obtain a second transformed image set. The feature extraction model to be trained is used to extract feature vectors of each transformed image in the first transformed image set and each transformed image in the second transformed image set, and first feature vector groups and second feature vector groups are obtained, respectively. The feature extraction model is trained based on the first feature vector groups and the second feature vector groups, and the above process is repeated to iteratively train the feature extraction model to obtain a trained feature extraction model. Through the method of semi-supervised contrast learning, the feature extraction model can focus on both the visual similarity of images and the semantic information similarity of images during the training process, thereby improving the accuracy of the feature vectors extracted by the feature extraction model and the accuracy of image classification. At the same time, only part of the sample images need to be labeled with image classification labels in the training process, which reduces the cost of manual labeling and makes the feature vectors extracted by the trained feature extraction model more suitable for classification scenarios where the images to be classified are not in the pre-defined category set during model training.

[0080] Based on the feature extraction model obtained by training the model training method shown in Figure 1 The feature extraction model obtained by training the model training method can be applied to the image classification scenario. Figure 2A flowchart of an image classification method provided by an example embodiment of the present application is shown, which can be executed by a computer device, which can be implemented as a terminal or a server, as shown in the figure. The image classification method can include the following steps: Figure 2

[0081] Step 210, obtaining an image to be classified.

[0082] Step 220, inputting the image to be classified into a feature extraction model to obtain a first feature vector of the image to be classified output by the feature extraction model.

[0083] The feature extraction model is obtained by a model training method provided by an example embodiment as shown. Figure 1

[0084] Step 230, performing similarity matching between the first feature vector of the image to be classified and each second feature vector corresponding to each image category label in the feature library respectively to obtain a similarity score between the first feature vector and each second feature vector corresponding to each image category label; the second feature vector is a feature vector of a sample image corresponding to the image category label grouped at any clustering center.

[0085] The feature library is constructed based on the trained feature extraction model and the feature vectors of the images with known image classification. Each image classification label can correspond to multiple clustering centers, and therefore, each image classification label can correspond to multiple second feature vectors.

[0086] Step 240, calculating an average similarity score between the first feature vector and each image category label.

[0087] The average similarity score between the first feature vector and the image category label is an average value of the similarity scores of the first vector and each second feature vector corresponding to the image category label.

[0088] Step 250, determining an image category of the image to be classified according to an image category indicated by a target image category label; the target image category label is one with the highest average similarity score among the image category labels.

[0089] ​​In summary, the image classification method provided by the embodiments of the present application uses the feature extraction model trained by the semi-supervised contrast learning method to perform feature extraction on the image to be classified, obtains the first feature vector of the image to be classified, performs similarity matching on the first feature vector and each second feature vector in the feature library having an image classification label, and determines the image category corresponding to the second feature vector with the highest similarity score as the image category of the image to be classified. Since the accuracy of the feature extraction model in extracting the feature vector is improved, the accuracy of image classification can be improved based on the feature vector of the image to be classified extracted by the feature extraction model.

[0090] The scheme related to the present application includes a model training phase and an image classification phase, Figure 3 is a framework diagram of model training and image classification according to an exemplary embodiment, as Figure 3 In the model training phase, the model training device 310 performs semi-supervised contrast learning training on the feature extraction model to be trained by using the pre-set extracted training samples (including n first sample images and m second sample images, and the second sample images have image category labels), to obtain the feature extraction model after training. In the image classification phase, the image classification device 320 performs feature extraction on the input image to be classified based on the feature extraction model, obtains the first feature vector of the image to be classified, and determines the image category of the image to be classified based on the similarity matching result between the first feature vector and each second feature vector in the pre-constructed feature library having an image category label.

[0091] The model training device 310 and the image classification device 320 can be a computer device, for example, the computer device can be a terminal or a server.

[0092] Optionally, the model training device 310 and the image classification device 320 can be the same device, or the model training device 310 and the image classification device 320 can be different devices. Moreover, when the model training device 310 and the image classification device 320 are different devices, the model training device 310 and the image classification device 320 can be the same type of device, for example, the model training device 310 and the image classification device 320 can both be servers; or the model training device 310 and the image classification device 320 can be different types of devices, for example, the image classification device 320 can be a terminal, and the model training device 310 can be a server, etc. The embodiments of the present application do not limit the specific types of the model training device 310 and the image classification device 320.

[0093] Figure 4A flowchart of an image classification method provided by an example embodiment of the present application is shown, which can be executed by a computer device, which can be implemented as a terminal or a server. The image classification method can include a model training phase, a feature library construction phase, and an image classification phase, as shown in Figure 4 Each phase can include the following steps, as shown.

[0094] Model training phase:

[0095] Step 401, a training sample is obtained, which contains n first sample images and m second sample images; the second sample images have image category labels; m and n are positive integers.

[0096] Optionally, the process of obtaining the training sample can be implemented as:

[0097] A first sample image set and a second sample image set are obtained; the number of first sample images contained in the first sample image set is much larger than the number of second sample images contained in the second sample image set.

[0098] The first sample image set and the second sample image set are combined to obtain a merged sample image set.

[0099] N sample images are randomly extracted from the merged sample image set to form the training sample, where N is a positive integer.

[0100] The N sample images forming the training sample can include n first sample images and m second sample images, i.e., m+n=N.

[0101] Illustratively, given a first sample image set U={u1, u2, u3,...,uK} and a second sample image set L={(l1, c1), (l2, c2), (l3, c3),..., (lP, cP)}, where li is an image with an image category label, ci is the corresponding image category label; K is the number of first sample images in the first sample image set, and P is the number of second sample images in the second sample image set, in the embodiments of the present application, P<<K. If there are T image categories to be recognized, the set of image category labels can be represented as C={c1, c2, c3,...,cT}, then ci∈T. K P P i i T i

[0102] The first sample image set and the second sample image set are combined, and N sample images are randomly selected therefrom to form a training sample, denoted as I={I1, I2, I3,...,IN}.​​​​​​​N}, wherein I i may be represented as a three-dimensional tensor with dimension HxWxC, H, W and C are the length, width and channel number of the sample image respectively.

[0103] Step 402, the first random transformation is performed on each of the m+n sample images in the training sample to obtain a first transformed image set.

[0104] The first random transformation is performed on each of the sample images in the training sample, such as rotation, cropping and adding noise, etc., to obtain a first transformed image set, which can be represented as: S={s1, s2, s3,..., s N}, wherein s i is three-dimensional data with dimension HxWxC.

[0105] Step 403, the second random transformation is performed on the n first sample images and the m second sample images corresponding to the same class images in the training sample, respectively, to obtain a second transformed image set; the same class image corresponding to the second sample image is an image with the same image class label as the second sample image.

[0106] For each selected sample image I j (j=1, 2, 3...N), first determine whether the sample image has an image class label; if the sample image has no image class label, the second random transformation is performed on the sample image I j to generate t i ; if the sample image has an image class label and the image class label is c k , another image with the same image class label c k (i.e. the same class image corresponding to the second sample image) can be selected from the second sample image set, and the second random transformation is performed on the same class image corresponding to the second sample image to generate t i ; after the traversal of all sample images in the training sample is completed, the second transformed image set T={t1, t2, t3,..., t N} can be constructed based on the second random transformation results of each sample image. Wherein, t i and s i are different transformed results of the same sample image, or the transformed results of two sample images with the same label.

[0107] Step 404: Input the first transformed image in the first transformed image set and the second transformed image in the second transformed image set into the feature extraction model to be trained, respectively, to obtain the first feature vector group and the second feature vector group output by the feature extraction model to be trained; the first feature vector group contains the feature vectors of each first transformed image, and the second feature vector group contains the feature vectors of each second transformed image.

[0108] In this embodiment, the feature extraction model can be a machine model built on a deep neural network; illustratively, the deep learning network can be implemented as AlexNet, ResNet (Deep residual network), MobileNet, Swing Transformer, etc.; using AI e xN e For example, t Figure 5 A structural diagram of AlexNet as illustrated in an exemplary embodiment of this application is shown, as follows: Figure 5 As shown, AlexNet can include several convolutional layers (Conv) and several fully connected layers (FC). After passing through different modules, the dimension of the image features may change. If the dimension of the input image to AlexNet is 227×227×3, after processing by AlexNet, the output is a feature vector with a dimension of d.

[0109] In the embodiments of this application, the processing of a deep neural network can be represented by the operation function f(z, θ).

[0110] By inputting the first transformed image set S into the feature extraction model to be trained, the first feature vector set output by the feature extraction model can be obtained. This first feature vector set can be represented as X = {x1, x2, x3, ... x}. N}, where x i =f(s) i Correspondingly, by inputting the second transformed image set T into the feature extraction model to be trained, the second feature vector set output by the feature extraction model can be obtained. This second feature vector set can be represented as Y = {y1, y2, y3, ... y}. N}, where y i = f (t i ,θ).

[0111] Step 405: Update the model parameters of the feature extraction model to be trained based on the first feature vector group and the second feature vector group.

[0112] In the embodiments of the present application, the process of updating the model parameters of the feature extraction model to be trained based on the first feature vector group and the second feature vector group can be implemented as follows:

[0113] calculating a function value of a contrast loss function based on the first feature vector group and the second feature vector group;

[0114] updating the model parameters of the feature extraction model to be trained based on the function value of the contrast loss function;

[0115] The function value of the contrast loss function is the ratio between the sum of the first similarities and the sum of the second similarities. The first similarity refers to the similarity between the feature vectors of the first transformed image and the second transformed image corresponding to the same sample image. The second similarity refers to the similarity between the feature vectors of the first transformed image and the second transformed image corresponding to different sample images.

[0116] The first transformed image corresponding to the first sample image is an image obtained by performing a first random transformation on the first sample image. The second transformed image corresponding to the first sample image is an image obtained by performing a second random transformation on the first sample image. The first transformed image corresponding to the second sample image is an image obtained by performing a first random transformation on the second sample image. The second transformed image corresponding to the second sample image is an image obtained by performing a second random transformation on the same category image corresponding to the second sample image.

[0117] Illustratively, after obtaining the feature vectors of the first transformed image and the second transformed image corresponding to each sample image, the computer device can calculate the similarity between the two feature vectors of the same sample image and the similarity between the two feature vectors of different sample images. For example, if N = 8, taking one of the sample images 1 as an example, the first similarity of the sample image 1 is the similarity between the feature vectors of the first transformed image of the sample image 1 and the feature vectors of the second transformed image of the sample image 1. The second similarity of the sample image 1 is the similarity between the feature vectors of the first transformed image of the sample image 1 and the feature vectors of the second transformed image of the sample images 2-8, respectively. The first similarity of the sample image 2 is the similarity between the feature vectors of the first transformed image of the sample image 2 and the feature vectors of the second transformed image of the sample image 2. The second similarity of the sample image 2 is the similarity between the feature vectors of the first transformed image of the sample image 2 and the feature vectors of the second transformed image of the sample images 1, 3-8, respectively.

[0118] Illustratively, the calculation formula of the contrast loss function can be expressed as:

[0119]

[0120] wherein <.,.> represents vector inner product, and is used to represent the similarity between the feature vectors, and is a summation symbol, x i represents the feature vector of the first transformed image of the i-th sample image, y i represents the feature vector of the second transformed image of the i-th sample image, y j represents the feature vector of the second transformed image of each sample image other than the i-th sample image, i≠j.

[0121] The smaller the function value of the contrast loss function is, the smaller the difference between the feature vectors of the two transformed images of the same sample image is, the higher the similarity is, and the larger the difference between the feature vectors of the transformed images of different sample images is. Therefore, in the model training process, the function value of the contrast loss function is made as small as possible.

[0122] Optionally, after obtaining the function value of the contrast loss function, the gradient of the contrast loss function with respect to the model parameter θ (i.e. the network parameter of the deep neural network) is calculated, i.e. and the model parameter θ in the feature extraction model is updated by using the following formula:

[0123]

[0124] wherein λ represents the learning rate, and the value can be set by relevant personnel, for example, the value is 0.1 or 0.01 or 0.001, etc.

[0125] Step 406: The above process is repeated to iteratively train the feature extraction model to be trained, and a trained feature extraction model is obtained.

[0126] The process of steps 401 to 405 is repeated, and the model parameter of the feature extraction model is iteratively updated based on the function value of the loss function obtained each time, until the function value of the loss function converges, or until the model parameter converges, to determine that the feature extraction model has been trained, and a trained feature extraction model is obtained.

[0127] Feature library construction phase:

[0128] Step 407: A target sample image set is obtained, and the target sample image set contains at least two third sample images; the third sample images have image category labels.

[0129] The target sample image set contains sample images of all currently known image categories. The number of third sample images contained in the target sample image set is much larger than the number of second sample images contained in the second sample image set.

[0130] Step 408, input at least two third sample images in the target sample image set into the trained feature extraction model respectively, and obtain a feature vector of each third sample image.

[0131] Illustratively, the set of feature vectors of each third sample image can be represented as M={m1, m2, m3,..., m p}。

[0132] Step 409, based on the feature vector of each third sample image and the image category label respectively possessed by each third sample image, construct a feature library.

[0133] In the embodiment of the present application, the third sample images can be grouped according to the image category label, and the standard feature vector corresponding to the image category can be determined according to the feature vectors of the third sample images in each group. Optionally, the process can be implemented as follows:

[0134] Group at least two third sample images based on the image category label, and obtain T sample image groups, T being a positive integer; each sample image group corresponds to the same image category label;

[0135] Cluster the feature vectors of the third sample images in each sample image group respectively, and obtain K clustering centers corresponding to each sample image group respectively; K being a positive integer;

[0136] Based on the corresponding relationship between the feature vectors at the K clustering centers corresponding to each sample image group and the image category label, construct a feature library, which includes each image category label and the K feature vectors corresponding thereto.

[0137] Illustratively, the feature vectors at the K clustering centers corresponding to each sample image group can be represented as Based on the different application scenarios, the value of K can be set differently. Illustratively, the value of K can be 1 / 16 / 32 / 64 / 128, etc.

[0138] That is, the feature vectors in each sample image group are clustered, and the feature vectors corresponding to the clustering centers in each sample image group are determined as the standard feature vectors of the sample image group; then, based on the corresponding relationship between the sample image group and the image category label, a feature library can be constructed, and each image category label in the feature library has a corresponding standard feature vector (i.e. K feature vectors).

[0139] Optionally, when clustering the feature vectors in each sample image group, K-means clustering can be used. ea n sThe clustering method is used for clustering; or, other clustering methods can also be used for clustering the feature vectors, and the application does not limit this.

[0140] The image classification stage:

[0141] Step 410: An image to be classified is obtained.

[0142] Step 411: The image to be classified is input into a feature extraction model to obtain a first feature vector of the image to be classified output by the feature extraction model.

[0143] The feature extraction model is a feature extraction model trained based on steps 401 to 406.

[0144] The first feature vector e of the image to be classified can be obtained by inputting the image to be classified I into the trained feature extraction model.

[0145] Step 412: The first feature vector of the image to be classified is respectively matched with each second feature vector corresponding to each image category label in the feature library to obtain a similarity score of the first feature vector and each second feature vector corresponding to each image category label; the second feature vector is a feature vector of a sample image corresponding to the image category label grouped at any clustering center.

[0146] That is, each second feature vector corresponding to the image category label is a standard feature vector corresponding to the image category label.

[0147] The feature library is a feature library constructed based on steps 407 to 409, and the feature library can include T image categories, denoted as C={c1, c2, c3,..., c T}.

[0148] Optionally, the similarity score between the feature vectors can be calculated by a cosine similarity calculation method, or can be calculated by other similarity calculation methods, and the application does not limit this.

[0149] Step 413: Calculate the average similarity score between the first feature vector and each image category label.

[0150] Taking the cosine similarity calculation method as an example, the average similarity between the first feature vector e and each image category label can be calculated by the following formula:

[0151]

[0152] Where sim(e, C i ) represents the average similarity score between the first feature vector e and the i-th image category label, indicates the average similarity between the first feature vector e and the second feature vector The cosine similarity is calculated. The average similarity between the first feature vector e and the image category label is the average of the similarities between the first feature vector e and each second feature vector corresponding to the image category label.

[0153] At step 414, the image category of the image to be classified is determined according to the image category indicated by the target image category label. The target image category label is the one with the highest average similarity score among the image category labels.

[0154] After obtaining the average similarity scores of the T image category labels with respect to the first feature vector, the image category indicated by the image category label with the highest similarity score can be determined as the image category of the image to be classified.

[0155] In a possible case, the image to be classified does not belong to any of the image categories corresponding to the T image category labels, i.e., the average similarity scores of the image category labels are not high, and in this case, the image to be classified needs to be classified again. Therefore, the method further includes:

[0156] In the case where the average similarity score of the target image category label is higher than the score threshold, the image category indicated by the target image category label is determined as the image category of the image to be classified.

[0157] In the case where the average similarity score of the target image category label is not higher than the score threshold, the image category of the image to be classified is determined as another category.

[0158] The score threshold can be set based on actual conditions, and the value of the score threshold is not limited in the present application. By setting the score threshold, the misclassification of the image to be classified can be avoided, and the accuracy of image classification can be further improved.

[0159] In summary, the image classification method provided by the embodiments of the present application uses the feature extraction model trained by the semi-supervised contrast learning method to extract the features of the image to be classified, obtains the first feature vector of the image to be classified, performs similarity matching between the first feature vector and each second feature vector corresponding to each image classification label in the feature library, calculates the average similarity corresponding to each image classification label, and determines the image classification corresponding to the image classification label with the highest average similarity score as the image category of the image to be classified. Since the accuracy of the feature extraction model in extracting the feature vector is improved, the accuracy of image classification based on the feature vector of the image to be classified extracted by the feature extraction model can be improved.

[0160] Figure 6A block diagram of a model training apparatus provided by an example embodiment of the present application is shown, which can be used to perform all or part of the steps in the embodiments shown in Figure 1 or Figure 4 as shown, the model training apparatus comprises: Figure 6

[0161] A sample obtaining module 610 is configured to obtain training samples, wherein the training samples contain n first sample images and m second sample images; each of the second sample images has an image category label; m and n are positive integers.

[0162] A first transformation module 620 is configured to perform first random transformation on the m+n sample images in the training samples respectively to obtain a first transformed image set.

[0163] A second transformation module 630 is configured to perform second random transformation on the n first sample images and the m second sample images respectively, to obtain a second transformed image set; the second sample images correspond to images of the same category as the second sample images, i.e., images having the same image category label as the second sample images.

[0164] A feature extraction module 640 is configured to input the first transformed images in the first transformed image set and the second transformed images in the second transformed image set into a feature extraction model to be trained respectively, to obtain a first feature vector group and a second feature vector group output by the feature extraction model to be trained respectively; the first feature vector group contains feature vectors of the first transformed images, and the second feature vector group contains feature vectors of the second transformed images.

[0165] A parameter updating module 650 is configured to update model parameters of the feature extraction model based on the first feature vector group and the second feature vector group.

[0166] An iterative training module 660 is configured to repeat the above process to perform iterative training on the feature extraction model, to obtain a trained feature extraction model.

[0167] In a possible implementation, the parameter updating module 650 comprises:

[0168] A function value calculation sub-module is configured to calculate a function value of a contrast loss function based on the first feature vector group and the second feature vector group.

[0169] A parameter updating sub-module is configured to update model parameters of the feature extraction model to be trained based on the function value of the contrast loss function.

[0170] ​The function value of the contrast loss function is a ratio between a sum of first similarities and a sum of second similarities; the first similarity refers to a similarity between feature vectors of the first transformed image and the second transformed image corresponding to the same sample image; the second similarity refers to a similarity between feature vectors of the first transformed image and the second transformed image corresponding to different sample images respectively; the second transformed image corresponding to the second sample image is an image obtained by performing a second random transformation on the same category image corresponding to the second sample image.

[0171] In a possible implementation, the apparatus further includes:

[0172] An image set obtaining module is configured to obtain a target sample image set, the target sample image set including at least two third sample images; the third sample images have image category labels;

[0173] A vector obtaining module is configured to input the at least two third sample images in the target sample image set into the trained feature extraction model respectively, to obtain feature vectors of the third sample images respectively;

[0174] A feature library constructing module is configured to construct a feature library based on the feature vectors of the third sample images and the image category labels respectively possessed by the third sample images.

[0175] In a possible implementation, the feature library constructing module includes:

[0176] A grouping submodule is configured to group the at least two third sample images based on the image category labels, to obtain T sample image groups, T being a positive integer; each sample image group corresponds to a same image category label;

[0177] A clustering submodule is configured to cluster the feature vectors of the third sample images in each sample image group respectively, to obtain K clustering centers corresponding to each sample image group respectively; K is a positive integer;

[0178] A feature library constructing submodule is configured to construct the feature library based on a corresponding relationship between the feature vectors at the K clustering centers corresponding to each sample image group respectively and the image category labels; the feature library includes each image category label and K feature vectors corresponding to the image category label respectively.

[0179] In summary, the model training apparatus provided by the embodiments of the present application uses training samples containing both labeled sample images and unlabeled sample images, performs first random transformation on the sample images in the training samples to obtain a first set of transformed images, performs second random transformation on the unlabeled sample images in the training samples, obtains images of the same category based on the image classification labels of the labeled sample images, and performs second random transformation on the images of the same category to obtain a second set of transformed images; extracts feature vectors of each transformed image in the first set of transformed images and each transformed image in the second set of transformed images by using a feature extraction model to be trained, respectively obtains a first feature vector group and a second feature vector group, and trains the feature extraction model based on the first feature vector group and the second feature vector group, iteratively trains the feature extraction model by repeating the above process, and obtains a trained feature extraction model; by using the method of semi-supervised contrastive learning, the similarity in image vision and the similarity in image semantic information can be focused on in the training process of the feature extraction model, thereby improving the accuracy of image classification; meanwhile, only part of the sample images need to be labeled with image classification labels in the training process, which reduces the cost of manual labeling and makes the feature vectors extracted by the feature extraction model obtained through training more suitable for the classification scenario when the image to be classified is not in the pre-defined category set during model training.

[0180] Figure 7 A block diagram of an image classification apparatus provided by an example embodiment of the present application is shown, which can be used to perform all or part of the steps in the embodiments shown in Figure 2 or Figure 4 , as shown in Figure 7 , the image classification apparatus comprises:

[0181] An image acquisition module 710 is configured to acquire an image to be classified.

[0182] A vector acquisition module 720 is configured to input the image to be classified into a feature extraction model to obtain a first feature vector of the image to be classified output by the feature extraction model; the feature extraction model is trained according to the model training method provided by the above embodiments.

[0183] A similarity matching module 730 is configured to perform similarity matching between the first feature vector of the image to be classified and each second feature vector corresponding to each image category label in a feature library, respectively, to obtain a similarity score between the first feature vector and each second feature vector corresponding to each image category label; the second feature vector is a feature vector of a sample image corresponding to the image category label at any clustering center.

[0184] The score calculation module 740 is configured to calculate average similarity scores between the first feature vector and each of the image category labels.

[0185] The category determination module 750 is configured to determine an image category of the image to be classified according to an image category indicated by a target image category label, the target image category label being one of the image category labels with the highest average similarity score.

[0186] In a possible implementation, the category determination module 750 is configured to, in a case where the average similarity score corresponding to the target image category label is higher than a score threshold, determine, as the image category of the image to be classified, an image category indicated by the image category label corresponding to the target sample image group.

[0187] In a possible implementation, the category determination module 750 is configured to, in a case where the average similarity score corresponding to the target image category label is not higher than the score threshold, determine the image category of the image to be classified as another category.

[0188] To sum up, the image classification apparatus provided by the embodiments of the present application extracts features of an image to be classified by using a feature extraction model trained by a semi-supervised contrast learning method, obtains a first feature vector of the image to be classified, performs similarity matching between the first feature vector and each second feature vector of each image classification label in a feature library, calculates average similarity scores corresponding to each image classification label, and determines an image classification corresponding to an image classification label with the highest average similarity score as the image category of the image to be classified. Since the accuracy of the feature extraction model in extracting feature vectors is improved, the accuracy of image classification can be improved based on the feature vector of the image to be classified extracted by the feature extraction model.

[0189] Figure 8 A structural block diagram of a computer device 800 is shown, which is used to illustrate an example embodiment of the present application. The computer device can be implemented as a server in the above-mentioned solutions of the present application. The computer device 800 includes a central processing unit (CPU) 801, a system memory 804 including a random access memory (RAM) 802 and a read-only memory (ROM) 803, and a system bus 805 connecting the system memory 804 and the central processing unit 801. The computer device 800 further includes a mass storage device 806 for storing an operating system 809, a client 810, and other program modules 811.

[0190] Without loss of generality, the computer readable medium can include computer storage media and communication media. Computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes RAM, ROM, Erasable Programmable Read Only Memory (EPROM), Electrically-Erasable Programmable Read-Only Memory (EEPROM), flash memory or other solid state memory technology, CD-ROM, Digital Versatile Disc (DVD), or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices. It should be understood by those skilled in the art that computer storage media does not limit to the above-mentioned several kinds. The system memory 804 and the mass storage device 806 mentioned above can be collectively referred to as memory.

[0191] According to various embodiments of the present application, the computer device 800 can also run on a remote computer connected to a network such as the Internet. That is, the computer device 800 can be connected to a network 808 through a network interface unit 807 connected to the system bus 805, or can be connected to other types of networks or remote computer systems (not shown) using the network interface unit 807.

[0192] The memory further includes at least one instruction, at least one program, a code set or an instruction set, which are stored in the memory, and the central processing unit 801 implements the model training method shown in each of the embodiments, and / or all or part of the steps in the image classification method by executing the at least one instruction, at least one program, code set or instruction set.

[0193] Figure 9 A structural block diagram of a computer device 900 shown in an example embodiment of the present application is shown. The computer device 900 can be implemented as the terminal described above, such as a smart phone, a tablet computer, a notebook computer, a desktop computer, etc. The computer device 900 can also be referred to as a user device, a portable terminal, a laptop terminal, a desktop terminal, and other names.

[0194] Generally, the computer device 900 includes a processor 901 and a memory 902.

[0195] The processor 901 can include one or more processing cores, such as a 4-core processor, a 9-core processor, and the like. The processor 901 can be implemented in at least one of a hardware form of a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), a PLA (Programmable Logic Array). The processor 901 can also include a main processor and a coprocessor, the main processor being a processor for processing data in an awake state, also known as a CPU (Central Processing Unit), and the coprocessor being a low-power processor for processing data in a standby state. In some embodiments, the processor 901 can be integrated with a GPU (Graphics Processing Unit) for rendering and drawing content required to be displayed by the display screen. In some embodiments, the processor 901 can further include an AI (Artificial Intelligence) processor for processing machine learning related computing operations.

[0196] The memory 902 can include one or more computer-readable storage media that can be non-transitory. The memory 902 can also include a high-speed random access memory, and a nonvolatile memory such as one or more disk storage devices, flash storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 902 is used to store at least one instruction for being executed by the processor 901 to implement the model training method shown in the method embodiments of the present application, and / or all or part of the steps in the image classification method.

[0197] In some embodiments, the computer device 900 can also optionally include a peripheral device interface 903 and at least one peripheral device. The processor 901, the memory 902, and the peripheral device interface 903 can be connected through a bus or a signal line. Each peripheral device can be connected to the peripheral device interface 903 through a bus, a signal line, or a circuit board. Specifically, the peripheral device includes at least one of a radio frequency circuit 904, a display screen 905, a camera component 906, an audio circuit 907, and a power supply 908.

[0198] In some embodiments, the computer device 900 further includes one or more sensors 909. The one or more sensors 909 include, but are not limited to, an acceleration sensor 910, a gyroscope sensor 911, a pressure sensor 912, an optical sensor 913, and a proximity sensor 914.

[0199] Those skilled in the art will understand that Figure 9 The structure shown does not constitute a limitation on the computer device 900, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0200] In one exemplary embodiment, a computer-readable storage medium is also provided, which stores at least one computer program that is loaded and executed by a processor to implement all or part of the steps in the model training method and / or image classification method described above. For example, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, or optical data storage device, etc.

[0201] In one exemplary embodiment, a computer program product is also provided, comprising at least one computer program that is loaded and executed by a processor. Figure 1 , Figure 2 ,or Figure 4 The model training method shown in any embodiment, and / or all or part of the steps of the image classification method.

[0202] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the claims.

[0203] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A model training method, characterized in that, The method comprises: obtaining training samples, wherein the training samples comprise n first sample images and m second sample images; the second sample images have image category labels; m and n are positive integers; performing first random transformation on the m+n sample images in the training samples respectively to obtain a first transformed image set; performing second random transformation on the n first sample images and the m second sample images and the same category images corresponding to the second sample images respectively to obtain a second transformed image set; the same category images corresponding to the second sample images are images having the same image category labels as the second sample images; inputting first transformed images in the first transformed image set and second transformed images in the second transformed image set into a feature extraction model to be trained respectively to obtain first feature vector groups and second feature vector groups output by the feature extraction model to be trained respectively; the first feature vector groups comprise feature vectors of the first transformed images respectively, and the second feature vector groups comprise feature vectors of the second transformed images respectively; updating model parameters of the feature extraction model to be trained based on the first feature vector groups and the second feature vector groups; iteratively training the feature extraction model to be trained by repeating the above process to obtain a trained feature extraction model.

2. The method of claim 1, wherein, The updating of the model parameters of the feature extraction model to be trained based on the first feature vector groups and the second feature vector groups comprises: calculating a function value of a contrast loss function based on the first feature vector groups and the second feature vector groups; updating the model parameters of the feature extraction model to be trained based on the function value of the contrast loss function; wherein the function value of the contrast loss function is a ratio between a sum of first similarities and a sum of second similarities; the first similarity refers to a similarity between feature vectors of the first transformed image and the second transformed image corresponding to the same sample image; the second similarity refers to a similarity between feature vectors of the first transformed image and the second transformed image corresponding to different sample images respectively; the second transformed image corresponding to the second sample image is an image obtained by performing second random transformation on the same category image corresponding to the second sample image.

3. The method of claim 1, wherein, After obtaining the trained feature extraction model, the method further comprises: obtaining a target sample image set, wherein the target sample image set comprises at least two third sample images; the third sample images have image category labels; inputting the at least two third sample images in the target sample image set into the trained feature extraction model respectively to obtain feature vectors of the third sample images respectively; constructing a feature library based on the feature vectors of the third sample images and the image category labels respectively possessed by the third sample images.

4. The method of claim 3, wherein, The constructing of the feature library based on the feature vectors of the third sample images and the image category labels respectively possessed by the third sample images comprises: grouping the third sample images based on the image class labels, to obtain T sample image groups, T being a positive integer, each of the sample image groups corresponding to a same image class label; performing clustering on feature vectors of the third sample images in each of the sample image groups, respectively, to obtain K cluster centers corresponding to each of the sample image groups, K being a positive integer; constructing the feature library based on a correspondence between feature vectors at the K cluster centers corresponding to each of the sample image groups and the image class labels, the feature library including each of the image class labels and K feature vectors corresponding thereto.

5. An image classification method characterized by, The method comprises: obtaining a to-be-classified image; inputting the to-be-classified image into a feature extraction model to obtain a first feature vector of the to-be-classified image output by the feature extraction model, the feature extraction model being trained according to the model training method of any one of claims 1 to 4; performing similarity matching on the first feature vector of the to-be-classified image and each second feature vector corresponding to each image class label in a feature library, respectively, to obtain a similarity score of the first feature vector and each second feature vector corresponding to each image class label, the second feature vector being a feature vector at any cluster center of a sample image group corresponding to the image class label; calculating an average similarity score between the first feature vector and each image class label; determining an image class of the to-be-classified image according to an image class indicated by a target image class label, the target image class label being one of the image class labels corresponding to the highest average similarity score.

6. The method of claim 5, wherein, The determining of the image class of the to-be-classified image according to the image class indicated by the target image class label comprises: in a case where the average similarity score corresponding to the target image class label is higher than a score threshold, determining the image class indicated by the target image class label as the image class of the to-be-classified image.

7. The method of claim 6, wherein, The determining of the image class of the to-be-classified image according to the image class indicated by the target image class label further comprises: in a case where the average similarity score corresponding to the target image class label is not higher than the score threshold, determining the image class of the to-be-classified image as another class.

8. An image classification apparatus characterized by comprising: The apparatus comprises: an image obtaining module configured to obtain a to-be-classified image; a vector obtaining module configured to input the to-be-classified image into a feature extraction model to obtain a first feature vector of the to-be-classified image output by the feature extraction model, the feature extraction model being trained according to the model training method provided in any one of claims 1 to 4. The similarity matching module is configured to perform similarity matching between the first feature vector of the image to be classified and each second feature vector corresponding to each image category label in the feature library, to obtain a similarity score between the first feature vector and each second feature vector corresponding to each image category label; the second feature vector is a feature vector of a sample image group corresponding to the image category label at any clustering center; The score calculation module is configured to calculate an average similarity score between the first feature vector and each image category label. The category determination module is configured to determine an image category of the image to be classified according to an image category indicated by a target image category label; the target image category label is one of the image category labels with the highest average similarity score.

9. A computer device, comprising: The computer device comprises a processor and a memory, and the memory stores at least one computer program, which is loaded and executed by the processor to implement the model training method according to any one of claims 1 to 4, and / or the image classification method according to any one of claims 5 to 7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores at least one computer program, which is loaded and executed by the processor to implement the model training method according to any one of claims 1 to 4, and / or the image classification method according to any one of claims 5 to 7.

Images