Image classification method, device, electronic device and storage medium

Abstract

The embodiment of the present invention discloses an image classification method, device, electronic device and storage medium. The method includes: obtaining an image to be processed, inputting the image into a pre-trained target image classification model, wherein the target image classification model is trained by an unsupervised learning method based on a first sample with a standard type label and a second sample without a standard type label; and determining the image type to which the image to be processed belongs according to the output result of the target image classification model. The image classification model is iteratively trained unsupervisedly by a large number of second samples without standard type labels and a small number of first samples with standard type labels, without the need to manually classify and set labels for each sample image, thereby reducing the difficulty and time of processing sample data, and further reducing the total training time of the target image classification model.

Description

Technical Field

[0001] The embodiments of the present invention relate to image processing technology, and more particularly to an image classification method, device, electronic device and storage medium. Background Art

[0002] With the rapid development of electronic technology and the Internet, especially mobile Internet, the functions of electronic devices, especially smart mobile terminals, are becoming more and more powerful. Users can install various applications on smart mobile terminals according to their own needs to complete various tasks. For example, image recognition is achieved through applications installed on electronic devices.

[0003] At present, in order to recognize images and identify the category to which the images belong, the relevant technologies generally implement it through machine learning models, and machine learning models are generally obtained through supervised learning training with a large amount of sample data. In the process of supervised learning training of machine learning models, not only a large number of data samples are required, but also each sample data needs to be manually classified, that is, the sampling process is a huge workload, which increases the difficulty of training machine learning models. If the number of sample data is reduced, the image classification accuracy of the machine learning model will be poor. Summary of the invention

[0004] The embodiments of the present invention provide an image classification method, device, electronic device and storage medium to reduce the difficulty of training while ensuring the accuracy of a machine learning model.

[0005] In a first aspect, an embodiment of the present invention provides an image classification method, comprising:

[0006] Acquire an image to be processed, and input the image into a pre-trained target image classification model, wherein the target image classification model is trained by an unsupervised learning method based on a first sample with a standard type label and a second sample without a standard type label;

[0007] The image type to which the image to be processed belongs is determined according to the output result of the target image classification model.

[0008] In a second aspect, an embodiment of the present invention further provides an image classification device, the device comprising:

[0009] The module for acquiring an image to be processed is used to acquire an image to be processed and input the image into a pre-trained target image classification model, wherein the target image classification model is trained by an unsupervised learning method based on a first sample with a standard type label and a second sample without a standard type label;

[0010] The image type determination module is used to determine the image type to which the image to be processed belongs according to the output result of the target image classification model.

[0011] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, an image classification method as provided in any embodiment of the present application is implemented.

[0012] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements an image classification method as provided in any embodiment of the present application.

[0013] The technical solution provided by the embodiment of the present invention performs unsupervised iterative training on the image classification model through a large number of second samples without standard type labels and a small number of first samples with standard type labels, thereby obtaining a target image classification model with image classification function, without the need to manually classify and set labels for each sample image, thereby reducing the difficulty and time of processing sample data, and further reducing the total training time of the target image classification model. At the same time, auxiliary training is performed through a small number of first samples with standard type labels, and correction is performed during the unsupervised training process, thereby ensuring the training accuracy of the target image classification model and improving the accuracy of image classification. The target image classification model trained in the above manner performs classification processing on the image to be processed, with fast classification speed and high accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a flowchart of an image classification method provided in Embodiment 1 of the present invention;

[0015] Figure 2 is a flowchart of an image classification method provided by Embodiment 2 of the present invention;

[0016] Figure 3 This is a schematic diagram of sample data and cluster center points improved by the second embodiment of the present invention;

[0017] Figure 4 is a flowchart of an image classification method provided in Embodiment 3 of the present invention;

[0018] Figure 5 is a structural schematic diagram of an image classification device provided by Embodiment 4 of the present invention;

[0019] Figure 6 It is a structural schematic diagram of an electronic device provided in Embodiment 5 of the present invention. DETAILED DESCRIPTION

[0020] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0021] Embodiment 1

[0022] Figure 1 This is a flow chart of an image classification method provided in Embodiment 1 of the present invention. This embodiment is applicable to the case where an image classification model trained in an unsupervised manner is used to classify images. The method can be performed by the image classification device provided in the embodiment of the present invention, and specifically includes the following steps:

[0023] S110, obtaining an image to be processed, and inputting the image into a pre-trained target image classification model, wherein the target image classification model is trained by an unsupervised learning method based on a first sample with a standard type label and a second sample without a standard type label.

[0024] S120. Determine the image type to which the image to be processed belongs according to the output result of the target image classification model.

[0025] The image to be processed may be an image stored locally in the electronic device, or may be an image collected in real time by an image acquisition module of the electronic device. The electronic device may include but is not limited to a mobile phone, a tablet computer, a computer, a smart camera, and a server. In this embodiment, the image to be processed is classified, and the image type of the image to be processed is determined, so as to facilitate processing of the image to be processed according to the image type. Optionally, the image to be processed may be classified and stored according to the image type. For example, taking a mobile phone as an example, the images taken by the camera of the mobile phone are classified, and the captured images are stored in corresponding folders according to the image type, so as to facilitate the user to manage the captured images. The image type may include but is not limited to animal type, person type, landscape type, file type, etc.; in some embodiments, the image to be processed may also be a medical detection image, and the medical detection images are classified, and the image type of each medical detection image is determined. The medical detection images are classified and stored according to the image type, so as to facilitate management, retrieval and comparison of medical detection images of the same type. For example, the image type of the medical detection image may include but is not limited to head detection images, leg detection images, breast detection images, etc. Optionally, the target object corresponding to the image to be processed may be classified according to the image type, wherein the target object is the target object photographed by the image to be processed. Exemplarily, the target object may be a logistics package to be sorted. By classifying the logistics package image, the type of the logistics package is further determined according to the image type of the logistics package image. The logistics packages are sorted according to their types, thereby facilitating rapid sorting of the logistics packages and replacing the task of classifying the logistics packages.

[0026] In this embodiment, the image to be processed is classified by a pre-trained target image classification model, wherein the target image classification model may be a strong classifier composed of multiple weak classifiers, such as a regression tree module or a boosted tree model, etc., or a deep learning neural network model, such as a convolutional network model. Optionally, the target image classification model is a prototype network model (Prototypical Networks), which can map sample data into a space, determine the prototype (class center point) of the type by extracting the mean of the sample data of the same type, and further determine the image type of the image to be processed by the distance between the image to be processed and the prototypes of each type, wherein the distance to the prototype of the same type is small, and the distance to the prototype of a different type is large.

[0027] In this embodiment, the target image classification model is obtained by training the first sample with a standard type label and the second sample without a standard type label through unsupervised learning, wherein the number of samples of the first sample with a standard type label is small, and the number of samples of the second sample without a standard type label is large. The target image classification model is obtained by training the image classification model with a large number of second samples without a standard type label and a small number of first samples with a standard type label, thereby reducing the workload of manually classifying and setting labels for sample data in the early stage of training, simplifying the total time consumed by training the target image classification model, and improving the training efficiency of the target image classification model. Among them, the standard type label of the first sample can be set according to the image type of the first sample by identifying the image type of the first sample. Exemplarily, the first sample and the second sample can be sample data (sample images) downloaded from the network, and the image type of the first sample is identified. According to the corresponding relationship between the image type and the type label, the standard type label of the first sample is set. For example, if the image type of the first sample is scenery, the standard type label of the first sample is set to a scenery label. The image type of the first sample can be determined by manual identification.

[0028] Exemplarily, taking the prototype network model as an example, the unsupervised training process of the target image classification model may be: the first sample includes at least two sample data of each image type, and a standard type label is set for the sample data in the first sample data, and the second sample includes multiple sample data, wherein the sample data may be obtained by downloading from the network. The first type center point of each image type can be determined by each sample data in the first sample, and the multiple sample data in the second sample are predicted by the initially established image classification model to obtain the predicted image type, and the second type center point of each image type is further determined, and the loss function is determined according to the difference between the first type center point and the second type center point of the same image type, and the initially established image classification model is adjusted according to the loss function, and the image classification model is updated, and the sample data in the second sample is cyclically predicted by the updated image classification model, and a new second type center point is determined based on the prediction result, and a new loss function is further determined, and the updated image classification model is iteratively adjusted according to the new loss function until the first type center point and the second type center point of the same image type overlap or the difference is less than the preset error, then it is determined that the training of the target image classification model is completed.

[0029] It should be noted that during the model training process, a loss function layer may be connected to the output layer of the target image classification model to calculate the loss function. Correspondingly, when the target image classification model training is completed, the loss function layer is cancelled.

[0030] In this embodiment, the image to be processed is input into the target image classification model to obtain the output result of the target image classification model, wherein the output result can be the probability that the image to be processed belongs to each image type, wherein the image type with the highest probability is determined as the image type of the image to be processed.

[0031] The technical solution of this embodiment is to perform unsupervised iterative training on the image classification model through a large number of second samples without standard type labels and a small number of first samples with standard type labels, so as to obtain a target image classification model with image classification function, without the need to manually classify and set labels for each sample image, thereby reducing the difficulty and processing time of sample data, and further reducing the total training time of the target image classification model. At the same time, auxiliary training is performed through a small number of first samples with standard type labels, and correction is performed during the unsupervised training process, thereby ensuring the training accuracy of the target image classification model and improving the accuracy of image classification. The target image classification model trained in the above manner performs classification processing on the images to be processed, and the classification speed is fast and the accuracy is high.

[0032] Embodiment 2

[0033] Figure 2 : is a flow chart of an image classification method provided in Embodiment 2 of the present invention. Based on the above embodiment, an unsupervised training method of a target image classification model is provided. The method specifically includes:

[0034] S210: Establish a first image classification model according to the first sample with the standard type label.

[0035] S220: Predict the second sample without the standard type label according to the first image classification model, and generate a predicted type label for the second sample according to the prediction result.

[0036] S230. Determine a first loss function according to the first sample, the standard type label of the first sample, the second sample, and the predicted type label of the second sample.

[0037] S240. Adjust the parameters of the first image classification model according to the first loss function to generate the target image classification model.

[0038] S250, obtaining an image to be processed, inputting the image into a pre-trained target image classification model, and determining the image type to which the image to be processed belongs according to an output result of the target image classification model.

[0039] In step S210, the first image classification model is established based on the first sample with the standard type label, which can be established and trained based on a small amount of the first sample through supervised training. Since the amount of sample data in the first sample is small, the first image classification model is generated quickly, further reducing the total training time of the target image classification model.

[0040] The sample data in the second sample is input into the first image classification model, and the predicted image type of the sample data is determined according to the output result of the first image classification model, wherein the output result of the first image classification model is the probability that the sample data belongs to each image type, the image type with the highest probability is determined as the predicted image type of the sample data, and the predicted type label of the sample data is set according to the predicted image type. The standard type label of the first sample and the predicted type label of the second sample can be a string formed by numbers, characters, symbols, or at least one combination of the above, which is used to characterize the image type of the sample data.

[0041] In this embodiment, the prediction accuracy of the predicted type label of the second sample is assisted in determining by the first sample and the standard type label of the first sample. When the prediction accuracy is high, the first loss function is small. Conversely, when the prediction accuracy is low, the first loss function is large. Optionally, the first loss function is determined according to the first sample, the standard type label of the first sample, the second sample, and the predicted type label of the second sample, including: determining the first type center point of each image type according to the first sample and the standard type label of the first sample; determining the second type center point of each image type according to the second sample and the predicted type label of the second sample; determining the third type center point of each image type according to the first sample, the standard type label of the first sample, the second sample, and the predicted type label of the second sample; determining the first loss function according to at least two of the first type center point, the second type center point, and the third type center point of each image type. For example, see Figure 3 , Figure 3 Schematic diagram of sample data and cluster center points provided by the second embodiment of the present invention. Figure 3 The method includes three image types, each of which includes a plurality of sample data and a class center point, wherein the class center point of each image type is determined according to the sample data of the image type.

[0042] Optionally, taking the first type of center point as an example, the method for determining the first type of center point includes: determining the image type of the first sample according to the standard type label; for the first sample of any image type, extracting the feature vector of the first sample, and determining the position of the first sample according to the feature vector and a preset function; determining the first type of center point according to the position of the first sample. Wherein, a correspondence between standard type labels and image types is preset in the electronic device, and the image type of the sample data is determined according to the standard type label of the sample data in the first sample, and the first type of center point of the image type is determined through the sample data of the same image type. Wherein, since the first image classification model is a multi-layer network model, the output feature vector of the preset level in the first image classification model may be used as the feature vector of the sample data. The sample data is projected onto a metric space through the first image classification model, exemplarily, may be through Determine the position of the sample data in the metric space, and further determine the first type center point of the c image type according to the following formula (1), where the c image type is any image type among all image types:

[0043]

[0044] in, is a first type center point of the c image type determined according to the first sample with a standard type label belonging to the c image type; is the first sample with a standard type label belonging to image type c, is the number of sample data in the first sample with a standard type label belonging to image type c; is the sample data, θ is what needs to be learned, and the parameter θ determines the position between the sample data.

[0045] Correspondingly, the method for determining the second type of center point includes: determining the image type of the second sample according to the predicted type label; for the second sample of any image type, extracting the feature vector of the second sample, determining the position of the second sample according to the feature vector and a preset function; and determining the second type of center point according to the position of the second sample. Determine the position of the sample data in the metric space, and determine the second type center point of the c image type according to the following formula (2):

[0046]

[0047] in, is a second type center point of the c image type determined according to a second sample belonging to the c image type; is the second sample of image type c, is the number of sample data in the second sample belonging to the c image type; is the sample data.

[0048] Correspondingly, the method for determining the third type of center point includes: determining the image type of the second sample according to the predicted type label, determining the image type of the second sample according to the predicted type label; for the second sample and the first sample of any image type, extracting the feature vectors of the first sample and the second sample, determining the position of the sample data in the first sample and the second sample according to the feature vector and the preset function; determining the third type of center point according to the position of the sample data in the first sample and the second sample. Similarly, the second type of center point of the c image type is determined according to the following formula (3):

[0049]

[0050] In the process of training the first image classification model, the first sample is sequentially input into the first image classification model, the output feature vector of the preset level in the first image classification model and the standard image type of the output layer are extracted, the feature vector and the standard image type of each sample data are stored in the loss function layer, and the loss function layer determines the first type center point of each image type according to the feature vector and the standard image type of all sample data in the first sample. Correspondingly, the second sample is input into the first image classification model, the output feature vector of the preset level in the first image classification model and the predicted image type of the output layer are extracted, the feature vector and the predicted image type of each sample data are stored in the loss function layer, and the loss function layer determines the second type center point of each image type according to the feature vector and the predicted image type of all sample data in the second sample, and at the same time, the loss function layer determines the third type center point of each image type according to the feature vector and the standard image type of all sample data in the first sample and the feature vector and the predicted image type of all sample data in the second sample.

[0051] In this embodiment, the first class center point, the second class center point and the third class center point of each image type are determined by the first sample and the standard type label of the first sample, the second sample and the predicted type label of the second sample, and the first sample, the standard type label of the first sample, the second sample and the predicted type label of the second sample, respectively, and the first loss function is determined according to the first class center point, the second class center point and the third class center point. Among them, the first loss function is used to represent the consistency of the first class center point, the second class center point and the third class center point. The worse the consistency of the first class center point, the second class center point and the third class center point, the greater the first loss function. Optionally, the first loss function is determined according to at least two of the first class center point, the second class center point and the third class center point of each image type, including: for any image type, determining the class center point difference according to at least two of the first class center point, the second class center point and the third class center point; determining the first loss function according to the mean of the class center point difference of each image type. The difference between any two of the first class center point, the second class center point and the third class center point of the current image type can be determined as the class center point difference of the current image type, or three differences can be obtained based on the difference between any two of the first class center point, the second class center point and the third class center point of the current image type, and the class center point difference of the current image type is determined based on the three differences. The class center point differences of various image types are averaged to determine the first loss function. Optionally, the first class center point, the second class center point and the third class center point are mapped to the Hilbert space (reproducing kernel Hilbert space, RKHS), and the class center point difference is determined in the Hilbert space, wherein the Hilbert space is a generalization of the n-dimensional Euclidean space and can be regarded as an "infinite-dimensional Euclidean space". The difference between the class center points in the Hilbert space represents the distance between the data distributions of each class, wherein the difference between the class center points can be represented by the Euclidean distance. The first loss function is determined according to the following formula (4):

[0052]

[0053] in, is the first loss function determined according to the first type of center point, the second type of center point and the third type of center point, and are the first type center point, the second type center point and the third type center point of the c image type in the Hilbert space, respectively, and C is the number of image types. It should be noted that the first loss function can also be or

[0054] In this embodiment, the first loss function may also include the classification error of the first sample, that is, the first loss function is and The sum of , which is used to characterize the error of setting the standard type label for the sample data in the first sample. For sample data The distribution error of

[0055]

[0056] in, For sample data The probability of belonging to image type c, and Wherein, d is the distance function, which can be the Euclidean distance in this embodiment. c' is any image type among a total of C image types. The meaning of this formula is to normalize the probability of each image type belonging to c based on the probabilities of all C image types.

[0057] Accordingly, after determining the first loss function, the first loss function is reversely input into the first image classification model, and the network parameters of the first image classification model are adjusted by the gradient descent method, wherein the network parameters at least include weights. Optionally, after adjusting the parameters of the first image classification model according to the loss function, it also includes: re-predicting the second sample according to the adjusted first image classification model, and updating the predicted type label of the second sample; determining a new loss function according to the first sample, the standard type label of the first sample, the second sample, and the updated predicted type label of the second sample; iteratively adjusting the parameters of the first image classification model according to the new loss function until the new loss function is less than a preset threshold, thereby generating a target image classification model.

[0058] The technical solution provided in this embodiment determines the first type of center point according to the first sample, determines the second type of center point according to the second sample, and determines the third type of center point according to the first sample and the second sample, determines the loss function according to the difference value between the first type of center point, the second type of center point and the third type of center point of the same image type, and iteratively adjusts the parameters of the first image classification model, updates the first image classification model, and updates the predicted image type of the second sample based on the updated first image classification model, continuously narrowing the difference between the first type of center point, the second type of center point and the third type of center point of the same image type, so that the first type of center point, the second type of center point and the third type of center point of the same image type are close to or even overlapped, and the target image classification model is generated. In the above scheme, by continuously updating the first image classification model and updating the prediction result of the second sample without the standard type label, the training of the first image classification model is realized in an unsupervised manner, replacing the supervised model training method in the related art, greatly reducing the processing workload of the sample data, and reducing the total training time of the target image classification model.

[0059] Embodiment 3

[0060] Figure 4 : is a flow chart of an image classification method provided by Embodiment 3 of the present invention. On the basis of the above embodiment, the unsupervised training method of the target image classification model is optimized. The method specifically includes:

[0061] S310: Establish a first image classification model according to the first sample with the standard type label.

[0062] S320: Predict the second sample without the standard type label according to the first image classification model, and generate a predicted type label for the second sample according to the prediction result.

[0063] S330: Determine a first type center point of each image type according to the first sample and the standard type label of the first sample.

[0064] S340: Determine a second type center point of each image type according to the second sample and the predicted type label of the second sample.

[0065] S350: Determine a third type center point of each image type according to the first sample, the standard type label of the first sample, the second sample, and the predicted type label of the second sample.

[0066] S360, determining a first loss function according to at least two of the first category center points, the second category center points, and the third category center points of each image type.

[0067] S370 , determining a first distribution probability of the test sample in various image types according to the first type center point of each image type.

[0068] S380 : Determine the second distribution probability of the test sample in various image types according to the second type center point of each image type.

[0069] S390: Determine the third distribution probability of the test sample in various image types according to the third category center point of each image type.

[0070] S3100. Determine a second loss function according to at least two of the first distribution probability, the second distribution probability, and the third distribution probability of the test sample in various image types.

[0071] S3110. Adjust the parameters of the first image classification model according to the first loss function and / or the second loss function to generate the target image classification model.

[0072] S3120, obtaining an image to be processed, inputting the image into a pre-trained target image classification model, and determining the image type to which the image to be processed belongs according to an output result of the target image classification model.

[0073] In this embodiment, since the image classification can be performed by determining the Euclidean distance between the image and the class center points of various image types, the image type of the image is determined according to the class center point that is closest to the image. By determining the first class center point, the second class center point or the third class center point of each image type, three metric spaces for image classification are formed, for example, a metric space composed of the first class center points of various image types, a metric space composed of the second class center points of various image types, and a metric space composed of the third class center points of various image types. Based on the above three metric spaces, the same test sample is classified, the distribution probability of various image types of the test sample in each metric space is determined, and the first image classification model is further adjusted according to the difference in distribution probability to reduce the difference in distribution probability of various image types of the test sample in each metric space.

[0074] In step S370, the method for determining the first distribution probability includes: extracting the feature vector of the test sample; determining the first distance between the feature vector of the test sample and the first type center point of each image type; determining the first probability that the test sample belongs to each image type according to the first distance, wherein the first probability is negatively correlated with the first distance, and determining the first distribution probability according to the first probability that the test sample belongs to each image type. Extracting the feature vector of the test sample may be inputting the test sample into a first image classification model, and using the output feature vector of the prediction level in the first image classification model as the feature vector of the test sample. Calculate the Euclidean distance between the feature vector of the test sample and the first type center point of each image type, that is, the first distance. For any image type, the first probability that the test sample belongs to the current image type is negatively correlated with the first distance between the test sample and the first type center point of the current image type.

[0075] Correspondingly, in step S380, the method for determining the second distribution probability includes: extracting the feature vector of the test sample; determining the second distance between the feature vector of the test sample and the second class center point of each image type; determining the second probability that the test sample belongs to each image type according to the second distance, wherein the second probability is negatively correlated with the second distance, and the second distribution probability is determined according to the second probability that the test sample belongs to each image type. In step S390, the method for determining the third distribution probability includes: extracting the feature vector of the test sample; determining the third distance between the feature vector of the test sample and the third class center point of each image type; determining the third probability that the test sample belongs to each image type according to the third distance, wherein the third probability is negatively correlated with the third distance, and the third distribution probability is determined according to the third probability that the test sample belongs to each image type.

[0076] Optionally, the second loss function is determined based on at least two of the first distribution probability, the second distribution probability and the third distribution probability of the test sample in various image types, including: determining the probability distribution difference of the test sample based on at least two of the first distribution probability, the second distribution probability and the third distribution probability; determining the first loss function based on the probability distribution difference of the test sample. In this embodiment, the probability distribution difference between two distribution probabilities is represented by KL divergence. Among them, the KL divergence of any two of the first distribution probability, the second distribution probability and the third distribution probability of the current image type can be determined as the probability distribution difference of the current image type, and the probability distribution difference of the current image type can also be determined based on the KL divergence of any two of the first distribution probability, the second distribution probability and the third distribution probability of the current image type. The differences in the class center points of various image types are averaged to determine the second loss function. Exemplarily, the second loss function can be determined according to the following formula (6):

[0077]

[0078] in, is a second loss function determined based on the first distribution probability, the second distribution probability and the third distribution probability, is the first distribution probability, is the second distribution probability, is the third distribution probability, D KL (·) is the divergence function, |S s | is the first sample size, is the second sample size.

[0079]

[0080]

[0081]

[0082] at the same time, Similarly, we can get and I will not go into details here.

[0083] In some embodiments, the second loss function can also be or No limitation is made here.

[0084] In this embodiment, the first loss function is determined according to the difference in class center points, and the second loss function is determined according to the difference in distribution probability of test samples. The parameters of the first image classification model can be adjusted based on the first loss function or the second loss function alone, or the parameters of the first image classification model can be adjusted based on the first loss function and the second loss function together. Until the first class center point, the second class center point and the third class center point of the same image type are close to or overlap, and the distribution probability of the test samples in various image types is the same or similar, it is determined that the training of the target image classification model is completed.

[0085] At this time, the new loss function determined by the first loss function and the second loss function can be:

[0086] Among them, α and β are preset constants. It should be noted that the first loss function can also be or The second loss function can also be or No limitation is made here.

[0087] The technical solution provided in this embodiment improves the training accuracy and training efficiency of the target image classification model by determining a first loss function and a second loss function, and performing unsupervised training on the first image classification model based on the first loss function and / or the second loss function, and adjusting the first image classification model by the dual loss function.

[0088] Embodiment 4

[0089] Figure 5 4 is a schematic diagram of the structure of an image classification device provided in the fourth embodiment of the present invention, the device includes a to-be-processed image acquisition module 410 and an image type determination module 420, wherein:

[0090] The to-be-processed image acquisition module 410 is used to acquire the to-be-processed image and input the image into a pre-trained target image classification model, wherein the target image classification model is trained by an unsupervised learning method based on a first sample with a standard type label and a second sample without a standard type label;

[0091] The image type determination module 420 is used to determine the image type to which the image to be processed belongs according to the output result of the target image classification model.

[0092] Optionally, the device further comprises:

[0093] A model building module, used to build a first image classification model according to the first sample with a standard type label;

[0094] A sample prediction module, used to predict the second sample without the standard type label according to the first image classification model, and generate a predicted type label of the second sample according to the prediction result;

[0095] A first loss function determination module, configured to determine a first loss function according to the first sample, the standard type label of the first sample, the second sample, and the predicted type label of the second sample;

[0096] The first model adjustment module is used to adjust the parameters of the first image classification model according to the first loss function to generate the target image classification model.

[0097] Optionally, the first loss function determination module includes:

[0098] A first type center point determination unit, configured to determine a first type center point of each image type according to the first sample and a standard type label of the first sample;

[0099] A second type center point determination unit, configured to determine a second type center point of each image type according to the second sample and a predicted type label of the second sample;

[0100] A third type center point determination unit, configured to determine a third type center point of each image type according to the first sample, the standard type label of the first sample, the second sample, and the predicted type label of the second sample;

[0101] The first loss function determining unit is used to determine a first loss function according to at least two of the first type of central points, the second type of central points and the third type of central points of each image type.

[0102] Optionally, the class center point determination unit is used to:

[0103] An image type of the first sample and / or the second sample according to the standard type label or the predicted type label;

[0104] For the first sample and / or the second sample of any image type, extract a feature vector of the first sample and / or the second sample, and determine a position of the first sample and / or a position of the second sample according to the feature vector and a preset function;

[0105] The first category center point, the second category center point or the third category center point is determined according to the position of the first sample and / or the position of the second sample.

[0106] Optionally, the first loss function determining unit is used to:

[0107] For any image type, determining a class center point difference according to at least two of the first class center points, the second class center points, and the third class center points;

[0108] The first loss function is determined according to the mean of the class center point differences of each image type.

[0109] Optionally, the device further comprises:

[0110] A first distribution probability determination module, used to determine a first distribution probability of a test sample in various image types according to a first type center point of each image type;

[0111] A second distribution probability determination module, used to determine the second distribution probability of the test sample in various image types according to the second type center point of each image type;

[0112] A third distribution probability determination module, used to determine the third distribution probability of the test sample in various image types according to the third type center point of each image type;

[0113] A second loss function determination module, configured to determine a second loss function according to at least two of the first distribution probability, the second distribution probability and the third distribution probability of the test sample in various image types;

[0114] Correspondingly, the model adjustment module is used to adjust the parameters of the first image classification model according to the first loss function and / or the second loss function to generate the target image classification model.

[0115] Optionally, the distribution probability determination module is used to:

[0116] Extracting a feature vector of the test sample;

[0117] Determine a first distance, a second distance or a third distance between the feature vector of the test sample and the first type center point, the second type center point or the third type center point of each image type;

[0118] Determine a first probability, a second probability, or a third probability that the test sample belongs to each image type according to the first distance, the second distance, or the third distance, wherein the first probability is negatively correlated with the first distance, the second probability is negatively correlated with the second distance, and the third probability is negatively correlated with the third distance;

[0119] The first distribution probability, the second distribution probability or the third distribution probability is determined according to the first probability, the second probability or the third probability that the test sample belongs to each of the image types.

[0120] Optionally, a second loss function determination module is used to:

[0121] Determine a probability distribution difference of the test sample according to at least two of the first distribution probability, the second distribution probability, and the third distribution probability;

[0122] The first loss function is determined according to the probability distribution difference of the test samples.

[0123] Optionally, the device further comprises:

[0124] A prediction type label updating module, configured to, after adjusting the parameters of the first image classification model according to the loss function, re-predict the second sample according to the adjusted first image classification model, and update the prediction type label of the second sample;

[0125] a loss function updating module, configured to determine a new loss function according to the first sample, the standard type label of the first sample, the second sample, and the updated predicted type label of the second sample;

[0126] The second model adjustment module is used to iteratively adjust the parameters of the first image classification model according to the new loss function until the new loss function is less than a preset threshold.

[0127] The image classification device provided by the embodiment of the present invention can execute the image classification method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the image classification method.

[0128] Embodiment 5

[0129] Figure 6 This is a schematic diagram of the structure of an electronic device provided in Embodiment 5 of the present invention. Figure 6 A block diagram of an electronic device 512 suitable for implementing embodiments of the present invention is shown. Figure 6 The electronic device 512 shown is only an example and should not limit the functions and scope of use of the embodiments of the present invention. The device 512 is typically an electronic device that performs image classification functions.

[0130] like Figure 6 As shown, the electronic device 512 is in the form of a general purpose computing device. The components of the electronic device 512 may include, but are not limited to: one or more processors 516, a storage device 528, and a bus 518 connecting different system components (including the storage device 528 and the processor 516).

[0131] Bus 518 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. For example, these architectures include but are not limited to Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus and Peripheral Component Interconnect (PCI) bus.

[0132] The electronic device 512 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device 512, including volatile and non-volatile media, removable and non-removable media.

[0133] The storage device 528 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 530 and / or cache memory 532. The electronic device 512 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 534 may be used to read and write non-removable, non-volatile magnetic media ( Figure 6 not shown, usually called a "hard drive"). Although Figure 6 Not shown in the figure, a disk drive for reading and writing a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing a removable non-volatile optical disk (e.g., a read-only optical disk (Compact Disc-Read Only Memory, CD-ROM), a digital video disk (Digital Video Disc-Read Only Memory, DVD-ROM) or other optical media) may be provided. In these cases, each drive may be connected to the bus 518 via one or more data medium interfaces. The storage device 528 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present invention.

[0134] A program 536 having a set (at least one) of program modules 526 may be stored, for example, in a storage device 528, such program modules 526 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination thereof may include an implementation of a network environment. The program modules 526 generally perform the functions and / or methods of the embodiments described herein.

[0135] The electronic device 512 may also communicate with one or more external devices 514 (e.g., keyboard, pointing device, camera, display 524, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 512, and / or communicate with any device that enables the electronic device 512 to communicate with one or more other computing devices (e.g., network card, modem, etc.). Such communication may be performed through an input / output (I / O) interface 522. In addition, the electronic device 512 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN) and / or a public network, such as the Internet) through a network adapter 520. As shown, the network adapter 520 communicates with other modules of the electronic device 512 through a bus 518. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 512, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, disk arrays (Redundant Arrays of Independent Disks, RAID) systems, tape drives, and data backup storage systems.

[0136] The processor 516 executes various functional applications and data processing by running the programs stored in the storage device 528, such as implementing the image classification method provided in the above embodiment of the present invention.

[0137] Embodiment 6

[0138] Embodiment 6 of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the image classification method provided in the embodiment of the present invention is implemented.

[0139] Of course, the computer-readable storage medium provided by the embodiment of the present invention stores a computer program which is not limited to the method operation described above, and can also execute the image classification method provided by any embodiment of the present invention.

[0140] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0141] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0142] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0143] Computer program code for performing the operations of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0144] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. An image classification method, characterized in that: include: Acquire an image to be processed, and input the image into a pre-trained target image classification model, wherein the target image classification model is trained by an unsupervised learning method based on a first sample with a standard type label and a second sample without a standard type label; Determine the image type to which the image to be processed belongs according to the output result of the target image classification model; The training method of the target image classification model includes: Establishing a first image classification model according to the first samples with standard type labels; wherein the number of samples of the first samples with standard type labels is small, and the number of samples of the second samples without standard type labels is large; Predicting the second sample without the standard type label according to the first image classification model, and generating a predicted type label for the second sample according to the prediction result; Determine a first loss function according to the first sample, the standard type label of the first sample, the second sample, and the predicted type label of the second sample; Adjusting the parameters of the first image classification model according to the first loss function to generate the target image classification model; wherein the target image classification model includes a prototype network model; The determining a first loss function according to the first sample, the standard type label of the first sample, the second sample, and the predicted type label of the second sample includes: Determine a first type center point of each image type according to the first sample and a standard type label of the first sample; Determine a second type center point of each image type according to the second sample and the predicted type label of the second sample; Determine a third type center point of each image type according to the first sample, the standard type label of the first sample, the second sample, and the predicted type label of the second sample; Determine a first loss function according to at least two of the first type of central points, the second type of central points, and the third type of central points of each image type; The method further comprises: Determine the first distribution probability of the test sample in various image types according to the first type center point of each image type; Determining the second distribution probability of the test sample in various image types according to the second type center point of each image type; Determining the third distribution probability of the test sample in various image types according to the third type center point of each image type; Determine a second loss function according to at least two of the first distribution probability, the second distribution probability, and the third distribution probability of the test sample in various image types; Accordingly, adjusting the parameters of the first image classification model according to the first loss function to generate the target image classification model includes: Adjusting the parameters of the first image classification model according to the first loss function and / or the second loss function to generate the target image classification model; Wherein, determining the second loss function according to at least two of the first distribution probability, the second distribution probability and the third distribution probability of the test sample in various image types includes: Determine the KL divergence of any two of the first distribution probability, the second distribution probability, and the third distribution probability of the current image type as the probability distribution difference of the current image type, or jointly determine the probability distribution difference of the current image type according to the KL divergence of any two of the first distribution probability, the second distribution probability, and the third distribution probability of the current image type; The probability distribution differences of each image type are averaged to determine the second loss function.

2. The method according to claim 1, characterized in that The method for determining the first type of central point, the second type of central point or the third type of central point comprises: Determine the image type of the first sample and / or the second sample according to the standard type label or the predicted type label; For the first sample and / or the second sample of any image type, extract a feature vector of the first sample and / or the second sample, and determine a position of the first sample and / or a position of the second sample according to the feature vector and a preset function; The first category center point, the second category center point or the third category center point is determined according to the position of the first sample and / or the position of the second sample.

3. The method according to claim 1, characterized in that Determining a first loss function according to at least two of the first type of central points, the second type of central points, and the third type of central points of each image type includes: For any image type, determining a class center point difference according to at least two of the first class center points, the second class center points, and the third class center points; The first loss function is determined according to the mean of the class center point differences of each image type.

4. The method according to claim 1, characterized in that: The method for determining the first distribution probability, the second distribution probability or the third distribution probability includes: Extracting a feature vector of the test sample; Determine a first distance, a second distance or a third distance between the feature vector of the test sample and the first type center point, the second type center point or the third type center point of each image type; Determine a first probability, a second probability, or a third probability that the test sample belongs to each image type according to the first distance, the second distance, or the third distance, wherein the first probability is negatively correlated with the first distance, the second probability is negatively correlated with the second distance, and the third probability is negatively correlated with the third distance; The first distribution probability, the second distribution probability or the third distribution probability is determined according to the first probability, the second probability or the third probability that the test sample belongs to each of the image types.

5. The method according to claim 1, characterized in that After adjusting the parameters of the first image classification model according to the loss function, the method further includes: Re-predicting the second sample according to the adjusted first image classification model, and updating the predicted type label of the second sample; Determine a new loss function according to the first sample, the standard type label of the first sample, the second sample, and the updated predicted type label of the second sample; Iteratively adjust the parameters of the first image classification model according to the new loss function until the new loss function is less than a preset threshold.

6. An image classification device, characterized in that: include: The module for acquiring an image to be processed is used to acquire an image to be processed and input the image into a pre-trained target image classification model, wherein the target image classification model is trained by an unsupervised learning method based on a first sample with a standard type label and a second sample without a standard type label; An image type determination module, used to determine the image type to which the image to be processed belongs according to the output result of the target image classification model; A model building module, used to build a first image classification model according to the first samples with standard type labels; wherein the number of samples of the first samples with standard type labels is small, and the number of samples of the second samples without standard type labels is large; A sample prediction module, used to predict the second sample without the standard type label according to the first image classification model, and generate a predicted type label of the second sample according to the prediction result; A first loss function determination module, configured to determine a first loss function according to the first sample, the standard type label of the first sample, the second sample, and the predicted type label of the second sample; A first model adjustment module, used for adjusting the parameters of the first image classification model according to the first loss function to generate the target image classification model; wherein the target image classification model includes a prototype network model; The first loss function determination module includes: a first-category center point determination unit, configured to determine a first-category center point of each image type according to the first sample and the standard type label of the first sample; a second-category center point determination unit, configured to determine a second-category center point of each image type according to the second sample and the predicted type label of the second sample; A third type center point determination unit, configured to determine a third type center point of each image type according to the first sample, the standard type label of the first sample, the second sample, and the predicted type label of the second sample; A first loss function determining unit, configured to determine a first loss function according to at least two of the first type of central points, the second type of central points, and the third type of central points of each image type; The device also includes: A first distribution probability determination module, used to determine a first distribution probability of a test sample in various image types according to a first type center point of each image type; A second distribution probability determination module, used to determine the second distribution probability of the test sample in various image types according to the second type center point of each image type; A third distribution probability determination module, used to determine the third distribution probability of the test sample in various image types according to the third type center point of each image type; A second loss function determination module, configured to determine a second loss function according to at least two of the first distribution probability, the second distribution probability and the third distribution probability of the test sample in various image types; Correspondingly, the first model adjustment module is used to adjust the parameters of the first image classification model according to the first loss function and / or the second loss function to generate the target image classification model; the second loss function determination module is used to: Determine the KL divergence of any two of the first distribution probability, the second distribution probability, and the third distribution probability of the current image type as the probability distribution difference of the current image type, or jointly determine the probability distribution difference of the current image type according to the KL divergence of any two of the first distribution probability, the second distribution probability, and the third distribution probability of the current image type; The probability distribution differences of each image type are averaged to determine the second loss function.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the image classification method as described in any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the image classification method as described in any one of claims 1 to 5 is implemented.

Images