Image Classification Model Training Method and Apparatus, Image Classification Method, and Device

Abstract

The present application discloses an image classification model training method and apparatus, an image classification method, and equipment, comprising: dividing training image sample data into closed-set image sample data and open-set image sample data; the closed-set image sample data includes K-class data of known categories, and the open-set image sample data includes K-class data and unknown-class data; extracting features of the closed-set image sample data; determining similar class data in the K-class data based on the closed-set sample features and through unsupervised learning, and grouping the similar class data into the same group to obtain M groups of data; using the M groups of data to train the target model structure of the image classification model, and during the training process, constructing batch data based on multiple groups of data in the M groups of data, and using the open-set image sample data to train the first classification branch structure of the image classification model to obtain a trained image classification model; improving the performance of the image classification model, thereby improving the accuracy of image classification.

Description

Technical Field

[0001] The present application relates to the field of image classification technology, and in particular to an image classification model training method and device, and an image classification method and equipment. Background Art

[0002] In the field of computer vision, both fine-grained image classification and open-set image classification are currently extremely challenging tasks. Fine-grained image classification primarily involves distinguishing basic categories while also performing more refined sub-categorization, such as distinguishing car models, bird species, and product SKUs. Compared to general classification tasks, fine-grained classification subcategories often have smaller feature differences between classes. However, image data within a subcategory can vary significantly due to factors such as the background, shooting angle, and the motion and posture of the objects. This makes classification particularly difficult. On the other hand, in open-set image classification, the classifier is required to correctly classify all data, but the data to be classified may contain new categories, making it extremely challenging to automatically identify data of unknown categories that do not belong to known categories.

[0003] At present, most fine-grained classification methods only care about general intra-class and inter-class classification during training, and it is difficult to accurately distinguish similar classes, resulting in inaccurate image classification results. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide an image classification model training method and device, an image classification method, and a device that can improve the accuracy of image classification. The specific solution is as follows:

[0005] In a first aspect, the present application discloses an image classification model training method, comprising:

[0006] The training image sample data is divided into closed set image sample data and open set image sample data; the closed set image sample data includes K class data of known categories, and the open set image sample data includes the K class data and unknown class data;

[0007] Extracting features of the closed set image sample data to obtain closed set sample features;

[0008] Determine similar class data in the K class data by unsupervised learning based on the closed set sample features, and group the similar class data into the same group to obtain M groups of data;

[0009] The target model structure of the image classification model is trained using the M groups of data, and during the training process, batch data is constructed based on multiple groups of data in the M groups of data, and the first classification branch structure of the image classification model is trained using the open set image sample data to obtain a trained image classification model; wherein the target model structure includes a feature extraction structure and a second classification branch structure.

[0010] Optionally, determining similar class data in the K class data by unsupervised learning based on the closed set sample features, and grouping the similar class data into the same group to obtain M groups of data, includes:

[0011] Classifying the closed set sample features through unsupervised learning to obtain N categories;

[0012] For any category of data in the K categories of data, determine the category with the largest amount of data in the N categories to obtain the target category corresponding to the category of data, and divide the category data corresponding to the same target category in the K categories of data into the same group to obtain M groups of data.

[0013] Optionally, during the training process, constructing batch data based on multiple groups of data in the M groups of data includes:

[0014] A plurality of groups are selected from the M groups of data, a plurality of classes are selected from the plurality of groups, and a plurality of data are selected from the plurality of classes to construct a batch of data.

[0015] Optionally, the training of a first classification branch structure of an image classification model using the open set image sample data includes:

[0016] Classifying the open set image sample data through unsupervised learning to obtain G categories;

[0017] Classify the data in the K-category data and the data in the unknown category data that belong to the same category in the G-category data into the same group to obtain G-group data;

[0018] The G group of data is used to train the first classification branch structure of the image classification model.

[0019] Optionally, the dividing the data in the K-category data and the data in the unknown category data belonging to the same category in the G categories into the same group to obtain G group data includes:

[0020] Generate a first class label for the data in the K class data; the first class label represents the corresponding class in the G class to which the data belongs;

[0021] Generate a second class label for the data in the unknown class data; the first class label represents the corresponding class in the G classes to which the data belongs; and the first class label and the second class label do not have the same label;

[0022] The data in the K-category data and the data in the unknown category data belonging to the same category in the G categories are divided into the same group to obtain G group data.

[0023] Optionally, the training of a first classification branch structure of an image classification model using the open set image sample data includes:

[0024] The open set image sample data is input into the trained feature extraction structure to obtain open set sample features, and the first classification branch structure of the image classification model is trained based on the open set sample features.

[0025] In a second aspect, the present application discloses an image classification method, comprising:

[0026] Get the image to be classified;

[0027] Inputting the image to be classified into the image classification model so that the first classification branch structure and the second classification branch structure in the image classification model output a first classification result and a second classification result corresponding to the image to be classified; wherein the image classification model is obtained based on the aforementioned image classification model training method;

[0028] A final classification result of the image to be classified is determined based on the first classification result and the second classification result.

[0029] In a third aspect, the present application discloses an image classification model training device, comprising:

[0030] An image sample data division module is used to divide the training image sample data into closed set image sample data and open set image sample data; the closed set image sample data includes K class data of known categories, and the open set image sample data includes the K class data and unknown class data;

[0031] A closed set sample feature extraction module, configured to extract features of the closed set image sample data to obtain closed set sample features;

[0032] A closed set sample data grouping module is used to determine similar class data in the K class data based on the closed set sample features through unsupervised learning, and group the similar class data into the same group to obtain M groups of data;

[0033] An image classification model training module is used to train the target model structure of the image classification model using the M groups of data, and during the training process, construct batch data based on multiple groups of data in the M groups of data, and use the open set image sample data to train the first classification branch structure of the image classification model to obtain a trained image classification model; wherein the target model structure includes a feature extraction structure and a second classification branch structure.

[0034] In a fourth aspect, the present application discloses an electronic device, including a memory and a processor, wherein:

[0035] The memory is used to store computer programs;

[0036] The processor is used to execute the computer program to implement the aforementioned image classification model training method and / or the aforementioned image classification method.

[0037] In a fifth aspect, the present application discloses a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the aforementioned image classification model training method and / or the aforementioned image classification method.

[0038] It can be seen that the present application first divides the training image sample data into closed-set image sample data and open-set image sample data, wherein the closed-set image sample data includes K-class data of known categories, and the open-set image sample data includes the K-class data and unknown category data, and then extracts the features of the closed-set image sample data to obtain closed-set sample features, and based on the closed-set sample features and through unsupervised learning, determines similar class data in the K-class data, and divides similar class data into the same group to obtain M groups of data, and finally uses the M groups of data to train the target model structure of the image classification model, and during the training process, constructs batch data based on multiple groups of data in the M groups of data, and uses the open-set image sample data to train the first classification branch structure of the image classification model to obtain the trained image classification model; wherein, the target model structure includes a feature extraction structure and a second classification branch structure. That is, the present application uses closed-set image sample data and open-set image sample data to train an image classification model. When using closed-set image sample data to train the target model structure of the image classification model, it is based on the closed-set sample features and determines similar class data in K class data through unsupervised learning. Similar class data are divided into the same group to obtain M groups of data, and training is performed using M groups of data. The batch data used for training is constructed based on multiple groups of data in the M groups of data. In this way, data of known categories are grouped according to the unsupervised learning method, and the groups include similar class data and dissimilar class data between groups. Batch data is constructed using multiple groups of data to ensure that each batch of data during training contains data of the same category, data with similar categories but different categories, and data with dissimilar categories, so that the model can fully distinguish similar categories in known categories. In addition, the use of open-set image sample data also ensures that the image classification model can recognize unknown category data, thereby improving the performance of the image classification model and thus improving the accuracy of image classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without any creative work.

[0040] Figure 1 This is a flow chart of an image classification model training method disclosed in this application;

[0041] Figure 2 This is a flow chart of an image classification method disclosed in this application;

[0042] Figure 3 This is a flowchart of a specific image classification method disclosed in this application;

[0043] Figure 4 This is a structural diagram of an image classification model training device disclosed in this application;

[0044] Figure 5 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0046] See also Figure 1 As shown, the embodiment of the present application discloses an image classification model training method, comprising:

[0047] Step S11: Divide the training image sample data into closed-set image sample data and open-set image sample data; the closed-set image sample data includes K-category data of known categories, and the open-set image sample data includes the K-category data and unknown category data.

[0048] In a specific implementation, unknown category data is classified into 1 category. The reason for classifying unknown category data into only 1 category is, on the one hand, to facilitate data labeling and reduce labeling workload; on the other hand, data in the unknown category may not be labeled and classified, such as human beings do not recognize this type of data.

[0049] Step S12: extracting features of the closed set image sample data to obtain closed set sample features.

[0050] In a specific embodiment, a pre-trained classification model (such as ResNet or VGG, etc., but not limited to this) or a self-designed classification model that has been trained on similar data can be used to extract the features of the closed set image sample data, for example, using the feature vector output by the Embedding layer as the sample feature.

[0051] Step S13: Based on the closed set sample features, similar class data in the K class data are determined by unsupervised learning, and the similar class data are grouped into the same group to obtain M groups of data.

[0052] In one embodiment, the closed set sample features can be classified through unsupervised learning to obtain N categories; for any category of data in the K categories of data, the category with the largest amount of data in the N categories is determined to obtain the target category corresponding to the category of data, and the category data corresponding to the same target category in the K categories of data are divided into the same group to obtain M groups of data.

[0053] For example, N categories include categories 0, 1, 2, and 3. A certain category of data in the K categories has the most data in category 2, so category 2 is determined as the target category corresponding to the certain category of data. At the same time, another category of data in the K categories also has the most data in category 2, and the target category of this category of data is also category 2. Then these two categories of data are divided into the same group.

[0054] In a specific embodiment, statistics can be collected for each class j in the K data sets that is assigned to each of the N classes, and the data of class j can be labeled as the class i with the most data assigned to it among the N classes. The data of class j is also labeled as group data i. Similarly, the K data sets can be divided into N groups, and the N groups of data can be directly used for training, that is, M = N. However, in some scenarios, there may be a group in the N groups that has no class data, and this group can be eliminated. In this case, M is less than N. Unsupervised learning methods can include, but are not limited to, clustering algorithms such as Kmeans.

[0055] Step S14: Use the M groups of data to train the target model structure of the image classification model, and during the training process, construct batch data based on multiple groups of data in the M groups of data, and use the open set image sample data to train the first classification branch structure of the image classification model to obtain a trained image classification model; wherein the target model structure includes a feature extraction structure and a second classification branch structure.

[0056] In a specific implementation, multiple groups can be selected from M groups of data, multiple classes can be selected from the multiple groups, and multiple data can be selected from the multiple classes to construct a batch of data. Moreover, multiple groups, multiple classes, and multiple data can all be randomly selected. The target model structure can use an open source pre-trained classification model (such as ResNet or VGG, etc.) or a self-designed classification model. Taking ResNet-18 as an example, the output channel of the fully connected layer can be modified to a known number of categories (i.e., K). The classification loss function uses Tiplet Loss and Equal Loss (or Softmax or other classification loss functions): Loss = Loss triplet +λ×Loss equal Among them, Loss triplet Tiplet Loss, Loss equalIt represents Equal Loss, and λ is a constant. Grouping the training data in this way and feeding it into the training can ensure that each batch of data during training contains data of the same category, data of similar but different categories, and data of other categories. This kind of training can ensure within-class consistency, and even if the classes are similar, they can be distinguished.

[0057] Furthermore, the first classification branch structure of the image classification model can be trained using the open-set image sample data, and specifically, it can include the following steps:

[0058] Step 00: Classify the open-set image sample data through unsupervised learning to obtain G classes.

[0059] In a specific implementation manner, the embedding feature vectors of the open-set image sample data can be extracted based on the trained target model structure, and the open-set image sample data can be classified through unsupervised learning based on the embedding feature vectors to obtain G classes. That is, the (K + 1) classes of data are divided into G classes (G > 1 and G < K). That is, unsupervised learning is performed using the feature vectors output by the embedding layer of the trained target model structure.

[0060] Step 01: Divide the data in the K classes of data belonging to the same class in the G classes and the data in the unknown class data into the same group to obtain G groups of data.

[0061] In a specific implementation manner, a first type of label can be generated for the data in the K classes of data; the first type of label represents the corresponding class in the G classes to which the data belongs; a second type of label can be generated for the data in the unknown class data; the first type of label represents the corresponding class in the G classes to which the data belongs; and, there are no identical labels between the first type of label and the second type of label; divide the data in the K classes of data belonging to the same class in the G classes and the data in the unknown class data into the same group to obtain G groups of data.

[0062] For example, for the data classified into category G among the K categories of data, corresponding class labels are generated, numbered 0, 1, ..., G-1; for the data classified into category G among the unknown categories, corresponding class labels are generated, numbered G, G+1, ..., 2×G-1. All data are divided into 2×G categories, of which category 0 and category G can be considered as similar data and are organized into group 0; category 1 and category G+1 can be considered as similar data and are organized into group 1; and so on, category G-1 and category 2×G-1 can be considered as similar data and are organized into group G-1. Through the embodiments of the present application, instead of classifying all unknown categories into one category, the originally disorganized data of the unknown category can be divided into a more orderly G category, and this G category of unknown data can also correspond one-to-one with the known G group data, which is beneficial to the subsequent model training. It should be pointed out that the reason why the closed-set known data is divided into G categories and the unknown data is divided into G corresponding categories, instead of dividing the unknown data into only one category, is that the unknown category data is disordered and messy, which makes classification difficult and the accuracy is very low. In addition, the unknown data is not divided into categories equal to the number of original categories of the closed-set data because when the extraction structure of known fine-grained categories is used to process the unknown data, the fine-grained features obtained are too messy for the unknown categories, resulting in poor learning of the unknown categories.

[0063] Step 02: Use the G group of data to train the first classification branch structure of the image classification model.

[0064] In a specific embodiment, multiple groups can be selected from the G groups of data, multiple classes can be selected from the multiple groups, and multiple data can be selected from the multiple classes to construct a batch of data. The classification loss function uses Tiplet Loss and Softmax Loss (or other classification loss functions): Loss = Loss triplet +β×Loss softmax , among which, Loss triplet Tiplet Loss, Loss softmax represents Softmax Loss, and β is a constant.

[0065] Furthermore, in one embodiment, the open set image sample data can be input into the trained feature extraction structure to obtain open set sample features, and the first classification branch structure of the image classification model can be trained based on the open set sample features. That is, the features of the batch data are input into the first classification branch structure. It should be pointed out that the reason why the intermediate features can be output based on the trained feature extraction structure for training is because the closed set data is part of the entire open set data, and this part of the feature extraction is also credible for the entire open set data set. Effective use of existing intermediate features can reduce processing time and resources. The feature extraction structure can be a multi-scale convolution structure.

[0066] Furthermore, the trained target model structure and the trained first classification branch structure can be merged to obtain the final image classification model, that is, the target model structure and the first classification branch structure are not merged before training. Alternatively, an initial model including the target model structure and the first classification branch structure can be pre-built, that is, the target model structure and the first classification branch structure are merged before training, and the target model structure is trained first, followed by the first classification branch structure, to obtain the image classification model.

[0067] It can be seen that the present application first divides the training image sample data into closed-set image sample data and open-set image sample data, wherein the closed-set image sample data includes K-class data of known categories, and the open-set image sample data includes the K-class data and unknown category data, and then extracts the features of the closed-set image sample data to obtain closed-set sample features, and based on the closed-set sample features and through unsupervised learning, determines similar class data in the K-class data, and divides similar class data into the same group to obtain M groups of data, and finally uses the M groups of data to train the target model structure of the image classification model, and during the training process, constructs batch data based on multiple groups of data in the M groups of data, and uses the open-set image sample data to train the first classification branch structure of the image classification model to obtain the trained image classification model; wherein, the target model structure includes a feature extraction structure and a second classification branch structure. That is, the present application uses closed-set image sample data and open-set image sample data to train an image classification model. When using closed-set image sample data to train the target model structure of the image classification model, it is based on the closed-set sample features and determines similar class data in K class data through unsupervised learning. Similar class data are divided into the same group to obtain M groups of data, and training is performed using M groups of data. The batch data used for training is constructed based on multiple groups of data in the M groups of data. In this way, data of known categories are grouped according to the unsupervised learning method, and the groups include similar class data and dissimilar class data between groups. Batch data is constructed using multiple groups of data to ensure that each batch of data during training contains data of the same category, data with similar categories but different categories, and data with dissimilar categories, so that the model can fully distinguish similar categories in known categories. In addition, the use of open-set image sample data also ensures that the image classification model can recognize unknown category data, thereby improving the performance of the image classification model and thus improving the accuracy of image classification.

[0068] In addition, the entire open set is reclassified using unsupervised learning methods. Known similar categories can be grouped into one category based on unsupervised learning, so that learning no longer focuses too much on detailed features. At the same time, known categories and similar unknown categories are grouped into the same group for training, which can fully distinguish between known categories and unknown categories.

[0069] See also Figure 2 As shown in the figure, an embodiment of the present application discloses an image classification method, including:

[0070] Step S21: Obtain the image to be classified.

[0071] Step S22: Input the image to be classified into the image classification model, so that the first classification branch structure and the second classification branch structure in the image classification model output the first classification result and the second classification result corresponding to the image to be classified; wherein, the image classification model is obtained by the image classification model training method disclosed in the foregoing embodiment.

[0072] Among them, the training method of the image classification model can refer to the content disclosed in the foregoing embodiment, and will not be elaborated herein.

[0073] Step S23: Determine the final classification result of the image to be classified based on the first classification result and the second classification result.

[0074] In a specific implementation manner, if the first classification result is the classification result corresponding to a known category, then the second classification result is determined as the final classification result; otherwise, the final classification result is determined as an unknown category.

[0075] In an implementation manner, during the training process, for the data of G categories among the K categories of data, corresponding class labels are generated, numbered 0, 1,..., G - 1; for the data of G categories among the unknown categories, corresponding class labels are generated, numbered G, G + 1,..., G - 1. The 0th category and the Gth category are grouped into the 0th group; the 1st category and the G + 1st category are grouped into the 1st group; and so on, the G - 1st category and the 2×G - 1st category are grouped into the G - 1st group. If the classification number (i.e., the classification result) output by the first classification branch structure < G, the final result is the classification result of the second classification branch structure; if the classification number output by the first classification branch structure ≥ G, the final result is an unknown category.

[0076] Furthermore, taking the commodity SKU image data as an example to further elaborate the solution provided by the present application, the commodity SKU classification is both a fine - grain classification and an open - set classification, which requires being able to distinguish similar commodities of the same brand or different brands; and also requires being able to identify commodities of various other brand parties. In particular, there are generally two processing methods for commodities of other brand parties. One is to process them as designated competing products, and the other is to process them as unknown commodities. When processing as unknown commodities, since there are various unknown commodities emerging in the real world, separating unknown commodities from brand - party commodities and designated competing products is a major problem in the industry. Through the solution provided by the present application, the problem of inaccurate fine - grain classification of open - set image data in the prior art can be effectively solved. See Figure 3As shown, an embodiment of the present application discloses a specific image classification method. Specifically, it includes the following steps:

[0077] (1) Mark the SKU image data according to requirements. The SKU image data includes K known classes (which may include brand-side SKUs and designated competitor SKUs) and a single unknown class. The reason for classifying the unknown class into only one class is that, on the one hand, it is convenient for data marking and reduces the marking workload; on the other hand, there may be a large number of unrecognized product SKUs in the unknown class, making it impossible to classify. Just because of this, the data of the unknown category is disorderly and chaotic, resulting in difficult classification and very low accuracy.

[0078] (2) Use a pre-trained classification model (such as ResNet or VGG, etc., not limited to this) or a self-designed classification model trained on similar data. Taking ResNet-18 as an example, it should be noted that ResNet-18 has a total of 17 convolutional layers and 1 fully connected layer. All the operations on the 1 / 2 scale of the input data are collectively referred to as the block1 layer, all the operations on the 1 / 4 scale of the input data are collectively referred to as the block2 layer, all the operations on the 1 / 8 scale of the input data are collectively referred to as the block3 layer, all the operations on the 1 / 16 scale of the input data are collectively referred to as the block4 layer, and all the operations on the 1 / 32 scale of the input data are collectively referred to as the block5 layer. After the output of the block5 layer, it is connected to the global average pooling layer, and the features of the Embedding layer are output. Finally, it is input to the fully connected layer for classification. The model obtained by using ResNet-18 on a large-scale dataset processes the data of the known class closed set to obtain the feature vectors of the data. Specifically, the output of the Embedding layer is used as the feature vector, and then an unsupervised learning method (such as the Kmeans clustering algorithm) is used for the feature vector. The K-class data is divided into N classes (N>1 and N<K). Then, for each class j in the K-class data, the data assigned to each class in the N classes is counted, and the class data j is marked as the class i with the most data in the N classes. The class data j is marked as the group data i; and so on, the K-class data can be divided into N groups. It should be emphasized that the group label i here is not the pseudo-label commonly used in other similar methods, but only the basis for grouping during training. In some scenarios, there may be groups in the N groups that do not have class data. This type of group can be excluded, and the K-class data can finally obtain M groups of data, M≤N, and N-M represents the number of groups without class data.

[0079] (3) Closed set data training: In step (2), the K-class data is divided into M groups. When training the data, in addition to expanding the data, randomly select t classes from any group of data (t>0, if t=0, it means that there is no data in this group and no training is required), select r data for each class (r is between 2 and 8, determined by the specific training category and training server, generally 4 can be taken), and randomly select s groups for each batch. Therefore, the training batch size is r×t×s. The network uses an open source pre-trained classification model (such as ResNet or VGG, etc.) or a self-designed classification model. Here, ResNet-18 is taken as an example. The output channel of the fully connected layer is modified to the known number of categories (K). The classification loss function uses Tiplet Loss and Equal Loss (or Softmax or other classification loss functions): Loss=Loss triplet +λ×Loss equal ; Among them, Loss triplet Tiplet Loss, Loss equal =(λ) / (λ) represents the Equal Loss, and λ is a constant. The reason for grouping the training data in this way is to ensure that each batch contains data of the same category, data of similar but different categories, and data of other categories. This ensures consistency within a class, and allows distinction between similar classes.

[0080] (4) Use the model trained in step (3) to process all the open-set data (including known and unknown classes) to obtain the eigenvalue of the data. Use the output of the Embedding layer as the eigenvalue. Next, use a clustering algorithm on the eigenvalue to divide the (K + 1) classes of data into G classes (G > 1 and G < K, the same below), or use other unsupervised learning methods to divide the (K + 1) classes of data into G classes. Then, for each class i in the K classes of data, the data assigned to the G classes are numbered 0, 1,..., G - 1; the unknown classes are numbered G, G + 1,..., 2×G - 1. All the data are divided into 2×G classes. Among them, the 0th class and the Gth class can be considered similar data and are grouped into the 0th group; the 1st class and the G + 1st class can be considered similar data and are grouped into the 1st group; and so on. The (G - 1)th class and the (2×G - 1)th class can be considered similar data and are grouped into the (G - 1)th group. In this way, the originally chaotic data of the unknown classes can be divided into G relatively ordered classes, and these G classes of unknown data can also correspond one by one with the known G groups of data, which is beneficial for subsequent model training. The reason for dividing the closed-set known data into G classes and the unknown data into G classes with corresponding relationships instead of dividing the unknown data into only 1 class is that the data of the unknown classes are disordered and chaotic, resulting in difficult classification and low accuracy. In addition, the reason for not dividing the unknown data into the same number of classes as the original number of classes of the closed-set data is that when the fine-grained feature extraction model of the known classes processes the unknown data, the obtained features are too chaotic, resulting in poor learning of the final unknown classes.

[0081] (5) Open-set data training: Expand the 2×G classes of data (a total of G groups) in step (4). Randomly select x classes of rows from any group of data (x > 0. If x = 0, it means that there is no data in this group and there is no need to train). Select y data for each class (y is between 2 and 8, determined according to the specific training classes and the training server, and generally 4 can be taken). Randomly select z groups for each batch. Therefore, the training batch size is x×y×z. When training, the input data is the feature of 1 / 4 scale or 1 / 8 scale or 1 / 16 scale output by the network structure trained in step (3) or the combined feature of them. In ResNet-18, the output of block2 layer or block3 output or block4 layer output or the comprehensive processing result of combining the output of block2 layer, block3 layer, and block4 layer can be used. Continue to design subsequent models similar to block5 layer and fully connected layer in ResNet-18. The classification loss function uses Tiplet Loss and Softmax Loss (or other classification loss functions): Loss = Loss triplet +β×Loss softmax ; where Loss triplet represents Tiplet Loss, Losssoftmax It represents Softmax Loss, and β is a constant. The reason why the intermediate features output by the network structure trained according to step (3) can be used for training is that the closed-set data is a part of the entire open-set data, and the feature extraction of this part is also credible for the entire open-set data set. In addition, the basis for classifying unknown categories in the closed set is also based on the model obtained from the network structure trained in step (3) of the open-set data training. Therefore, it can be considered that the intermediate features output by the network structure in step (3) are effective for the entire open-set data set.

[0082] (6) Online deployment: Combine the models obtained in steps (3) and (5) to get a comprehensive model with two branches. One branch is for fine-grained classification, and the other branch is for classifying known and unknown categories. Combine the results of these two branches to obtain the final classification result. If the classification number <G obtained by the known and unknown category classification branches, the final result will output the specific SKU of the known category. If the classification number obtained by the known and unknown category classification branches ≥G, the final result will output the unknown category.

[0083] It should be pointed out that most of the existing fine-grained classification methods only care about the common intra-class similarities and common inter-class distinctions during training. The embodiments of the present application group known categories according to the unsupervised learning method, and use group training to learn the known categories of closed-set data. Each training session pays attention to the common intra-class similarities, common inter-class distinctions, and similar inter-class distinctions. Each training session fully learns the data features of these three methods, which can fully distinguish similar categories in known categories while reducing intra-class differences. In most of the existing open-set classifications, the unknown categories in the open-set data are either too cluttered to be classified during training; or the pseudo-label classification is poor, and there are situations where the pseudo-label categories are indistinguishable from the known categories, or the pseudo-label categories are too detailed to learn any features at all. During training, the present application makes full use of the features of known data, comprehensively classifies known category data and unknown category data, fully distinguishes known categories from similar data of unknown categories, and uses unsupervised learning methods to reclassify known categories to obtain a number of categories less than the total number of original labels (the specific categories can be determined according to actual conditions), thereby reducing the learning cost of known categories and preventing the fine-grained feature extraction model of known categories from processing unknown data. The features obtained are too messy, resulting in poor learning of the final unknown categories. When the unsupervised learning method is used to reclassify the entire open set, similar categories of known fine-grained categories can be classified into one category according to unsupervised learning, so that learning no longer pays too much attention to detailed features. At the same time, known categories and similar unknown categories are classified into the same group for training, which can fully distinguish known categories from unknown categories. Finally, the models trained twice are merged to obtain a comprehensive model with two branches, wherein one branch performs fine-grained classification and the other branch performs classification of known categories and unknown categories. The results of these two branches are combined to obtain the final classification result.

[0084] See also Figure 4 As shown, the embodiment of the present application discloses an image classification model training device, comprising:

[0085] An image sample data division module 11 is configured to divide the training image sample data into closed-set image sample data and open-set image sample data; the closed-set image sample data includes K-class data of known categories, and the open-set image sample data includes the K-class data and unknown-class data;

[0086] A closed set sample feature extraction module 12 is used to extract features of the closed set image sample data to obtain closed set sample features;

[0087] A closed set sample data grouping module 13 is configured to determine similar class data in the K class data based on the closed set sample features by unsupervised learning, and group the similar class data into the same group to obtain M groups of data;

[0088] The image classification model training module 14 is used to train the target model structure of the image classification model using the M groups of data, and during the training process, construct batch data based on multiple groups of data in the M groups of data, and use the open set image sample data to train the first classification branch structure of the image classification model to obtain a trained image classification model; wherein, the target model structure includes a feature extraction structure and a second classification branch structure.

[0089] It can be seen that the present application uses closed-set image sample data and open-set image sample data to train an image classification model. When using closed-set image sample data to train the target model structure of the image classification model, it is based on the closed-set sample features and uses unsupervised learning to determine similar class data in K class data, and the similar class data are divided into the same group to obtain M groups of data. The M groups of data are used for training, and the batch data used for training is constructed based on multiple groups of data in the M groups of data. In this way, data of known categories are grouped according to the unsupervised learning method, and the groups include similar class data, and the groups include dissimilar class data. Batch data is constructed using multiple groups of data to ensure that each batch of data contains both data of the same category, data with similar categories but different categories, and data with dissimilar categories during training, so that the model can fully distinguish similar categories in known categories, and the use of open-set image sample data also ensures that the image classification model can recognize unknown category data, thereby improving the performance of the image classification model and thus improving the accuracy of image classification.

[0090] Among them, the closed set sample data grouping module 13 is specifically used to: classify the closed set sample features through unsupervised learning to obtain N categories; for any category of data in the K categories of data, determine the category with the largest amount of data in the N categories to obtain the target category corresponding to the category of data, and divide the category data corresponding to the same target category in the K categories of data into the same group to obtain M groups of data.

[0091] The image classification model training module 14 is specifically used to select multiple groups from the M groups of data, select multiple classes from the multiple groups, and select multiple data from the multiple classes to construct a batch of data.

[0092] The image classification model training module 14 specifically includes:

[0093] an open set image sample data classification unit, configured to classify the open set image sample data into G categories through unsupervised learning;

[0094] an open set image sample data grouping unit, configured to group the data in the K class data and the data in the unknown class data that belong to the same class in the G class into the same group, to obtain G group data;

[0095] The first classification branch structure training unit is used to train the first classification branch structure of the image classification model using the G group of data.

[0096] Furthermore, the open set image sample data grouping unit is specifically used to generate a first category label for the data in the K category data; the first category label represents the corresponding category in the G category to which the data belongs; generate a second category label for the data in the unknown category data; the first category label represents the corresponding category in the G category to which the data belongs; and the first category label and the second category label do not have the same label; the data in the K category data and the data in the unknown category data belonging to the same category in the G category are divided into the same group to obtain G group data.

[0097] The image classification model training module 14 is specifically used to: input the open set image sample data into the trained feature extraction structure to obtain open set sample features, and train the first classification branch structure of the image classification model based on the open set sample features.

[0098] See also Figure 5 As shown, an embodiment of the present application discloses an electronic device 20, including a processor 21 and a memory 22; wherein the memory 22 is used to store a computer program; the processor 21 is used to execute the computer program, the image classification model training method disclosed in the aforementioned embodiment, and / or the image classification method.

[0099] Regarding the above-mentioned image classification model training method, and / or the specific process of the image classification method, reference can be made to the corresponding content disclosed in the aforementioned embodiments, which will not be repeated here.

[0100] Furthermore, the memory 22 as a carrier for resource storage may be a read-only memory, a random access memory, a magnetic disk or an optical disk, etc., and the storage method may be temporary storage or permanent storage.

[0101] In addition, the electronic device 20 also includes a power supply 23, a communication interface 24, an input / output interface 25 and a communication bus 26; wherein the power supply 23 is used to provide an operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and an external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input / output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0102] Furthermore, an embodiment of the present application also discloses a computer-readable storage medium for storing a computer program, wherein, when the computer program is executed by a processor, it implements the image classification model training method and / or the image classification method disclosed in the aforementioned embodiment.

[0103] Regarding the above-mentioned image classification model training method, and / or the specific process of the image classification method, reference can be made to the corresponding content disclosed in the aforementioned embodiments, which will not be repeated here.

[0104] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. Reference can be made to the descriptions of the identical or similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and the relevant parts can be referred to the descriptions of the methods.

[0105] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0106] The above is a detailed introduction to the image classification model training method and device, image classification method, and equipment provided by this application. Specific examples are used in this article to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method of this application and its core idea; at the same time, for general technical personnel in this field, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on this application.

Claims

1. A method for training an image classification model, characterized in that: include: The training image sample data is divided into closed set image sample data and open set image sample data; the closed set image sample data includes K class data of known categories, and the open set image sample data includes the K class data and unknown class data; Extracting features of the closed set image sample data to obtain closed set sample features; Determine similar class data in the K class data by unsupervised learning based on the closed set sample features, and group the similar class data into the same group to obtain M groups of data; The target model structure of the image classification model is trained using the M sets of data, and during the training process, batch data is constructed based on multiple sets of data in the M sets of data, and the first classification branch structure of the image classification model is trained using the open set image sample data to obtain a trained image classification model; wherein the target model structure includes a feature extraction structure and a second classification branch structure; The step of training the first classification branch structure of the image classification model using the open set image sample data includes: Classifying the open set image sample data through unsupervised learning to obtain G categories; Generate a first class label for the data in the K class data; the first class label represents the corresponding class in the G class to which the data belongs; Generate a second class label for the data in the unknown class data; the second class label represents the corresponding class in the G classes to which the data belongs; and the first class label and the second class label do not have the same label; Classify the data in the K-category data and the data in the unknown category data that belong to the same category in the G-category data into the same group to obtain G-group data; The G group of data is used to train the first classification branch structure of the image classification model.

2. The image classification model training method according to claim 1, characterized in that: The method of determining similar class data in the K class data based on the closed set sample features by unsupervised learning, and grouping the similar class data into the same group to obtain M groups of data, includes: Classifying the closed set sample features through unsupervised learning to obtain N categories; For any category of data in the K categories of data, determine the category with the largest amount of data in the N categories to obtain the target category corresponding to the category of data, and divide the category data corresponding to the same target category in the K categories of data into the same group to obtain M groups of data.

3. The image classification model training method according to claim 2, characterized in that: During the training process, batch data is constructed based on multiple groups of data in the M groups of data, including: A plurality of groups are selected from the M groups of data, a plurality of classes are selected from the plurality of groups, and a plurality of data are selected from the plurality of classes to construct a batch of data.

4. The image classification model training method according to any one of claims 1 to 3, characterized in that: The training of the first classification branch structure of the image classification model using the open set image sample data includes: The open set image sample data is input into the trained feature extraction structure to obtain open set sample features, and the first classification branch structure of the image classification model is trained based on the open set sample features.

5. An image classification method, characterized in that: include: Get the image to be classified; Inputting the image to be classified into the image classification model, so that the first classification branch structure and the second classification branch structure in the image classification model output a first classification result and a second classification result corresponding to the image to be classified; wherein the image classification model is obtained based on the image classification model training method according to any one of claims 1 to 4; A final classification result of the image to be classified is determined based on the first classification result and the second classification result.

6. An image classification model training device, characterized in that: include: An image sample data division module is used to divide the training image sample data into closed set image sample data and open set image sample data; the closed set image sample data includes K class data of known categories, and the open set image sample data includes the K class data and unknown class data; A closed set sample feature extraction module, configured to extract features of the closed set image sample data to obtain closed set sample features; A closed set sample data grouping module is used to determine similar class data in the K class data based on the closed set sample features through unsupervised learning, and group the similar class data into the same group to obtain M groups of data; an image classification model training module, configured to train a target model structure of an image classification model using the M sets of data, construct batch data based on multiple sets of data in the M sets of data during the training process, and train a first classification branch structure of the image classification model using the open set image sample data to obtain a trained image classification model; wherein the target model structure includes a feature extraction structure and a second classification branch structure; Among them, the use of the open set image sample data to train the first classification branch structure of the image classification model includes: classifying the open set image sample data through unsupervised learning to obtain G categories; generating a first category label for the data in the K category data; the first category label represents the corresponding category in the G category to which the data belongs; generating a second category label for the data in the unknown category data; the second category label represents the corresponding category in the G category to which the data belongs; and the first category label and the second category label do not have the same label; the data in the K category data and the data in the unknown category data belonging to the same category in the G category are divided into the same group to obtain G group data; and the first classification branch structure of the image classification model is trained using the G group data.

7. An electronic device, characterized in that: comprising a memory and a processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program to implement the image classification model training method according to any one of claims 1 to 4, and / or the image classification method according to claim 5.

8. A computer-readable storage medium, characterized in that Used to store a computer program, wherein when the computer program is executed by a processor, it implements the image classification model training method according to any one of claims 1 to 4, and / or the image classification method according to claim 5.

Images