Image classification method and device, electronic equipment, medium and product
By acquiring and iterating the training of the same type of sample data sets of the image classification model, the problem of forgetting old knowledge on the dynamic data sets is solved, and the classification accuracy and learning effect are improved.
Patent Information
- Application Number
- CN202410990280.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-23
- Publication Date
- 2024-11-12
AI Technical Summary
When facing dynamic data sets, existing image classification models are difficult to effectively learn new knowledge without forgetting old knowledge, resulting in a decrease in classification accuracy, especially when new and old data are unbalanced.
By obtaining the same and different numbers of samples under the same type for model training, the model is iteratively trained by combining data sets to reduce the forgetting rate of old knowledge and improve model performance. The specific methods include obtaining the first and second training sets, determining the third training set based on the type tag, merging and sampling the training set, calculating cross entropy and distillation loss parameters, and iterating the model to generate the fusion loss parameters until the preset requirements are met.
It improves the accuracy of the image classification model on dynamic data sets, reduces the forgetting rate of old knowledge when learning new content, and improves the learning effect of the model in the imbalance between old and new data.
Smart Images

Figure CN118941859A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of image processing, and in particular to an image classification method, device, electronic equipment, medium and product. Background Art
[0002] In recent years, thanks to deep convolutional neural networks and huge amounts of training data, research on image classification has made significant progress. Models have outperformed humans on large datasets (such as ImageNet), and researchers have begun to focus on more difficult image classification scenarios, such as incremental learning. The training of general models is to prepare all the data in advance, and the data is fixed and static. However, in real life, all walks of life on the Internet will generate a large amount of new data. Obviously, this method of training models based solely on static data is no longer applicable. Therefore, a method is needed that allows the model to continuously learn new knowledge and try not to forget the old knowledge learned previously, so as to improve the accuracy of image classification in the case of dynamic data. Summary of the invention
[0003] The present disclosure provides an image classification method, device, electronic device, medium and product to solve the problems in the related art. By using the same and different numbers of samples of the same type to train the model, the learning effect of the model when the new and old training data are unbalanced is improved. The model is trained by combining data sets to reduce the model's forgetting rate of old knowledge when learning new content, thereby improving the model performance of the trained image classification model and improving the accuracy of image classification for dynamic data sets.
[0004] A first aspect of the present disclosure provides an image classification method, comprising: obtaining a first training set and a second training set, wherein image samples in the first training set and the second training set have type labels, and the number of image samples of each type in the first training set is the same; performing image classification training on a model to be trained using the first training set to obtain the first training model; determining a third training set from the second training set based on the type labels of the image samples in the first training set, wherein the types of the image samples in the third training set are different from the types of the image samples in the first training set, and the number of samples of each type in the third training set is the same as the number of image samples of each type in the first training set; iterating the first training model based on the first training set, the second training set, and the third training set to obtain an image classification model; and classifying the image to be classified using the image classification model.
[0005] In some embodiments of the present disclosure, based on the first training set, the second training set and the third training set, iterating the first training model to obtain an image classification model includes: removing image samples in the third training set from the second training set to obtain a fourth training set; merging the first training set and the third training set to obtain a first fusion training set; merging the first training set and the fourth training set to obtain a second fusion training set; sampling the first fusion training set and the second fusion training set to obtain a sampled training set; based on the first fusion training set, the second fusion training set and the sampled training set, iterating the first training model to obtain an image classification model.
[0006] In some embodiments of the present disclosure, based on the first fusion training set, the second fusion training set and the sampling training set, the first training model is iterated to obtain an image classification model, including: based on the sampling training set, using the first training model, determining a cross entropy loss parameter; based on the sampling training set, using the model to be trained and the first training model, determining a distillation loss parameter; based on the cross entropy loss parameter and the distillation loss parameter, determining a fusion loss parameter; using the fusion loss parameter, iterating the first training model to obtain an image classification model.
[0007] In some embodiments of the present disclosure, based on the sampled training set, using the first training model, determining the cross entropy loss parameter includes: using the first training model to classify and score the image samples in the sampled training set that belong to the first fusion training set to obtain a first classification score; using the first training model to classify and score the image samples in the sampled training set that belong to the second fusion training set to obtain a second classification score; determining the cross entropy loss parameter based on the first sampled training set, the first classification score and the second classification score.
[0008] In some embodiments of the present disclosure, based on the sampled training set, using the model to be trained and the first training model, determining the distillation loss parameter includes: using the model to be trained, classifying and scoring the image samples in the sampled training set to obtain a third classification score; using the first training model, classifying and scoring the image samples in the sampled training set to obtain a fourth classification score; and determining the distillation loss parameter based on the third classification score and the fourth classification score.
[0009] In some embodiments of the present disclosure, using a fusion loss parameter to iterate the first training model to obtain an image classification model includes: iterating the first training model according to a preset step size; and determining the model corresponding to when the fusion loss parameter meets the preset requirements as the image classification model.
[0010] A second aspect of the present disclosure provides an image classification device, which includes: an acquisition unit, used to acquire a first training set and a second training set, the image samples in the first training set and the second training set have type labels, and the number of image samples of each type in the first training set is the same; a first training unit, used to use the first training set to perform image classification training on a model to be trained, so as to obtain the first training model; a determination unit, used to determine a third training set from the second training set based on the type labels of the image samples in the first training set, the types of the image samples in the third training set are different from the types of the image samples in the first training set, and the number of samples of each type in the third training set is the same as the number of image samples of each type in the first training set; a second training unit, used to iterate the first training model based on the first training set, the second training set and the third training set, so as to obtain an image classification model; and a classification unit, used to classify the image to be classified using the image classification model.
[0011] The third aspect embodiment of the present disclosure proposes an electronic device, comprising: a processor and a memory for storing a computer program that can be run on the processor, wherein the processor, when used to run the computer program, executes the method described in the first aspect embodiment of the present disclosure.
[0012] The fourth aspect embodiment of the present disclosure proposes a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the method described in the first aspect embodiment of the present disclosure.
[0013] The fifth aspect embodiment of the present disclosure provides a computer program product, including a computer program, which implements the method described in the first aspect embodiment of the present disclosure when executed by a processor.
[0014] In summary, the image classification method proposed in the present disclosure includes: obtaining a first training set and a second training set, the image samples in the first training set and the second training set have type labels, and the number of image samples of each type in the first training set is the same; using the first training set to perform image classification training on the training model to obtain the first training model; based on the type labels of the image samples in the first training set, determining a third training set from the second training set, the type of the image samples in the third training set is different from the type of the image samples in the first training set, and the number of samples of each type in the third training set is the same as the number of image samples of each type in the first training set; iterating the first training model based on the first training set, the second training set and the third training set to obtain an image classification model; using the image classification model, classifying the images to be classified. This method uses the same and different numbers of samples of the same type for model training, improves the learning effect of the model when the new and old training data are unbalanced, trains the model by combining data sets, reduces the model's forgetting rate of old knowledge when learning new content, thereby improving the model performance of the trained image classification model and improving the accuracy of image classification for dynamic data sets. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.
[0016] Figure 1 An example flow chart of an image classification training method provided in an embodiment of the present disclosure;
[0017] Figure 2 A flowchart of an image classification model training method provided in an embodiment of the present disclosure;
[0018] Figure 3 A flowchart of another image classification method provided by an embodiment of the present disclosure;
[0019] Figure 4 A flowchart of another image classification method provided by an embodiment of the present disclosure;
[0020] Figure 5 An example diagram of an image classification model training method provided in an embodiment of the present disclosure;
[0021] Figure 6 A schematic diagram of the structure of an image classification device provided by an embodiment of the present disclosure;
[0022] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION
[0023] The embodiments of the present disclosure are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions.
[0024] In recent years, thanks to deep convolutional neural networks and huge amounts of training data, image classification research has made significant progress. Models have outperformed humans on large datasets (such as ImageNet), and researchers have begun to focus on more difficult image classification scenarios, such as incremental learning. The training of general models requires all data to be prepared in advance, and the data is fixed and static. However, in real life, all walks of life on the Internet will generate a large amount of new data. Obviously, this method of training models based solely on static data is no longer applicable. The content of incremental learning research is to face a steady stream of new data, so that the model can continuously learn new knowledge and try not to forget the old knowledge learned previously, that is, how to effectively help the model alleviate catastrophic forgetting that occurs in the process of incremental category learning. The process of incremental learning is like Figure 1 As shown in the figure, in each task, the data in the current task is called new data, and the data that appeared in all previous tasks are collectively called old data. Note that the test data of each task contains all the seen classes. Incremental learning can be divided into task incremental learning and category incremental learning according to whether the sample has a task label during testing (that is, it represents which task the sample appears in). This scheme only discusses category incremental learning (that is, the sample does not have a task label during testing). Since category incremental learning is more difficult, it is also the focus of current research.
[0025] The methods of incremental learning in image classification problems are mainly divided into replay-based, regularization-based, parameter isolation-based and knowledge distillation-based. Replay-based means that when training a model in a new task, a small amount of old samples are added to the training to ensure that the model does not forget the old knowledge. Regularization-based methods usually impose restrictions on the update of each model parameter so as to consolidate the previously learned knowledge while learning the new task, thereby reducing catastrophic forgetting in incremental learning. Parameter isolation-based methods usually assign different model parameters to different tasks to prevent subsequent tasks from interfering with previously learned knowledge. Knowledge distillation-based means that in a new task, the old model is used as a teacher and the new model as a student to minimize the KL divergence between the outputs of the two models, so that the old model transfers the old knowledge to the new model. Many works are based on combined methods, such as replay-based and knowledge distillation, and their performance is also better than all methods. Although replay can help the model resist catastrophic forgetting, the imbalance in the number of new and old data will cause the model to have prediction bias. Therefore, a method is needed that allows the model to continuously learn new knowledge and try not to forget the old knowledge learned previously, so as to improve the accuracy of image classification in the case of dynamic data.
[0026] The method proposed in the present invention improves the learning effect of the model when the new and old training data are unbalanced by using the same and different numbers of samples of the same type for model training, and reduces the model's forgetting rate of old knowledge when learning new content by training the model through combining data sets, thereby improving the model performance of the trained image classification model and improving the accuracy of image classification for dynamic data sets.
[0027] The image classification method provided in this application is described in detail below with reference to the accompanying drawings.
[0028] Figure 2 Flow chart of an image classification method provided by an embodiment of the present disclosure. Figure 2 As shown, the image classification method includes steps 101-105.
[0029] Step 101: Obtain a first training set and a second training set.
[0030] In some embodiments, the first training set may be data used for the first training of the model to be trained, and the second training data may be data used for training the model obtained after the first training. In other words, the first training set is old data used for model training, and the second training set is new data used for model training.
[0031] In some embodiments, the image samples in the first training set and the second training set have type labels, wherein the sample labels are used to indicate the types of the image samples.
[0032] In some embodiments, the number of image samples of each type in the first training set is the same, that is, the first training set is a class-balanced data set, and balanced training can be performed on the training model.
[0033] In some embodiments, the number of image samples of each type in the second training set may be the same or different, which is not limited in this disclosure.
[0034] It should be understood that there may be no intersection between the image samples in the first training set and the second training set, so that the model to be trained can achieve category incremental learning.
[0035] Step 102: Perform image classification training on the model to be trained using the first training set to obtain a first training model.
[0036] In some embodiments, the first training set is used to perform image classification training on the training model, that is, the training model is trained for the first time. It should be understood that the present disclosure does not limit the first training method, and it can be, for example, SVM neural network, BP neural network, etc. to realize image classification training.
[0037] Step 103: Determine a third training set from the second training set based on the type labels of the image samples in the first training set.
[0038] In some embodiments, the type of image samples in the third training set is different from the type of image samples in the first training set, and the number of samples of each type in the third training set is the same as the number of image samples of each type in the first training set, so as to combine the first training set with the third training set to form a new category-balanced data set, thereby performing balanced training on the first training model.
[0039] Step 104, iterating the first training model based on the first training set, the second training set and the third training set to obtain an image classification model.
[0040] In some embodiments, based on the first training set, the second training set and the third training set, iterating the first training model to obtain an image classification model may include the following steps: removing image samples in the third training set from the second training set to obtain a fourth training set; merging the first training set and the third training set to obtain a first fusion training set; merging the first training set and the fourth training set to obtain a second fusion training set; sampling the first fusion training set and the second fusion training set to obtain a sampled training set; using the first fusion training set and the sampled training set to perform category balanced training on the first training model, and using the second fusion training set and the sampled training set to perform category imbalance training on the first training set, thereby iterating the first training model to obtain an image classification model.
[0041] For details, please refer to Figure 2 The embodiments shown will not be described in detail here.
[0042] Step 105: classify the image to be classified using the image classification model.
[0043] In some embodiments, the image to be classified may be input into an image classification model, and the image classification model may score the degree of belonging of the image to be classified to each category, and classify the image to be classified into the category with the highest score.
[0044] For example, if image 1 scores 5 points in type A and 7 points in type B, the image classification model will classify image 1 into type B.
[0045] In summary, the image classification method proposed in the present disclosure includes: obtaining a network security situation data set; based on the network security situation data set, using the triangular oscillation diffusion function in the pre-trained model, determining the output parameter set of the neural network gating unit in the pre-trained model; based on the output parameter set, determining the hidden layer state parameters in the pre-trained model; based on the hidden layer state parameters, determining the image classification result. The method disclosed in the present disclosure utilizes the triangular oscillation diffusion function in the pre-trained model, thereby alleviating the problems of gradient vanishing, gradient explosion, neuron necrosis, etc. generated when the model is used, and improving the convergence speed of the model and the accuracy of image classification.
[0046] Figure 3 A flowchart of an image classification method proposed in the present disclosure is further shown. Figure 2 Step 104 is further explained based on the embodiment shown. Figure 3 As shown, the method comprises the following steps:
[0047] Step 201: remove image samples in the third training set from the second training set to obtain a fourth training set.
[0048] In some embodiments, the fourth training set is an image sample set consisting of image samples in the second training set other than the image samples in the third training set.
[0049] In other words, the fourth training set may be a class-imbalanced image sample set.
[0050] Step 202: merge the first training set and the third training set to obtain a first fusion training set.
[0051] In some embodiments, by merging image samples in the first training set and image samples in the third training set, a category-balanced training set (i.e., a first fused training set) containing new and old training samples is generated, so as to facilitate the subsequent calculation of the cross-loss entropy of the first training model through the first fused training set, thereby implementing constraints on the model and avoiding prediction bias.
[0052] Step 203: merge the first training set and the fourth training set to obtain a second fusion training set.
[0053] In some embodiments, by merging image samples in the first training set and image samples in the fourth training set, since the number of image samples of each type in the fourth training set is different, a class-imbalanced training set (i.e., a second fusion training set) containing new and old training samples can be generated. Since the second fusion training set contains image samples in the first training set, when the first training model is trained using the second fusion training set, the first training model can learn discriminant row features that distinguish new data (second fusion training set) from old data (first training set).
[0054] Step 204: Sample the first fusion training set and the second fusion training set to obtain a sampled training set.
[0055] In some embodiments, since there are a large number of image samples in the first fusion training set and the second fusion training set, it is necessary to sample the above training sets to facilitate the model's processing of the image samples; and in order to prevent the model from forgetting old training results when performing new training, the image samples obtained by sampling the first fusion training set and the second fusion training set can be determined as a training set, namely, a sampling training set, for training the first training model.
[0056] In some embodiments, the sampling ratio is not limited. For example, the first fusion training set and the second fusion training set may be sampled in a 1:1 ratio, but the present invention is not limited thereto.
[0057] It should be understood that multiple samplings may be performed, that is, the first training model may be trained multiple times, thereby using all the image samples in the first fusion training set and the second fusion training set to train the first training model.
[0058] Step 204, iterating the first training model based on the first fusion training set, the second fusion training set and the sampling training set to obtain an image classification model.
[0059] In some embodiments, by iteratively training the first training model using a sampling training set, it should be understood that when the image samples in the sampling training set used in multiple trainings fully cover the image samples in the first fusion training set and the second fusion training set, the iterative training is considered to be completed.
[0060] In some embodiments, the fusion loss parameters of the model can be determined by using the first fusion training set, the second fusion training set and the sampling training set, and then whether the model still needs iterative training can be determined based on whether the fusion loss parameters of the model meet the preset requirements. When the fusion loss parameters of the model do not meet the requirements, the model can be further iteratively trained using a preset step size until the fusion loss parameters of the model meet the preset requirements, and the model corresponding to the fusion loss parameters meeting the preset requirements is determined as an image classification model.
[0061] In summary, the image classification method disclosed in the present invention includes: removing the image samples in the third training set from the second training set to obtain the fourth training set; merging the first training set and the third training set to obtain the first fusion training set; merging the first training set and the fourth training set to obtain the second fusion training set; sampling the first fusion training set and the second fusion training set to obtain the sampling training set; iterating the first training model based on the first fusion training set, the second fusion training set and the sampling training set to obtain the image classification model. The method disclosed in the present invention reorganizes the new and old training sets to generate a class-balanced training set (i.e., the first fusion training set) and a class-unbalanced training set (i.e., the second fusion training set) containing new and old training samples, so as to facilitate the subsequent calculation of the cross loss entropy of the first training model through the first fusion training set, realize the constraint of the model, avoid prediction bias, and at the same time, when the first training model is trained using the second fusion training set, the first training model can learn the discriminant row features that distinguish new data (the second fusion training set) from old data (the first training set).
[0062] Figure 4 A flowchart of an image classification method proposed in the present disclosure is further shown. Figure 3 Step 205 is further explained based on the embodiment shown. Figure 4 As shown, the method comprises the following steps:
[0063] Step 301, based on the sampled training set, using the first training model, determine the cross entropy loss parameter.
[0064] In some embodiments, the first training model can be used to classify and score image samples in the sampling training set that belong to the first fusion training set to obtain a first classification score; the first training model can be used to classify and score image samples in the sampling training set that belong to the second fusion training set to obtain a second classification score; and the cross entropy loss parameter is determined based on the first sampling training set, the first classification score, and the second classification score.
[0065] In some embodiments, the cross entropy loss parameter may be determined by the following formula:
[0066]
[0067] in, e represents the base of natural numbers, is the indicator function. dss is the abbreviation of discriminative softmax. represents the image samples in the sampling data set belonging to the first fusion training set, represents the image samples in the sampling data set belonging to the second fusion training set, for The label space of the image samples in , for The label space of the image samples in , represents the first category score, Indicates the second category score.
[0068] Step 302: Based on the sampled training set, using the model to be trained and the first training model, determine the distillation loss parameter.
[0069] In some embodiments, the model to be trained can be used to classify and score the image samples in the sampled training set to obtain a third classification score; the first training model can be used to classify and score the image samples in the sampled training set to obtain a fourth classification score; and then based on the third classification score and the fourth classification score, the distillation loss parameter is determined.
[0070] In some embodiments, the distillation loss parameter may be determined by the following formula:
[0071]
[0072] in, τ is the temperature scalar. gkd is the abbreviation of general knowledge distillation. represents the third category score, and o represents the fourth category score.
[0073] Step 303: Determine a fusion loss parameter based on the cross entropy loss parameter and the distillation loss parameter.
[0074] In some embodiments, the fusion loss parameter may be determined by the following formula:
[0075]
[0076] Where L represents the fusion loss parameter, and the scalar λ is used to help the model balance the two losses. In an optional embodiment, the λ value can be set to In this way, the weight relationship between the classification loss and the distillation loss is balanced, but not limited to this, and the present disclosure does not limit the value of λ.
[0077] Step 304, iterating the first training model according to a preset step size.
[0078] In some embodiments, when the calculated fusion loss parameter of the first model does not meet the preset requirements, it means that the classification accuracy of the first training model is low, so the first training model can be iterated according to the preset step size.
[0079] Step 305: determine the model corresponding to when the fusion loss parameter meets the preset requirement as the image classification model.
[0080] In some embodiments, when the fusion loss parameters of the iterated model meet the preset requirements, the model corresponding to the fusion loss parameters that meet the requirements can be determined as the image classification model.
[0081] In summary, the image classification method disclosed in the present invention includes: determining the loss parameter based on the sampled training set and using the first training model; determining the distillation loss parameter based on the sampled training set and using the model to be trained and the first training model; determining the fusion loss parameter based on the cross entropy loss parameter and the distillation loss parameter; iterating the first training model according to a preset step size; determining the model corresponding to the fusion loss parameter when the fusion loss parameter meets the preset requirement as the image classification model. The method disclosed in the present invention generates a fusion loss parameter from the cross entropy loss parameter and the distillation loss parameter to evaluate whether the model meets expectations, and can more accurately evaluate the classification accuracy of the model.
[0082] The following is an exemplary description of the disclosed method.
[0083] The core idea of the disclosed method is to construct a discriminative separation softmax and use the overall distillation loss. The scheme is divided into 4 parts: the first part is problem definition, the second part is data reorganization, the third part is loss calculation, and the fourth part is model reasoning.
[0084] 1. Ask for definitions
[0085] In category incremental learning, there is a sequence of T tasks In the tth task, the model obtains a new training data set in is the input image, is the corresponding label. There is no intersection between the category sets of any two tasks, that is, After the model training is completed in the current task, some samples will be saved to the memory, which can be represented as Where m is the memory size and remains constant in all tasks. The sample selection principle refers to the reference iCaRL [2], that is, selecting those samples that are closest to their class center. Therefore, the available training data in each task is Here will The number of new categories in is expressed as The number of old categories in is expressed as in The goal of training the model is to ) can complete the prediction well.
[0086] 2. Data Reorganization
[0087] In order to solve the problems existing in the separation softmax, this proposal proposes a discriminative separation softmax to maintain the discriminability between new data and old data, while enabling the model to avoid prediction bias. Specifically, the discriminability is added in two steps. The whole process is as follows Figure 5 As shown. The first step is to convert the new data Divided into two parts, namely in Contains all new categories, and the number of samples of each new category and The number of samples of each old category in is equal. Therefore, we can and merged to form a class-balanced dataset and denoted as From the balanced dataset Sampling from the dataset and calculating the cross entropy loss on the logits of all seen categories can not only provide the discriminability between new and old data, but also constrain the model to avoid the problem of prediction bias. After the samples in the dataset are input into the model, the logits output on all seen categories can be expressed as:
[0088]
[0089] Where o(x) represents the score of the input image x in a certain category, its subscript represents the corresponding category, and the superscript bal is the abbreviation of balance. Copy it, then and Merge to get a class-imbalanced dataset and represent it as Although it has Divided into two parts, The number of samples in is still much larger than If we directly calculate the cross entropy loss on the logits of all seen categories, there will still be a problem of prediction bias due to the imbalance in the number of samples of the new and old categories. To solve this problem, we treat all the old data as one class, so that we get a Therefore, in the fully connected layer of the model, an auxiliary branch is added, that is, an output unit is added to the fully connected layer to help the model classify new and old data. In this way, the model can learn the discriminative features to distinguish new data from old data. Note that this auxiliary branch is discarded during testing. After the samples in the dataset are input into the model, the logits output on all new categories and auxiliary branches can be expressed as:
[0090]
[0091] The superscript dis is short for discrimination. Since the logits output by the model are divided, and The label space is also divided. The label space in the dataset is The label space of the dataset is given by becomes The labels of the old data are
[0092] 3. Loss calculation
[0093] When training the model, and In a certain proportion, a mini-batch is formed. Then the modified cross entropy loss function can be expressed as:
[0094]
[0095] in e represents the base of natural numbers, is the indicator function. dss is the abbreviation of discriminative softmax.
[0096] In addition, knowledge distillation can transfer key knowledge of the old model to the new model, which is a common method to help the model overcome catastrophic forgetting. Here, the output of the old model is represented as Then the overall distillation loss can be expressed as:
[0097]
[0098] in τ is the temperature scalar. gkd is the abbreviation of general knowledge distillation.
[0099] Then the overall loss is the combination of classification loss and overall distillation loss, which can be expressed as:
[0100]
[0101] The scalar λ can help the model balance these two losses. Here, the value is set to In this way, the weight relationship between classification loss and distillation loss is balanced.
[0102] 4. Model Reasoning
[0103] After training the model using the above data reorganization and modified cross entropy loss, the model can make good predictions for all seen categories (all categories that appeared in the previous task). This method is applicable to all types of recognition, whether the task is animals, buildings, or vehicles, the trained model has good reasoning ability, that is, this method has good versatility.
[0104] Corresponding to the methods provided in the above-mentioned embodiments, the present disclosure also provides an image classification device. Since the device provided in the embodiment of the present disclosure corresponds to the methods provided in the above-mentioned embodiments, the implementation method is also applicable to the device provided in this embodiment and will not be described in detail in this embodiment.
[0105] Figure 6 FIG. 4 is a schematic diagram of the structure of an image classification device 400 provided in an embodiment of the present disclosure. Figure 6 As shown, the image classification device includes:
[0106] An acquisition unit 410 is used to acquire a first training set and a second training set, wherein the image samples in the first training set and the second training set have type labels, and the number of image samples of each type in the first training set is the same;
[0107] A first training unit 420, configured to perform image classification training on a to-be-trained model using a first training set to obtain a first training model;
[0108] A determining unit 430 is configured to determine a third training set from the second training set based on the type labels of the image samples in the first training set, wherein the types of the image samples in the third training set are different from the types of the image samples in the first training set, and the number of samples of each type in the third training set is the same as the number of image samples of each type in the first training set;
[0109] A second training unit 440 is used to iterate the first training model based on the first training set, the second training set and the third training set to obtain an image classification model;
[0110] The classification unit 450 is used to classify the image to be classified using the image classification model.
[0111] In some embodiments, the second training unit 440 is also used to remove image samples in the third training set from the second training set to obtain a fourth training set; merge the first training set and the third training set to obtain a first fusion training set; merge the first training set and the fourth training set to obtain a second fusion training set; sample the first fusion training set and the second fusion training set to obtain a sampled training set; iterate the first training model based on the first fusion training set, the second fusion training set and the sampled training set to obtain an image classification model.
[0112] In some embodiments, preprocessing includes at least one of data filtering processing, standardization processing, digitization processing and time series processing.
[0113] In some embodiments, the second training unit 440 is also used to determine the loss parameter based on the sampled training set using the first training model; determine the distillation loss parameter based on the sampled training set using the model to be trained and the first training model; determine the fusion loss parameter based on the cross entropy loss parameter and the distillation loss parameter; and iterate the first training model using the fusion loss parameter to obtain an image classification model.
[0114] In some embodiments, the second training unit 440 is also used to, using the first training model, classify and score the image samples in the sampling training set that belong to the first fusion training set to obtain a first classification score; using the first training model, classify and score the image samples in the sampling training set that belong to the second fusion training set to obtain a second classification score; and determine the cross entropy loss parameter based on the first sampling training set, the first classification score and the second classification score.
[0115] In some embodiments, the second training unit 440 is also used to classify and score the image samples in the sampled training set using the model to be trained to obtain a third classification score; classify and score the image samples in the sampled training set using the first training model to obtain a fourth classification score; and determine the distillation loss parameter based on the third classification score and the fourth classification score.
[0116] In some embodiments, the second training unit 440 is further used to iterate the first training model according to a preset step size; and determine the model corresponding to when the fusion loss parameter meets the preset requirement as the image classification model.
[0117] In summary, through the image classification device, the acquisition unit is used to acquire the first training set and the second training set, the image samples in the first training set and the second training set have type labels, and the number of image samples of each type in the first training set is the same; the first training unit is used to use the first training set to perform image classification training on the training model to obtain the first training model; the determination unit is used to determine the third training set from the second training set based on the type labels of the image samples in the first training set, the types of the image samples in the third training set are different from the types of the image samples in the first training set, and the number of samples of each type in the third training set is the same as the number of image samples of each type in the first training set; the second training unit is used to iterate the first training model based on the first training set, the second training set and the third training set to obtain the image classification model; the classification unit is used to classify the image to be classified using the image classification model. The device uses the same and different numbers of samples of the same type for model training, improves the learning effect of the model when the new and old training data are unbalanced, trains the model by combining data sets, reduces the model's forgetting rate of old knowledge when learning new content, thereby improving the model performance of the trained image classification model and improving the accuracy of image classification for dynamic data sets.
[0118] In the embodiments provided in the present application, the methods and devices provided in the embodiments of the present application are introduced. In order to implement the functions in the methods provided in the embodiments of the present application, the electronic device may include a hardware structure and a software module, and implement the functions in the form of a hardware structure, a software module, or a hardware structure plus a software module. A function of the functions may be executed in the form of a hardware structure, a software module, or a hardware structure plus a software module.
[0119] Figure 7 is a block diagram of an electronic device 500 for implementing the above-mentioned image classification method according to an exemplary embodiment. For example, the electronic device 500 may be a mobile phone, a computer, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.
[0120] Reference Figure 7 The electronic device 500 may include a communication interface 501, which can interact with other devices; a processor 502, which is connected to the communication interface 501 to interact with other devices and is used to execute the method provided by one or more of the above technical solutions when running a computer program; and a memory 503, on which the computer program is stored. Specifically, the specific processing process of the processor 502 can refer to the image classification method described in the above embodiment of the present disclosure.
[0121] Of course, in actual application, the various components in the electronic device 500 are coupled together through the bus system 505. It can be understood that the bus system 505 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 505 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Figure 7 Various buses are labeled as bus system 505 .
[0122] The memory 503 in the embodiment of the present application is used to store various types of data to support the operation of the electronic device 500. Examples of such data include: any computer program used to operate on the electronic device 500.
[0123] The method disclosed in the above embodiment of the present application can be applied to the processor 502, or implemented by the processor 502. The processor 502 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 502 or the instruction in the form of software. The above processor 502 may be a general processor, a digital signal processor (DSP, Digital Signal Processor), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor 502 can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general processor may be a microprocessor or any conventional processor, etc. In combination with the steps of the method disclosed in the embodiment of the present application, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, which is located in the memory 503, and the processor 502 reads the information in the memory 503 and completes the steps of the above method in combination with its hardware.
[0124] In an exemplary embodiment, the electronic device 500 can be implemented by one or more application specific integrated circuits (ASIC), DSP, programmable logic device (PLD), complex programmable logic device (CPLD), field programmable gate array (FPGA), general processor, controller, microcontroller (MCU), microprocessor, or other electronic components to execute the aforementioned method.
[0125] The embodiments of the present disclosure further provide a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the image classification method described in the above embodiments of the present disclosure.
[0126] The embodiments of the present disclosure further provide a computer program product, including a computer program, which executes the image classification method described in the above embodiments of the present disclosure when a processor is used to execute the computer program.
[0127] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.
[0128] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiments or examples are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.
[0129] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention belong.
[0130] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processing module, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (control method), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or otherwise processing in a suitable manner if necessary, and then stored in a computer memory.
[0131] It should be understood that the various parts of the embodiments of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0132] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.
[0133] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium. The above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.
[0134] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.
Claims
1. An image classification method, characterized in that: The method comprises: Obtaining a first training set and a second training set, wherein the image samples in the first training set and the second training set have type labels, and the number of image samples of each type in the first training set is the same; Using the first training set to perform image classification training on a model to be trained to obtain a first training model; Based on the type labels of the image samples in the first training set, determining a third training set from the second training set, wherein the types of the image samples in the third training set are different from the types of the image samples in the first training set, and the number of samples of each type in the third training set is the same as the number of image samples of each type in the first training set; Iterating the first training model based on the first training set, the second training set, and the third training set to obtain an image classification model; The image classification model is used to classify the image to be classified.
2. The method according to claim 1, characterized in that The iterating the first training model based on the first training set, the second training set and the third training set to obtain the image classification model comprises: Eliminate the image samples in the third training set from the second training set to obtain a fourth training set; Merging the first training set and the third training set to obtain a first fusion training set; Merging the first training set and the fourth training set to obtain a second fused training set; Sampling the first fusion training set and the second fusion training set to obtain a sampled training set; Based on the first fusion training set, the second fusion training set and the sampling training set, the first training model is iterated to obtain an image classification model.
3. The method according to claim 2, characterized in that The iterating the first training model based on the first fusion training set, the second fusion training set and the sampling training set to obtain the image classification model includes: Based on the sampled training set, using the first training model, determining a cross entropy loss parameter; Based on the sampled training set, using the model to be trained and the first training model, determining a distillation loss parameter; Determining a fusion loss parameter based on the cross entropy loss parameter and the distillation loss parameter; The first training model is iterated using the fusion loss parameter to obtain an image classification model.
4. The method according to claim 3, characterized in that Determining the cross entropy loss parameter based on the sampled training set and using the first training model includes: Using the first training model, classify and score the image samples in the sampling training set that belong to the first fusion training set to obtain a first classification score; Using the first training model, classify and score the image samples in the sampling training set that belong to the second fusion training set to obtain a second classification score; The cross entropy loss parameter is determined based on the first sampled training set, the first classification score, and the second classification score.
5. The method according to claim 3, characterized in that: The determining of the distillation loss parameter based on the sampled training set and using the model to be trained and the first training model includes: Using the model to be trained, classify and score the image samples in the sampled training set to obtain a third classification score; Using the first training model, classify and score the image samples in the sampled training set to obtain a fourth classification score; A distillation loss parameter is determined based on the third category score and the fourth category score.
6. The method according to claim 3, characterized in that The using the fusion loss parameter to iterate the first training model to obtain the image classification model includes: Iterate the first training model according to a preset step size; The model corresponding to the fusion loss parameter that meets the preset requirements is determined as the image classification model.
7. An image classification device, characterized in that: The device comprises: An acquiring unit, configured to acquire a first training set and a second training set, wherein the image samples in the first training set and the second training set have type labels, and the number of image samples of each type in the first training set is the same; A first training unit is used to perform image classification training on a to-be-trained model using the first training set to obtain a first training model; A determining unit, configured to determine a third training set from the second training set based on the type labels of the image samples in the first training set, wherein the types of the image samples in the third training set are different from the types of the image samples in the first training set, and the number of samples of each type in the third training set is the same as the number of image samples of each type in the first training set; A second training unit is used to iterate the first training model based on the first training set, the second training set and the third training set to obtain an image classification model; The classification unit is used to classify the image to be classified using the image classification model.
8. An electronic device, characterized in that: include: a processor and a memory for storing a computer program capable of being executed on the processor, Wherein, when the processor is used to run the computer program, it executes the method according to any one of claims 1 to 6.
9. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-6.
10. A computer program product, characterized in that The invention comprises a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 6.
Citation Information
Patent Citations
Incremental training method of classification network, classification method and device, equipment and medium
CN110210560A
Image big data-oriented class increment classification method, system and device and medium
CN112990280A
Federal incremental learning method and device for cross-mobile edge network, and storage medium
CN117540822A
Incremental training method of image classification model, image classification model and method
CN117746184A
Few-shot image incremental classification method and apparatus based on embedding enhancement and adaption
WO2023137889A1
