Image classification method based on multi-level adaptive feature fusion incremental learning

Abstract

The present invention discloses an image classification method based on multi-level adaptive feature fusion incremental learning, including step 1, the first training, obtaining the first image sample set and the first model after training; step 2, when a new type needs to be identified, incremental update is performed, and after the Mth incremental update, the M+1th model and the fusion coefficient of the Mth model and the M+1th model are obtained; M represents the number of incremental updates, which is a natural number greater than or equal to 1; step 3, during testing, the M+1th model and the Mth model are used to perform feature fusion according to the fusion coefficient, and the prediction result is output. This method solves the problem that the current object recognition algorithm cannot continuously learn new categories and new knowledge, and the incremental learning algorithm cannot effectively utilize the information of the old model and forgets the old knowledge to a large extent.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and in particular relates to an image classification method based on multi-level adaptive feature fusion incremental learning. Background Art

[0002] Object classification in computer vision currently has a wide range of application scenarios, such as text recognition, product recognition, vehicle recognition, etc. However, traditional models are static and cannot continuously learn new knowledge. Whenever new categories of data are added, they must be retrained, which requires a lot of extra time and space. As new tasks increase, the marginal benefits decrease. Class incremental learning proposes a solution to this problem.

[0003] The current mainstream methods for incremental learning mainly include the following three categories:

[0004] 1. Replay-based quasi-incremental learning method: This method directly stores samples of old tasks in their original format, or uses a generative model to generate pseudo samples of old tasks, and merges samples of old tasks with samples of new tasks for training.

[0005] 2. Regularization-based quasi-incremental learning method: This method imposes more constraints on the weights in the neural network to consolidate old knowledge. This is usually achieved by adding additional regularization terms to penalize changes in the mapping function of the neural network. There are two main strategies for this type of method. One is to adjust the network parameters based on prior probability, and the other is based on data.

[0006] 3. Dynamic structure-based methods: This type of method mainly dynamically adds neurons or network layers during training, and responds to new information by changing the network structure properties. There are two main strategies for this type of method. One is to add a fixed number of network layers or neurons for each new task, and the new network parameters are isolated from the previous network; the other is to adaptively adjust the network structure and weights by balancing network complexity and loss.

[0007] The direct storage of old samples in method 1 does not utilize the knowledge of the old model, resulting in catastrophic forgetting. The method of generating pseudo samples is difficult in actual operation, and the generated pseudo samples often cannot meet the needs. Method 2 often cannot accurately quantify the importance of weights in the network, so it cannot accurately constrain key weights. At the same time, it does not utilize the representation ability of the old model, resulting in poor resistance to catastrophic forgetting. Method 3 will cause the model to continue to expand. Although the old model is retained to a certain extent, as the tasks increase, the network structure becomes more complex, and when only some network nodes are added, the back propagation algorithm is inefficient. Summary of the invention

[0008] Purpose of the invention: The technical problem to be solved by the present invention is to provide an image classification method based on multi-level adaptive feature fusion incremental learning in view of the shortcomings of the prior art.

[0009] In order to solve the above technical problems, the present invention discloses an image classification method based on multi-level adaptive feature fusion incremental learning, comprising:

[0010] Step 1: the first training, obtaining the first image sample set and the first trained model;

[0011] Step 2: When a new type needs to be identified, an incremental update is performed. After the Mth incremental update, the M+1th model and the fusion coefficient of the Mth model and the M+1th model are obtained; M represents the number of incremental updates, which is a natural number greater than or equal to 1.

[0012] Step 3: During testing, use the M+1th model and the Mth model to perform feature fusion according to the fusion coefficient and output the prediction result.

[0013] Furthermore, in step 2, when M is equal to 1, after the first incremental update, the second model is obtained, and the fusion coefficient of the first model and the second model includes:

[0014] Step 2-1, creating a new classifier and completing initialization, wherein the second model includes the first model and the new classifier;

[0015] Step 2-2, obtaining second training image data, preprocessing the second training image data, and initializing a fusion coefficient; the second training image data includes a first image sample set and an incrementally updated image data set, and the second training image data is denoted as N; the fusion coefficient is used to perform feature fusion of the features of the first model and the features of the second model in proportion;

[0016] Step 2-3: Train the first model and the second model. The training process is divided into two stages:

[0017] In the first stage, the second training image data is input into the first model and the second model for training respectively, the parameters and fusion coefficients of the first model are fixed, and only the parameters of the second model are trained to obtain the trained parameters of the second model;

[0018] The second stage is carried out after the training of the first stage is completed. The second image sample set is constructed and input into the first model and the second model for training respectively. All parameters of the first model and the second model are fixed, and only the fusion coefficient is trained to obtain the fusion coefficient after training.

[0019] When M is greater than 1, the process of obtaining the M+1th model and the fusion coefficient of the Mth model and the M+1th model is the same as when M is equal to 1, wherein the second training image data includes the Mth image sample set constructed by the M-1th incremental update and the image data set updated by the Mth incremental update, and the second stage in step 2-3 constructs the M+1th image sample set.

[0020] Furthermore, the first training to obtain the first image sample set and the trained first model includes:

[0021] Step 1-1, preprocessing the image data set for the first training;

[0022] Step 1-2, establishing and training the first model to obtain the trained first model; the first model includes a feature extraction network and a classifier layer, and the number of nodes in the classifier layer is the number of categories of the image data set trained for the first time; using cross entropy as a loss function, and then using a gradient descent algorithm to train the first model; when the loss converges, the training of the first model is completed;

[0023] Step 1-3, select the first image sample set from the image data set of the first training, and record the first image sample set as N 0 .

[0024] Furthermore, creating a new classifier and completing initialization in step 2-1 includes:

[0025] If Y' new categories of objects need to be identified, a classifier with Y' nodes is created;

[0026] Initialize the new classifier weights using imprint weights;

[0027] The incrementally updated image dataset samples are input into the feature extraction network of the second model according to classification to obtain low-dimensional embedding data of the incrementally updated image dataset samples, the low-dimensional embedding data of the incrementally updated image dataset samples are standardized and the average value is calculated, and the obtained results are assigned to the corresponding classifier nodes to complete the initialization of the new classifier.

[0028] Furthermore, the feature extraction network in step 1-2 adopts a ResNet32 network; in step 2-3, when training the first model and the second model, the features of the first model and the features of the second model are fused according to the initial fusion coefficient, and in the ResNet32 network, fusion is performed once every 5 residual blocks.

[0029] Furthermore, in step 2-3, the first stage includes:

[0030] The second training image data sample is denoted as x, and the second training image data sample x is input into the feature extraction network of the first model and the second model respectively to obtain the low-dimensional embedding data f of the second training image data sample in the first model. * (x) and the low-dimensional embedded data f(x) in the second model;

[0031] The low-dimensional embedding data f of the second training image data sample in the first model * (x) and the low-dimensional embedded data f(x) in the second model are standardized to obtain the standardized low-dimensional embedded data, which are recorded as and

[0032] Apply distillation loss to the second model The formula is as follows:

[0033]

[0034] in, express and The cosine similarity between ;

[0035] Add a new constraint L mr (x), which increases the distance between classes and avoids confusion between classes.

[0036]

[0037] Where m is the boundary threshold, The low-dimensional embedding data representing the true category of the second training image data sample x is normalized. is the kth low-dimensional embedding closest to the low-dimensional embedding and of a different category from the second training image data sample x, 1≤k≤K, where K represents a natural number greater than or equal to 1;

[0038] The final loss function L is shown in the following formula:

[0039]

[0040] Where L ce is the cross entropy, λ represents the balance coefficient of the number of new and old categories, and the calculation method is as follows:

[0041]

[0042] λ base represents the balance cardinality of the number of new and old categories, which is a constant and is adjusted according to the different image datasets; Cn represents the image dataset updated for the Mth incremental update, that is, the new category set, C oRepresents the image dataset for the first training and the image dataset from the 1st incremental update to the M-1th incremental update, that is, the old category set;

[0043] Use the stochastic gradient descent algorithm for training, wait for the second model to converge, and then enter the second stage.

[0044] Furthermore, in the first stage of step 2-3, the low-dimensional embedding data f of the second training image data sample in the first model is * (x) and the low-dimensional embedded data f(x) in the second model are standardized, and the obtained standardized low-dimensional embedded data is a vector after l2 normalization, and the formula is as follows:

[0045]

[0046]

[0047] The output of the classifier layer is normalized by the softmax function. i (x) is shown in the following formula;

[0048]

[0049] Among them, i and j represent the classification category index, which are both less than or equal to the total number of classification categories. and Respectively represent the standardized weights of the i-th node and the j-th node of the classifier layer,

[0050]

[0051]

[0052] The second training image data θ i and θ j Respectively represent the weights of the i-th node and the j-th node of the classifier layer, measure and The cosine similarity between Constrained to [-1, 1].

[0053] Furthermore, in step 2-3, the loss function of the second stage uses only cross entropy, and the training is performed using the stochastic gradient descent algorithm until convergence;

[0054] When M is equal to 1, the second image sample set includes the first image sample set and an incremental update image sample set, and the incremental update image sample set is obtained by screening the incremental update image data set;

[0055] When M is greater than 1, the M+1th image sample set includes the Mth image sample set constructed by the M-1th incremental update and the incremental update image sample set.

[0056] Furthermore, the preprocessing of the image data in step 1-1 and step 2-2 includes: randomly flipping the image data, including horizontal flipping and vertical flipping, and then converting it into a tensor and standardizing it. The standardization calculation process is to first calculate the mean and variance of the training image data, subtract the mean from each image, and then divide it by the variance.

[0057] Furthermore, the first image sample set in step 1-3 and the incremental updated image sample set in the second stage of step 2-3 are screened based on the herding algorithm for the first training image dataset and the incremental updated image dataset, respectively, including:

[0058] The filtered image dataset is X = {X 1 , X 2 , ..., X Y}, there are Y categories in total, Y is a positive integer, category i contains n i samples, 1≤i≤Y,n i is a positive integer, then the image dataset of category i is The current feature extraction network is For each category, select z samples, 1≤z≤min{n 1 , n 2 , ..., n Y}, then when selecting samples for the i-th category, first calculate the feature mean of the category, and the calculation method is as follows:

[0059]

[0060] Then let u = 1, 2, ..., z, and find z samples p according to the following formula 1 , p 2 , ..., p z ; Finally, the obtained p 1 , p 2 , ..., p z Composition image sample set P i .

[0061]

[0062] The above operation is performed for each category in the filtered image dataset to obtain the filtered image sample set P = {P 1 , P 2 , ..., P Y}.

[0063] Beneficial effects:

[0064] In the existing incremental learning schemes, most of them discard the old model and only save the image sample set, or only use the old model to calculate the distillation loss, which wastes the representation ability of the old model. Saving the image sample set can make the model recall the old knowledge to a certain extent, but once the new model is incrementally trained, it will be offset and gradually weaken the representation ability of the old category objects, thus causing forgetting. The method proposed in the present invention solves the problem that the current object recognition algorithm cannot continuously learn new categories and new knowledge, and the incremental learning algorithm cannot effectively utilize the information of the old model and forgets the old knowledge to a large extent.

[0065] 1. Not only the old samples are saved in the incremental learning process, but also the old model is saved. The representation of the old model is used to limit the changes of the new model. The old model is used to extract multi-level features and merge them into the new model, which further improves the memory of the entire model and reduces the new model's forgetting of old knowledge.

[0066] 2. Two-stage training is used in incremental learning. In the first stage, the new and old model features are integrated, only the new model weights are updated, and all other parameters are fixed. In the second stage, only the integrated parameters are trained, and the weights of the new and old models are fixed. The two-stage training method avoids the large difference in the distribution of the image data before and after, which leads to a large impact between the new and old models. The fusion coefficient can gradually adjust the degree of feature fusion, which can maximize the stability of the model while ensuring the plasticity of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more clear.

[0068] Figure 1 A flowchart of an image classification method based on multi-level adaptive feature fusion incremental learning provided in an embodiment of the present application.

[0069] Figure 2 A schematic diagram of the imprinting weights of an image classification method based on multi-level adaptive feature fusion incremental learning provided in an embodiment of the present application.

[0070] Figure 3 Schematic diagram of the fusion of new and old models of the image classification method based on multi-level adaptive feature fusion incremental learning provided in an embodiment of the present application.

[0071] Figure 4 A schematic diagram of low-dimensional embedding standardization of an image classification method based on multi-level adaptive feature fusion incremental learning provided in an embodiment of the present application.

[0072] Figure 5 Schematic diagram of distillation loss.

[0073] Figure 6 A schematic diagram of increasing the class distance of an image classification method based on multi-level adaptive feature fusion class incremental learning provided in an embodiment of the present application. DETAILED DESCRIPTION

[0074] The embodiments of the present invention will be described below in conjunction with the accompanying drawings.

[0075] The present application provides an image classification method based on multi-level adaptive feature fusion incremental learning, which can be applied to quickly build an image classification system with continuous autonomous learning.

[0076] The specific processing flow of this method is as follows Figure 1 As shown:

[0077] In the whole process, we must first determine whether it is the first time to conduct training. If it is the first time to conduct training, no incremental update operation is involved. Otherwise, the model needs to be incrementally updated.

[0078] Next, the algorithm details in two cases are elaborated in detail:

[0079] 1. First training:

[0080] Since there is no incremental update process, the image data can be preprocessed directly. The preprocessing process includes randomly flipping the image data, including horizontal flipping and vertical flipping, and then converting it into a tensor and standardizing it. The standardization calculation process is to first calculate the mean and variance of the training image data, subtract the mean from each image, and then divide it by the variance.

[0081] After the image data processing is completed, a network model needs to be established, namely the first model. The first model includes a feature extraction network and a classifier layer. This embodiment uses a ResNet32 network as a feature extraction network. The ResNet32 network has a total of 32 layers. The number of nodes in the classifier layer is the number of categories of the current image data set. Here, cross entropy is used as the loss function, and then the gradient descent algorithm is used to train the model. For details about ResNet, please refer to He, Kaiming, et al. "Deep residual learning for image recognition." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.

[0082] When the loss converges, the training of the first model is completed. Next, appropriate images are selected from the image dataset of the first training to be stored and form the first image sample set, so that they can be added to the next training to reduce the forgetting of old categories. The screening algorithm of the first image sample set is based on the herding algorithm. The screened image dataset is denoted as X = {X 1 , X 2 , ..., X Y}, there are Y categories in total, Y is a positive integer, category i contains n i samples, 1≤i≤Y,n i is a positive integer, then the image dataset of category i is The current feature extraction network is For each category, select z samples, 1≤z≤min{n 1 , n 2 , ..., n Y}, then when selecting samples for the i-th category, first calculate the feature mean of the category. The calculation method is shown in formula (1):

[0083]

[0084] Then let u = 1, 2, ..., z, and find z samples p according to formula (2) 1 , p 2 , ..., p z ; Finally, the obtained p 1 , p 2 , ..., p z Composition image sample set P i ;

[0085]

[0086] The above operation is performed for each category in the filtered image dataset to obtain the filtered image sample set P = {P 1 , P 2 , ..., P Y}.

[0087] At this point, the first training is completed, and then the image sample set, the trained model, and the weights are saved for incremental updates.

[0088] 2. Incremental update model:

[0089] When a new object needs to be identified, errors will occur if the model is not updated, because only one of the categories learned by the old model can be output. If most object classification algorithms want to identify more types of objects, they need to obtain all the previous and subsequent training image data and retrain the model, which not only wastes the knowledge learned by the old model, but also consumes more time and space. For these two problems, this application proposes effective solutions, as follows:

[0090] First, copy the old model (model 1) to get a new model (model 2). Assuming that Y' new categories of objects need to be identified, a classifier with Y' nodes is created in model 2. Since randomly initializing the weights of the new classifier and retraining it will take more time, in order to speed up this process, the imprinted weights are used to initialize the weights of the new classifier. Figure 2 As shown, the incrementally updated image dataset samples are input into the feature extraction network of the second model according to classification to obtain low-dimensional embedding data of the incrementally updated image dataset samples, the low-dimensional embedding data of the incrementally updated image dataset samples are standardized and the average value is calculated, and the obtained results are assigned to the corresponding classifier nodes.

[0091] After the new classifier is created and initialized, it is necessary to organize the image data required for the current incremental training, that is, the second training image data. The second training image data consists of two parts. The first part is all the image data of the newly added category, and the second part is the image sample set saved at the end of the previous task. Then initialize the fusion coefficient ∈. There are three fusion coefficients in total (such as Figure 3 As shown, are three fusion coefficients, among which ), are initialized to 0.5. Then the second training image data is preprocessed. The preprocessing process is the same as the first training. First, it is randomly flipped, then converted into a tensor, and finally standardized. Note that the image data set has changed at this time, and the mean and variance should also be recalculated. When standardizing, the new mean is subtracted and then divided by the new variance.

[0092] like Figure 3 As shown, the new and old models need to be combined with the fusion coefficients to perform feature fusion in proportion. In ResNet32, fusion is performed every 5 residual blocks. The training process is divided into two stages. The first stage only trains the parameters of the new model, and the second stage only trains the fusion coefficients. The following is a detailed introduction to the training process in stages:

[0093] 1. Phase I

[0094] In this stage, only the new model parameters are trained, and the old model parameters and fusion coefficients are fixed.

[0095] Since there are more image data in the new category and less image data in the old category, the classification results of the classifier will be tilted towards the new category. By analyzing the low-dimensional embedding of the feature extraction network, we can know that the modulus of the low-dimensional embedding of the new category is often larger than the low-dimensional embedding of the old category, so it needs to be standardized, such as Figure 4 shown.

[0096] In common CNN (Convolutional Neural Networks), the predicted value of a sample, that is, the output of the classifier layer after normalization by the softmax function, is often calculated as shown in formula (3). The second training image data sample is denoted as x. The second training image data sample x is input into the feature extraction network of the first model and the second model respectively to obtain the low-dimensional embedded data f of the second training image data sample in the first model. * (x) and the low-dimensional embedded data f(x) in the second model;

[0097]

[0098] Where i and j represent the classification category index, both less than or equal to the total number of classification categories, θ i and θ j Respectively represent the weights of the i-th node and the j-th node of the classifier layer, b i and b j They represent the bias items of the i-th node and the j-th node of the classifier layer respectively.

[0099] The normalization method uses cosine normalization, as shown in formula (4):

[0100]

[0101] in, represents the vector after 12 normalization, Measures the cosine similarity between two normalized vectors, η is used to convert Constrained to [-1, 1].

[0102] The low-dimensional embedding data f of the second training image data sample in the first model * (x) and the low-dimensional embedded data f(x) in the second model are standardized to obtain the standardized low-dimensional embedded data, which are recorded as and

[0103]

[0104]

[0105] and Respectively represent the standardized weights of the i-th node and the j-th node of the classifier layer,

[0106]

[0107]

[0108] A common problem in incremental learning is catastrophic forgetting. Since only a very small amount of old category image data is stored, and a large amount of new category image data is simultaneously introduced, and as the tasks continue to increase and the model continues to be updated, the network will gradually forget the previously learned knowledge and drift, such as Figure 5 Therefore, in order to reduce the network's forgetfulness and have a better recognition ability for both the previous and next categories, a distillation loss needs to be applied to the second model, as shown in formula (5).

[0109]

[0110] This loss encourages the features extracted by the new and old models to be as similar as possible, so as to reduce forgetting. While limiting the changes in the new model, it is necessary to supplement the knowledge of the old model. Integrating the knowledge of the old model into the new model can make the network recall the old knowledge, effectively reducing the model's forgetfulness of old categories and improving the recognition effect. ResNet32 has a residual module. Every five residual modules are nodes, and the model is divided into four parts. 1 , f 2 , f 3 , f 4 Indicates that the old model is also divided into four parts, respectively Indicates that, using f1 and f2 as examples, taking the first and second parts as examples, the features output after the image passes through f1 are go through The output features are The fusion coefficients at this time are and Assume that the input of f2 is Specifically, as shown in formula (6), f2 and f3 are similar.

[0111]

[0112] As new categories of objects are added, the boundaries between new and old categories will become increasingly blurred, leading to mutual recognition errors. Therefore, it is necessary to increase the distance between classes so that there is no confusion between classes, such as Figure 6To this end, a new constraint needs to be added. Consider the low-dimensional embedding of the true category of a sample x as a positive example, find the category in the new category that produces the highest response to x as the difficult negative class, and use their low-dimensional embedding as the negative example corresponding to the low-dimensional embedding of the old sample, as shown in formula (7).

[0113]

[0114] Where m is the boundary threshold, which is a constant and is set to 0.5. The low-dimensional embedding data representing the true category of the second training image data sample x is normalized. is the kth low-dimensional embedding closest to the low-dimensional embedding and of a different category from the second training image data sample x, 1≤k≤K, K represents a natural number greater than or equal to 1, and in this embodiment, the value of k is set to 2. The final loss function is shown in formula (8),

[0115]

[0116] Where L ce is the cross entropy, N is the second training image data, N o is the saved old image sample set, and the calculation method of λ is shown in formula (9):

[0117]

[0118] λ base Indicates the balance base number of new and old categories, which is a constant and needs to be adjusted according to different image datasets. In this embodiment, it is set to 7.5. n represents the image dataset updated for the Mth incremental time, i.e., the new category set, C o Represents the image dataset for the first training and the image dataset from the 1st incremental update to the M-1th incremental update, that is, the old category set;

[0119] The initial value of the learning rate is set to 0.1, the number of training image data in each batch is set to 128, and a total of 160 epochs are trained. It is adjusted to 0.01 at the 80th epoch and to 0.001 at the 120th epoch. The momentum is set to 0.9, and the stochastic gradient descent algorithm is used for training. After waiting for the model to converge, it can enter the second stage.

[0120] 2. Second stage

[0121] After the first stage of training, it is necessary to construct an image sample set for the new category of samples. The construction method of the image sample set is the same as the first training. After the construction is completed, the previous image sample set is merged to form the second image sample set.

[0122] The second image sample set will be used as the training set for the second stage.

[0123] Because the image data distribution of the new and old categories is different, the new and old models cannot be fused in a fixed ratio of 1:1, so the fusion coefficient needs to be adjusted in time.

[0124] In the second stage of training, all parameters of the new and old models are fixed, and only the fusion coefficient is opened for training. The loss function only uses cross entropy, the initial value of the learning rate is set to 0.1, the number of training image data in each batch is set to 128, and a total of 100 generations are trained. The learning rate is adjusted to 0.01 at the 50th generation and to 0.001 at the 75th generation. The momentum is set to 0.9, and the stochastic gradient descent algorithm is also used for training until convergence.

[0125] After the first and second stages of training are completed, save the new and old models. During testing, use the new and old models to perform feature fusion according to the trained fusion coefficients and output the prediction results.

[0126] When there are new categories of objects that need to be identified, the old model is discarded, the new image data is processed, a new classifier is established, and then the incremental update process is repeated using the new model.

[0127] The following table shows the experimental results of the method of the embodiment of the present application and the related methods of incremental learning in the prior art. The image data set uses CIFAR-100, which has 100 categories of pictures. The first training uses 50 categories of image data. "2 categories / stage" means that the remaining categories increase by 2 categories each time until they increase to 100 categories. The values ​​in the table are the average accuracy of all stages. It can be seen from the table below that the average accuracy of the method proposed in the embodiment of the present application is higher than that of other incremental learning methods.

[0128]

[0129] References are as follows:

[0130] [1] Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935–2947, 2017.

[0131] [2]Sylvestre-Alvise Rebuffffi,Alexander Kolesnikov,Georg Sperl,andChristoph H Lampert.icarl:Incremental classifier and representationlearning.In Proceedings of the IEEE conference on Computer Vision and PatternRecognition,pages 2001–2010,2017.

[0132] [3]Liu,Yaoyao,et al."Mnemonics training:Multi-class incrementallearning without forgetting."Proceedings of the IEEE / CVF conference onComputer Vision and Pattern Recognition.2020.

[0133] [4]Wu,Y.,Chen,Y.,Wang,L.,Ye,Y.,Liu,Z.,Guo,Y.,&Fu,Y.(2019).Large scaleincremental learning.In Proceedings of the IEEE / CVF Conference on ComputerVision and Pattern Recognition(pp.374-382).

[0134] [5]Saihui Hou,Xinyu Pan,Chen Change Loy,Zilei Wang,and DahuaLin.Learning a unified classifier incrementally via rebalancing.InProceedings of the IEEE / CVF Conference on Computer Vision and PatternRecognition,pages 831–839,2019.

[0135] The present invention provides an image classification method based on multi-level adaptive feature fusion incremental learning. There are many methods and ways to implement the technical solution. The above is only a specific implementation of the present invention. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present invention, several improvements and modifications can be made, and these improvements and modifications should also be regarded as the protection scope of the present invention. All components not specified in this embodiment can be implemented by existing technologies.

Claims

1. An image classification method based on multi-level adaptive feature fusion incremental learning, characterized in that: include: Step 1: the first training, obtaining the first image sample set and the first trained model; Step 2: When a new type needs to be identified, an incremental update is performed. After the Mth incremental update, the M+1th model and the fusion coefficient of the Mth model and the M+1th model are obtained; M represents the number of incremental updates, which is a natural number greater than or equal to 1. Step 3: During testing, the M+1th model and the Mth model are used to perform feature fusion according to the fusion coefficient, and the prediction results are output to perform image classification; In step 2, when M is equal to 1, after the first incremental update, the second model is obtained, and the fusion coefficients of the first model and the second model include: Step 2-1, creating a new classifier and completing initialization, wherein the second model includes the first model and the new classifier; Step 2-2, obtaining second training image data, preprocessing the second training image data, and initializing a fusion coefficient; the second training image data includes a first image sample set and an incrementally updated image data set, and the second training image data is denoted as N; the fusion coefficient is used to perform feature fusion of the features of the first model and the features of the second model in proportion; Step 2-3: Train the first model and the second model. The training process is divided into two stages: In the first stage, the second training image data is input into the first model and the second model for training respectively, the parameters and fusion coefficients of the first model are fixed, and only the parameters of the second model are trained to obtain the trained parameters of the second model; The second stage is carried out after the training of the first stage is completed. The second image sample set is constructed and input into the first model and the second model for training respectively. All parameters of the first model and the second model are fixed, and only the fusion coefficient is trained to obtain the fusion coefficient after training. When M is greater than 1, the process of obtaining the M+1th model and the fusion coefficient of the Mth model and the M+1th model is the same as when M is equal to 1, wherein the second training image data includes the Mth image sample set constructed by the M-1th incremental update and the image data set updated by the Mth incremental update, and the second stage in step 2-3 constructs the M+1th image sample set; The first training to obtain the first image sample set and the trained first model includes: Step 1-1, preprocessing the image data set for the first training; Step 1-2, establishing and training the first model to obtain the trained first model; the first model includes a feature extraction network and a classifier layer, and the number of nodes in the classifier layer is the number of categories of the image data set trained for the first time; using cross entropy as a loss function, and then using a gradient descent algorithm to train the first model; when the loss converges, the training of the first model is completed; Step 1-3, the first image sample set is obtained by screening from the image data set of the first training, and the first image sample set is recorded as N0.

2. The image classification method based on multi-level adaptive feature fusion incremental learning according to claim 1 is characterized in that: The step 2-1 of creating a new classifier and completing initialization includes: If Y' new categories of objects need to be identified, a classifier with Y' nodes is created; Initialize the new classifier weights using the imprinted weights; The incrementally updated image dataset samples are input into the feature extraction network of the second model according to classification to obtain low-dimensional embedding data of the incrementally updated image dataset samples, the low-dimensional embedding data of the incrementally updated image dataset samples are standardized and the average value is calculated, and the obtained results are assigned to the corresponding classifier nodes to complete the initialization of the new classifier.

3. The image classification method based on multi-level adaptive feature fusion incremental learning according to claim 2 is characterized in that: The feature extraction network in step 1-2 adopts the ResNet32 network; in step 2-3, when training the first model and the second model, the features of the first model and the features of the second model are fused according to the initial fusion coefficient, and in the ResNet32 network, fusion is performed once every 5 residual modules.

4. The image classification method based on multi-level adaptive feature fusion incremental learning according to claim 3 is characterized in that: In the steps 2-3, the first stage includes: The second training image data sample is denoted as x, and the second training image data sample x is input into the feature extraction network of the first model and the second model respectively to obtain the low-dimensional embedding data f of the second training image data sample in the first model. * (x) and the low-dimensional embedded data f(x) in the second model; The low-dimensional embedding data f of the second training image data sample in the first model * (x) and the low-dimensional embedded data f(x) in the second model are standardized to obtain the standardized low-dimensional embedded data, which are recorded as and Apply distillation loss to the second model The formula is as follows: in, express and The cosine similarity between ; Add a new constraint L mr (x), which increases the distance between classes and avoids confusion between classes. Where m is the boundary threshold, The low-dimensional embedding data representing the true category of the second training image data sample x is normalized. is the kth low-dimensional embedding closest to the low-dimensional embedding and of a different category from the second training image data sample x, 1≤k≤K, where K represents a natural number greater than or equal to 1; The final loss function L is shown in the following formula: Where L ce is the cross entropy, λ represents the balance coefficient of the number of new and old categories, and the calculation method is as follows: λ base Indicates the balance base of the number of new and old categories, which is a constant and is adjusted according to different image datasets; C n represents the image dataset updated for the Mth incremental time, i.e., the new category set, C o Represents the image dataset for the first training and the image dataset from the 1st incremental update to the M-1th incremental update, that is, the old category set; Use the stochastic gradient descent algorithm for training, wait for the second model to converge, and then enter the second stage.

5. The image classification method based on multi-level adaptive feature fusion incremental learning according to claim 4 is characterized in that: In the first stage of step 2-3, the low-dimensional embedding data f of the second training image data sample in the first model is * (x) and the low-dimensional embedded data f(x) in the second model are standardized as follows: The output of the classifier layer is normalized by the softmax function. i (x) is shown in the following formula; Among them, i and j represent the classification category index, which are both less than or equal to the total number of classification categories. and Respectively represent the standardized weights of the i-th node and the j-th node of the classifier layer, The second training image data θ i and θ j Respectively represent the weights of the i-th node and the j-th node of the classifier layer, Used to measure and The cosine similarity between Clamped to [-1,1].

6. The image classification method based on multi-level adaptive feature fusion incremental learning according to claim 5, characterized in that: In step 2-3, the loss function of the second stage uses only cross entropy, and the stochastic gradient descent algorithm is used for training until convergence; When M is equal to 1, the second image sample set includes the first image sample set and an incremental update image sample set, and the incremental update image sample set is obtained by screening the incremental update image data set; When M is greater than 1, the M+1th image sample set includes the Mth image sample set constructed by the M-1th incremental update and the incremental update image sample set.

7. The image classification method based on multi-level adaptive feature fusion incremental learning according to claim 6 is characterized in that: The preprocessing of the image data in step 1-1 and step 2-2 includes: randomly flipping the image data, including horizontal flipping and vertical flipping, and then converting it into a tensor and standardizing it. The standardization calculation process is to first calculate the mean and variance of the training image data, subtract the mean from each image, and then divide it by the variance.

8. The image classification method based on multi-level adaptive feature fusion incremental learning according to claim 7 is characterized in that: The first image sample set in step 1-3 and the incremental update image sample set in the second stage of step 2-3 are both screened based on the herding algorithm for the first training image dataset and the incremental update image dataset, respectively, including: The filtered image dataset is X = {X 1 ,X 2 ,…,X Y }, there are Y categories in total, Y is a positive integer, category i contains n i samples, 1≤i≤Y,n i is a positive integer, then the image dataset of category i is The current feature extraction network is Select z samples for each category, 1≤z≤min{n1,n2,…,n Y }, then when selecting samples for the i-th category, first calculate the feature mean of the category, and the calculation method is as follows: Then let u=1,2,…,z, and find z samples p1,p2,…,p according to the following formula. z ; Finally, we get p1, p2, ..., p z Composition image sample set P i ; The above operation is performed for each category in the filtered image dataset to obtain the filtered image sample set P = {P 1 ,P 2 ,…,P Y }.

Images