Multi-regularization strategy image classification learning method, device, equipment and medium

Abstract

This application discloses a multi-regularization strategy image classification learning method, apparatus, device, and medium. The method includes: constructing a first loss function for a convolutional neural network (CNN) based on LWF regularization and pooling knowledge distillation; training the feature extraction layer and main classification layer of the CNN using the first loss function; constructing a second loss function for the CNN based on the cross-entropy loss function of the sub-classification layer and labeled data, and the classification layer difference loss function between sub-classification layers; and training the sub-classification layer using the second loss function. The feature extraction layer, main classification layer, and sub-classification layer are used to train the CNN. Applying LWF regularization to the pooling distillation function enhances the constraint on the first loss function. Compared to training with a single classification layer, using multiple sub-classification layers to calculate the second loss function makes the training process of the CNN more robust.

Description

Technical Field

[0001] This application relates to the field of image classification technology, and for example to multi-regularization strategy image classification learning methods, apparatus, devices and media. Background Technology

[0002] Convolutional neural networks (CNNs) have been widely used in recent years. However, in image classification, CNNs suffer from catastrophic forgetting, meaning that after training on a new set of samples, their classification accuracy on older sets drops significantly. To address this, incremental learning methods are currently employed. The main incremental learning methods include playback-based, constraint-based, and parameter isolation-based methods, all of which can mitigate the catastrophic forgetting problem to some extent. However, traditional image classification tasks rely on a single classifier, and activation-based regularization strategies are heavily influenced and limited by the performance of this single classifier. Summary of the Invention

[0003] This application provides a multi-regularization strategy image classification learning method, apparatus, device, and medium, aiming to solve the problem that activation-based regularization strategy methods are largely affected by the classification performance of a single classifier.

[0004] To solve the above problems, this application adopts the following technical solution:

[0005] This paper presents a multi-regularization strategy image classification learning method, including:

[0006] The first loss function of the convolutional neural network is constructed based on LWF (Learning without Forgetting) regularization and pooling knowledge distillation. This first loss function is then used to train the feature extraction layer and the main classification layer of the convolutional neural network. The formula for the first loss function is as follows:

[0007] Loss1 = Loss MW (F,M t )+λLoss LWF-Pooling ;

[0008] Among them, Loss MW The cross-entropy loss function is used for the main classification layer, F is the feature extraction layer, and M is the cross-entropy loss function. t Let λ be the main classification layer for the t-th task, and λ be the pooling adjustment parameter. Loss LWF-Pooling Let LWF be the regularized pooling distillation loss function;

[0009] Based on the cross-entropy loss function of the subclassification layer and the labeled data, and the classification layer difference loss function between the subclassification layers, a second loss function of the convolutional neural network is constructed; the subclassification layer is then trained using the second loss function.

[0010] The feature extraction layer, the main classification layer, and the sub-classification layer are used to train the convolutional neural network.

[0011] The first loss function for constructing the convolutional neural network based on LWF regularization and pooling knowledge distillation includes:

[0012] Based on the concatenation result of the output vectors of the main classification layer of the convolutional neural network, the pooling distillation loss of the main classification layer is calculated using LWF regularization.

[0013] Based on the concatenation result of the output vectors of the subclassification layer of the convolutional neural network, the pooling distillation loss of the subclassification layer is calculated using LWF regularization;

[0014] The LWF regularized pooling distillation loss function is calculated based on the pooling distillation loss of the main classification layer and the pooling distillation loss of the sub-classification layer.

[0015] The formula for calculating the LWF regularized pooling distillation loss function is as follows:

[0016] Loss LWF-Pooling =Loss LWF-main-pooling +Loss LWF-side-pooling ;

[0017] Among them, Loss LWF-main-pooling Main classification layer pooling distillation loss, Loss LWF-side-pooling This represents the loss from pooling distillation in the sub-classification layer;

[0018] The first loss function is constructed based on the cross-entropy loss function of the main classification layer and the LWF regularized pooling distillation loss function.

[0019] The step of calculating the pooling distillation loss of the main classification layer using LWF regularization based on the concatenation result of the main classification layer output vector of the convolutional neural network includes:

[0020] The formula for the pooling distillation loss in the main classification layer is:

[0021]

[0022] Among them, KD M The knowledge distillation equation for the main classification layer is given, where σ is the softmax (normalized exponential function) classification layer. It is the i-th image in the input image sample set at time t. This represents the concatenated vector output of the main classification layer corresponding to the (t-1)th task of the current convolutional neural network. This represents the concatenated vector output of the main classification layer corresponding to the first t-1 tasks of the convolutional neural network;

[0023] The knowledge distillation equation for the main classification layer is an equation that has undergone spatial dimension pooling and planar constraints.

[0024] The step of calculating the pooling distillation loss of the sub-classification layer using LWF regularization based on the concatenation result of the sub-classification layer output vectors of the convolutional neural network includes:

[0025] The formula for the pooling distillation loss of the sub-classification layer is:

[0026]

[0027] Among them, KD S For the knowledge distillation equation of the subclassification layer, This represents the concatenation result of the output vectors of the k subclassing layers corresponding to the first t-1 tasks of the current convolutional neural network. This represents the concatenation of the output vectors of the k subclassification layers corresponding to the first t-1 tasks of the convolutional neural network;

[0028] The knowledge distillation equation for the subclassification layer is an equation that has undergone spatial dimension pooling and planar constraints.

[0029] The main classification layer cross-entropy loss function is composed of the cross-entropy of the cosine similarity of the current main classification layer and the sum of the cosine similarities of the main classification layers;

[0030] The formula for the cross-entropy loss function of the main classification layer is as follows:

[0031]

[0032] Where η is the learnable translation parameter, and δ is a decimal used to distinguish between different categories. Let f be the cosine similarity of the f-th class. Let be the cosine similarity of the i-th class.

[0033] The construction of the second loss function based on the cross-entropy of the sub-classification layer and the labeled data, as well as the classification layer difference loss function between the sub-classification layers, includes:

[0034] Calculate the cross-entropy loss function of the sub-classification layer, which is the sum of the cross-entropy of the cosine similarity of each sub-classification layer and the sum of the cosine similarities of the sub-classification layers;

[0035] Construct the classification layer difference loss function based on the differences in classification results of all the sub-classification layers;

[0036] The second loss function is constructed based on the cross-entropy loss function of the subclassification layer and the difference loss function of the classification layer.

[0037] Furthermore, before training the subclassification layer using the second loss function, the method further includes:

[0038] An additional sample dataset is introduced as a supplementary training set for the subclassification layer, and the supplementary training set and the original training set are combined to form an expanded training set; wherein, the expanded training set is used to train the subclassification layer.

[0039] This application also provides a multi-regularization strategy image classification learning device, including:

[0040] The first loss function construction module is used to construct the first loss function of the convolutional neural network based on LWF regularization and pooling knowledge distillation.

[0041] The feature extraction layer and main classification layer training module is used to train the feature extraction layer and main classification layer of the convolutional neural network using the first loss function; the formula of the first loss function is as follows:

[0042] Loss1 = Loss MW (F,M t )+λLoss LWF-Pooling ;

[0043] Among them, Loss MW The cross-entropy loss function is used for the main classification layer, F is the feature extraction layer, and M is the cross-entropy loss function. t Let λ be the main classification layer for the t-th task, and λ be the pooling adjustment parameter. Loss LWF-Pooling Let LWF be the regularized pooling distillation loss function;

[0044] The second loss function construction module is used to construct the second loss function of the convolutional neural network based on the cross-entropy loss function of the sub-classification layer and the labeled data, as well as the classification layer difference loss function between the sub-classification layers.

[0045] The subclassification layer training module is used to train the subclassification layer using the second loss function.

[0046] This application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the multi-regularization strategy image classification learning method described in any of the above claims.

[0047] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the multi-regularization strategy image classification learning method described in any of the preceding claims.

[0048] This application's multi-regularization strategy image classification learning method constructs a first loss function for a convolutional neural network (CNN) based on LWF regularization and pooling knowledge distillation. This first loss function is then used to train the feature extraction layer and the main classification layer of the CNN. A second loss function is constructed based on the cross-entropy loss function between the sub-classification layers and labeled data, as well as the classification layer difference loss function between the sub-classification layers. The sub-classification layers are then trained using the second loss function. The feature extraction layer, the main classification layer, and the sub-classification layers are used to train the CNN. Applying LWF regularization to the pooling distillation function enhances the constraint on the first loss function, resulting in a better classification performance for older tasks using the CNN trained with the first loss function. Compared to training with a single classification layer, using multiple sub-classification layers to calculate the second loss function makes the training process of the CNN more robust. Attached Figure Description

[0049] Figure 1 This is a flowchart illustrating a multi-regularization strategy image classification learning method according to one embodiment.

[0050] Figure 2 This is a schematic diagram of the process for constructing the first loss function based on pooling knowledge distillation in one embodiment;

[0051] Figure 3 This is a schematic diagram illustrating the process of constructing a second loss function according to one embodiment;

[0052] Figure 4 This is a schematic block diagram of the structure of a multi-regularization strategy image classification learning device according to an embodiment.

[0053] Figure 5 This is a schematic block diagram of the structure of a computer device according to one embodiment.

[0054] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0056] Those skilled in the art will understand that, unless explicitly stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in the specification of this application means the presence of features, integers, steps, operations, elements, units, cells, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, units, cells, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless couplings. The term “and / or” as used herein includes all or any of the units and all combinations thereof of one or more associated listed items.

[0057] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0058] Reference Figure 1 This is a flowchart illustrating the multi-regularization strategy image classification learning method proposed in this application, including:

[0059] S1: The first loss function for constructing a convolutional neural network based on LWF regularization and pooling knowledge distillation.

[0060] Based on the concatenation result of the output vectors of the main classification layer of the convolutional neural network, the pooling distillation loss of the main classification layer is calculated using LWF regularization.

[0061] Using LWF regularization to calculate the pooling distillation loss of the master classification layer can strengthen the constraint on the pooling distillation loss of the master classification layer, thereby improving the accuracy of the master classification layer in classifying old tasks while maintaining a high accuracy in classifying new tasks.

[0062] Based on the concatenation of the output vectors of the subclassification layer of the convolutional neural network, the pooling distillation loss of the subclassification layer is calculated using LWF regularization.

[0063] Using LWF regularization to calculate the pooling distillation loss of the subclassification layer can strengthen the constraint on the pooling distillation loss of the subclassification layer, thereby improving the accuracy of the subclassification layer in classifying old tasks while maintaining a high accuracy in classifying new tasks.

[0064] The LWF regularized pooling distillation loss function is calculated based on the pooling distillation loss of the main classification layer and the pooling distillation loss of the sub-classification layer.

[0065] The formula for calculating the LWF regularized pooling distillation loss function is as follows:

[0066] Loss LWF-Pooling =Loss LWF-main-pooling +Loss LWF-side-pooling ;

[0067] Among them, Loss LWF-main-pooling Main classification layer pooling distillation loss, Loss LWF-side-pooling This represents the loss from pooling distillation in the sub-classification layer;

[0068] The first loss function is constructed based on the cross-entropy loss function of the main classification layer and the LWF regularized pooling distillation loss function.

[0069] The formula for the first loss function is as follows:

[0070] Loss1 = Loss MW (F,M t )+λLoss LWF-Pooling ;

[0071] Among them, Loss MW The cross-entropy loss function is used for the main classification layer, F is the feature extraction layer, and M is the cross-entropy loss function. t Let λ be the main classification layer for the t-th task, and λ be the pooling adjustment parameter. Loss LWF-Pooling Let LWF be the regularized pooling distillation loss function.

[0072] S2: Use the first loss function to train the feature extraction layer and the main classification layer of the convolutional neural network.

[0073] Obtain N training tasks in the training process, wherein the training process includes N / T training groups, and one training group includes T training tasks;

[0074] The t-th training task includes C t There are training samples, and the input image set corresponding to the t-th training task is X. t The tag category is Y t .

[0075] The training sample set includes N t N training samples, N t Each training sample corresponds to N training tasks. By constructing a training sample set, the feature extraction layer, main classification layer, and sub-classification layer can be trained using the training sample set.

[0076] The feature extraction layer and main classification layer of the convolutional neural network are trained using the first loss function on the training sample set. The parameters of the feature extraction layer and the main classification layer are updated by backpropagation based on the first loss function.

[0077] The feature extraction layer is used to extract features from the input image, and the main classification layer is used to classify the input image. Commonly used feature extraction layers include recurrent neural networks, convolutional neural networks, and long short-term memory neural networks. Commonly used classification layers include SVM (Support Vector Machine) classification layers and softmax classification layers.

[0078] S3: Construct the second loss function of the convolutional neural network based on the cross-entropy loss function of the subclassification layer and the labeled data, as well as the classification layer difference loss function between the subclassification layers.

[0079] Calculate the cross-entropy loss function of the sub-classification layer, which is obtained by the cross-entropy between the softmax classification results of all the sub-classification layers and the labeled data;

[0080] Construct the classification layer difference loss function based on the differences in classification results of all the sub-classification layers;

[0081] The second loss function is constructed based on the cross-entropy and the classification layer difference loss function.

[0082] The cross-entropy loss function for the subclassification layer is used to measure the difference between the subclassification layer and the labeled data, while the difference loss function for the classification layer is used to measure the difference between different subclassification layers.

[0083] S4: Train the subclassification layer using the second loss function.

[0084] This causes the feature extraction layer to stop operating.

[0085] The total number of subclassification layers is K. The K subclassification layers are trained simultaneously according to the second loss function. The parameters of the K subclassification layers are updated by backpropagation according to the second loss function.

[0086] By stopping the feature extraction layer before using the second loss function to train the subclassification layer, the features extracted by the feature extraction layer can be prevented from affecting the training process of the subclassification layer.

[0087] Using the second loss function to train the subclassification layer can take into account both the classification performance of multiple subclassification layers as a whole and the differences between subclassification layers.

[0088] The feature extraction layer, the main classification layer, and the sub-classification layer are used to train the convolutional neural network.

[0089] The multi-regularization strategy image classification learning method of this application constructs a first loss function for a convolutional neural network based on LWF regularization and pooling knowledge distillation. This first loss function is used to train the feature extraction layer and the main classification layer of the convolutional neural network. A second loss function for the convolutional neural network is constructed based on the cross-entropy loss function between the sub-classification layers and the labeled data, as well as the classification layer difference loss function between the sub-classification layers. The sub-classification layers are then trained using the second loss function. The feature extraction layer, the main classification layer, and the sub-classification layers are used to train the convolutional neural network. Applying LWF regularization to the pooling distillation function enhances the constraint on the first loss function, resulting in a better classification performance for older tasks using the convolutional neural network trained with the first loss function. Compared to training with a single classification layer, using multiple sub-classification layers to calculate the second loss function makes the training process of the convolutional neural network more robust.

[0090] In one embodiment, the first loss function of the convolutional neural network is constructed based on LWF regularization and pooling knowledge distillation, referring to... Figure 2 , Figure 2 This is a schematic diagram of a process for constructing a first loss function based on pooling knowledge distillation, as shown in one embodiment, including:

[0091] S11: Based on the concatenation result of the output vectors of the main classification layer of the convolutional neural network, use LWF regularization to calculate the pooling distillation loss of the main classification layer.

[0092] The formula for the pooling distillation loss in the main classification layer is:

[0093]

[0094] Among them, KD M The knowledge distillation equation is used for the main classification layer, where σ represents the softmax classification layer. It is the i-th image in the input image sample set at time t. This represents the concatenated vector output of the main classification layer corresponding to the (t-1)th task of the current convolutional neural network. This represents the concatenated vector output of the main classification layer corresponding to the first t-1 tasks of the convolutional neural network;

[0095] The knowledge distillation equation for the main classification layer is an equation that has undergone spatial dimension pooling and planar constraints.

[0096] Spatial dimensions include width and height. By performing pooling operations on the width and height dimensions and aggregating them, the loss function that does not use pooling operations for knowledge distillation is calculated instead, making the model more flexible.

[0097] The formula for pooling the width dimension is as follows:

[0098]

[0099] Where t represents the t-th task, l represents the l-th convolutional neural network layer, c represents the number of channels, C is the total number of channels, h represents the width, H is the total width, w represents the height, W is the total height, and s l t L represents the output of the l-th layer of the convolutional neural network for the t-th task. width This indicates pooling along the width dimension.

[0100] The formula for pooling the height dimension is as follows:

[0101]

[0102] L height This indicates pooling based on the height dimension.

[0103] The formula for pooling spatial dimensions is as follows:

[0104]

[0105] Pooling the spatial dimension is the sum of pooling the width dimension and pooling the height dimension.

[0106] Based on the spatial dimension-based pooling knowledge distillation described above, a planar constraint is set on the output of the last convolutional neural network layer, and the calculation formula is as follows:

[0107]

[0108] in, This represents the output of the last layer of the convolutional neural network for the t-th task. This is the output of the last layer of the convolutional neural network for the (t-1)th task.

[0109] Ultimately, the pooling distillation loss of the main classification layer consists of two parts: spatial dimension and planar constraint, as shown in the following formula:

[0110]

[0111] Among them, KD M For the knowledge distillation equation of the main classification layer, λ c λ is a parameter used to measure spatial dimension. f λ is a parameter used to measure planar constraints. c With λ f The value is between 0 and 1.

[0112] S12: Based on the concatenation result of the output vectors of the subclassification layer of the convolutional neural network, use LWF regularization to calculate the pooling distillation loss of the subclassification layer.

[0113] The formula for the pooling distillation loss of the sub-classification layer is:

[0114]

[0115] Among them, KD S For the knowledge distillation equation of the subclassification layer, This represents the concatenation result of the output vectors of the k subclassing layers corresponding to the first t-1 tasks of the current convolutional neural network. This represents the concatenation of the output vectors of the k subclassification layers corresponding to the first t-1 tasks of the convolutional neural network;

[0116] The knowledge distillation equation for the subclassification layer is an equation that has undergone spatial dimension pooling and planar constraints.

[0117] For k subclassification layers, the output of the l-th layer convolutional neural network for the t-th task is the average of the outputs of the k subclassification layers.

[0118] The calculation methods for width-dimensional pooling, height-dimensional pooling, spatial-dimensional pooling, and planar constraints are consistent between the sub-classification layer and the main classification layer, and will not be elaborated here.

[0119] The formula for the knowledge distillation equation of the subclassification layer is as follows:

[0120]

[0121] Among them, KD S denoted as pooling distillation loss of the subclassification layer, k is the subclassification layer number, and K is the total number of subclassification layers.

[0122] S13: Calculate the LWF regularized pooling distillation loss function based on the pooling distillation loss of the main classification layer and the pooling distillation loss of the sub-classification layer.

[0123] The formula for calculating the LWF regularized pooling distillation loss function is as follows:

[0124] Loss LWF-Pooling =Loss LWF-main-pooling +Loss LWF-side-pooling ;

[0125] Among them, Loss LWF-main-pooling Main classification layer pooling distillation loss, Loss LWF-side-pooling This represents the loss from pooling distillation in the sub-classification layer;

[0126] S14: Construct the first loss function based on the main classification layer cross-entropy loss function and the LWF regularized pooling distillation loss function.

[0127] The formula for the first loss function is as follows:

[0128] Loss1 = Loss MW (F,M t )+λLoss LWF-Pooling ;

[0129] Among them, Loss MW The cross-entropy loss function is used for the main classification layer, F is the feature extraction layer, and M is the cross-entropy loss function. t Let λ be the main classification layer for the t-th task, and λ be the pooling adjustment parameter. Loss LWF-Pooling Let LWF be the regularized pooling distillation loss function;

[0130] The formula for the cross-entropy loss function of the main classification layer is as follows:

[0131]

[0132] Where η is the learnable translation parameter, and δ is a decimal used to distinguish between different categories. Let f be the cosine similarity of the f-th class. Let F be the cosine similarity of the i-th class. F is the feature extraction layer, and M is the cosine similarity. t This is the main classification layer for the t-th task.

[0133] This embodiment of the application constructs a first loss function for a convolutional neural network based on LWF regularization and pooling knowledge distillation. The pooling distillation loss of the main classification layer is calculated using LWF regularization based on the concatenation result of the output vectors of the main classification layer. Similarly, the pooling distillation loss of the sub-classification layer is calculated using LWF regularization based on the concatenation result of the output vectors of the sub-classification layer. The LWF regularized pooling distillation loss function is then calculated based on both the main classification layer and sub-classification layer pooling distillation losses. The first loss function is constructed based on the main classification layer cross-entropy loss function and the LWF regularized pooling distillation loss function. Compared to traditional knowledge distillation loss functions, pooling distillation loss functions incorporating pooling operations offer better flexibility. Using LWF regularization enhances the constraint of the pooling distillation function, enabling the convolutional neural network trained using the first loss function composed of LWF regularized pooling distillation loss functions to adapt to new tasks.

[0134] In one embodiment, a second loss function is constructed based on the cross-entropy between the subclassification layer and the labeled data, and the classification layer difference loss function between the subclassification layers. (Refer to...) Figure 3 , Figure 3A flowchart illustrating the construction of a second loss function according to one embodiment includes:

[0135] S31: Calculate the cross-entropy loss function of the subclassification layer.

[0136] The cross-entropy loss function of the sub-classification layer is calculated. The cross-entropy loss function of the sub-classification layer is obtained by the cross-entropy of the softmax classification results of all the sub-classification layers and the labeled data.

[0137] There are a total of k subclassing layers. The formula for the cross-entropy between the softmax classification result of the subclassing layer and the labeled data is as follows:

[0138]

[0139] Where η is the learnable translation parameter, and δ is a decimal used to distinguish between different categories. Let f be the cosine similarity of the f-th class. Let F be the cosine similarity of the i-th class. F is the feature extraction layer, and M is the cosine similarity. t This is the main classification layer for the t-th task.

[0140] The cross-entropy loss function of the subclassification layers represents the difference between the classification results obtained by the k subclassification layers and the categories of the labeled data. The smaller the difference, the closer the classification result obtained by the k subclassification layers as a whole is to the labeled data, and the better the classification effect of the subclassification layers.

[0141] S32: Construct the classification layer difference loss function based on the differences in classification results of all the sub-classification layers.

[0142] During the training of the K subclassification layers, a playback-based incremental learning method was adopted, introducing an additional sample dataset as the subclassification layer dataset x. out ∈X out The dataset size is N out The difference loss function between subclassification layers is calculated based on L1 regularization of the output results of each pair of subclassification layers. The formula for the difference loss function between subclassification layers is as follows:

[0143]

[0144] Where, p m (y|x out ) and p n (y|x out ) represent the softmax classification results of subclassification layers with serial numbers m and n, respectively. d represents L1 regularization, m is the number of one subclassification layer, and n is the number of the other subclassification layer.

[0145] The formula for L1 regularization of the softmax classification results of the subclassing layers with indices m and n is as follows:

[0146] d(p m (y|x out ),p n (y|x out ))=|p m (y|x out )-p n (y|x out )|;

[0147] Here, || represents the absolute value operation.

[0148] For k subclassing layers, there are a total of Each sub-category layer is paired.

[0149] The classification layer difference loss function represents the difference between different subclassification layers. The larger the difference, the smaller the coupling between the subclassification layers, and the higher the classification accuracy of multiple subclassification layers as a whole.

[0150] S33: Construct the second loss function based on the cross-entropy and the classification layer difference loss function.

[0151] The second loss function consists of the cross-entropy loss function of the subclassification layer and the difference loss function of the classification layer. The formula for the second loss function is as follows:

[0152]

[0153] in, The cross-entropy loss function for the subclassification layer is... This is the difference loss function for the classification layer.

[0154] The k subclassification layers are trained based on the second loss function. The training objective is to minimize the cross-entropy loss function of the first part and maximize the difference loss function of the second classification layer.

[0155] The second loss function provided in this application embodiment is constructed by calculating the cross-entropy loss function of the sub-classification layers, constructing the classification layer difference loss function based on the differences in classification results of all the sub-classification layers, and constructing the second loss function based on the cross-entropy loss function of the sub-classification layers and the classification layer difference loss function. The cross-entropy loss function of the sub-classification layers is used to measure the classification performance of multiple sub-classification layers as a whole, and the classification layer difference loss function is used to measure the degree of coupling between different sub-classification layers. During the training of the sub-classification layers based on the second loss function, the cross-entropy loss function of the sub-classification layers becomes smaller and smaller, while the classification layer difference loss function becomes larger and larger, thereby improving the classification performance of the sub-classification layers.

[0156] Reference Figure 4 This is a schematic block diagram of a multi-regularization strategy image classification learning device proposed in this application. The device includes:

[0157] The first loss function construction module 10 is used to construct the first loss function of the convolutional neural network based on LWF regularization and pooling knowledge distillation.

[0158] The feature extraction layer and main classification layer training module 20 is used to train the feature extraction layer and main classification layer of the convolutional neural network using the first loss function; the formula of the first loss function is as follows:

[0159] Loss1 = Loss MW (F,M t )+λLoss LWF-Pooling ;

[0160] Among them, Loss MW The cross-entropy loss function is used for the main classification layer, F is the feature extraction layer, and M is the cross-entropy loss function. t Let λ be the main classification layer for the t-th task, and λ be the pooling adjustment parameter. Loss LWF-Pooling Let LWF be the regularized pooling distillation loss function;

[0161] The second loss function construction module 30 is used to construct the second loss function of the convolutional neural network based on the cross-entropy loss function of the sub-classification layer and the labeled data, as well as the classification layer difference loss function between the sub-classification layers.

[0162] The subclassification layer training module 40 is used to train the subclassification layer using the second loss function.

[0163] The multi-regularization strategy image classification learning device of this application includes a first loss function construction module, a feature extraction layer and a main classification layer training module, a second loss function construction module, and a sub-classification layer training module. This device can implement a multi-regularization strategy image classification learning method.

[0164] Reference Figure 5 This application also provides a computer device, which may be a server, and its internal structure may be as follows: Figure 5As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor is designed to provide computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database stores parameters for the primary and secondary classification layers, etc. The network interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a multi-regularization strategy image classification learning method.

[0165] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer equipment on which the present application is applied.

[0166] One embodiment of this application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a multi-regularization strategy image classification learning method. It is understood that the computer-readable storage medium in this embodiment can be a volatile readable storage medium or a non-volatile readable storage medium.

[0167] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media provided in this application and in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual-speed SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0168] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, apparatus, article, or method. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, apparatus, article, or method that includes that element.

[0169] The above description is only a preferred embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural changes made based on the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A multi-regularization strategy image classification learning method, characterized in that, include: The first loss function of the convolutional neural network is constructed based on LWF regularization and pooling knowledge distillation. This first loss function is then used to train the feature extraction layer and the main classification layer of the convolutional neural network. The formula for the first loss function is as follows: Loss1=Loss MW (F,M t )+λLoss LWF-Pooling ; Among them, Loss MW The cross-entropy loss function is used for the main classification layer, F is the feature extraction layer, and M is the cross-entropy loss function. t Let λ be the main classification layer for the t-th task, and λ be the pooling adjustment parameter. Loss LWF-Pooling The LWF regularized pooling distillation loss function is used. The construction of the LWF regularized pooling distillation loss function includes: calculating the pooling distillation loss of the main classification layer using LWF regularization based on the concatenation result of the main classification layer output vectors, where the main classification layer knowledge distillation equation is an equation that has undergone spatial dimension pooling operations and planar constraints. The spatial dimension pooling operations include pooling and aggregating the width and height dimensions respectively, and the planar constraints are L2 norm constraints on the output of the last convolutional neural network layer; calculating the pooling distillation loss of the sub-classification layer using LWF regularization based on the concatenation result of the sub-classification layer output vectors, where the sub-classification layer knowledge distillation equation has also undergone spatial dimension pooling operations and planar constraints. A second loss function for the convolutional neural network is constructed based on the cross-entropy loss function of the subclassification layer and the labeled data, as well as the classification layer difference loss function between the subclassification layers. The subclassification layer is then trained using the second loss function, wherein the feature extraction layer is stopped before the subclassification layer is trained using the second loss function. The feature extraction layer, the main classification layer, and the sub-classification layer are used to train the convolutional neural network; The construction of the second loss function based on the cross-entropy of the sub-classification layer and the labeled data, as well as the classification layer difference loss function between the sub-classification layers, includes: Calculate the cross-entropy loss function of the sub-classification layer, which is the sum of the cross-entropy of the cosine similarity of each sub-classification layer and the sum of the cosine similarities of the sub-classification layers; The difference loss function between the sub-classification layers is constructed based on the differences in classification results of all sub-classification layers. Referring to the incremental learning method based on playback, an additional sample dataset is introduced as a supplementary training set for the sub-classification layers. This supplementary training set and the original training set form an expanded training set, which is used to train the sub-classification layers. The calculation of the difference loss function between sub-classification layers is based on L1 regularization of the output results of each pair of sub-classification layers. The formula for the difference loss function between sub-classification layers is as follows: Where, p m (y|x out ) and p n (y|x out ) represent the softmax classification results of subclassing layers with serial numbers m and n, respectively, where d represents L1 regularization, m is the number of one subclassing layer, and n is the number of the other subclassing layer; The formula for L1 regularization of the softmax classification results of the subclassing layers with indices m and n is as follows: d(p m (y|x out ),p n (y|x out ))=|p m (y|x out )-p n (y|x out )|; Where || represents the absolute value operation; For k subclassing layers, there are a total of One sub-category layer pair; The second loss function is constructed based on the cross-entropy loss function of the subclassification layer and the difference loss function of the classification layer.

2. The multi-regularization strategy image classification learning method according to claim 1, characterized in that, The first loss function for constructing the convolutional neural network based on LWF regularization and pooling knowledge distillation includes: Based on the concatenation result of the output vectors of the main classification layer of the convolutional neural network, the pooling distillation loss of the main classification layer is calculated using LWF regularization. Based on the concatenation result of the output vectors of the subclassification layer of the convolutional neural network, the pooling distillation loss of the subclassification layer is calculated using LWF regularization; The LWF regularized pooling distillation loss function is calculated based on the pooling distillation loss of the main classification layer and the pooling distillation loss of the sub-classification layer. The formula for calculating the LWF regularized pooling distillation loss function is as follows: Loss LWF-Pooling =Loss LWF-main-pooling +Loss LWF-side-pooling ; Among them, Loss LWF-main-pooling Main classification layer pooling distillation loss, Loss LWF-side-pooling This represents the loss from pooling distillation in the sub-classification layer; The first loss function is constructed based on the cross-entropy loss function of the main classification layer and the LWF regularized pooling distillation loss function.

3. The multi-regularization strategy image classification learning method according to claim 2, characterized in that, The step of calculating the pooling distillation loss of the main classification layer using LWF regularization based on the concatenation result of the main classification layer output vector of the convolutional neural network includes: The formula for the pooling distillation loss in the main classification layer is: Among them, KD M The knowledge distillation equation is used for the main classification layer, where σ represents the softmax classification layer. It is the i-th image in the input image sample set at time t. This represents the concatenated vector output of the main classification layer corresponding to the (t-1)th task of the current convolutional neural network. This represents the concatenated vectors of the outputs of the main classification layer corresponding to the first t-1 tasks of the convolutional neural network; The knowledge distillation equation for the main classification layer is an equation that has undergone spatial dimension pooling and planar constraints.

4. The multi-regularization strategy image classification learning method according to claim 2, characterized in that, The step of calculating the pooling distillation loss of the sub-classification layer using LWF regularization based on the concatenation result of the sub-classification layer output vectors of the convolutional neural network includes: The formula for the pooling distillation loss of the sub-classification layer is: Among them, KD S For the knowledge distillation equation of the subclassification layer, This represents the concatenation result of the output vectors of the k subclassing layers corresponding to the first t-1 tasks of the current convolutional neural network. This represents the concatenation of the output vectors of the k subclassification layers corresponding to the first t-1 tasks of the convolutional neural network; The knowledge distillation equation for the subclassification layer is an equation that has undergone spatial dimension pooling and planar constraints.

5. The multi-regularization strategy image classification learning method according to claim 1, characterized in that, The main classification layer cross-entropy loss function is composed of the cross-entropy of the cosine similarity of the current main classification layer and the sum of the cosine similarities of the main classification layers; The formula for the cross-entropy loss function of the main classification layer is as follows: Where η is the learnable translation parameter, and δ is a decimal used to distinguish between different categories. Let f be the cosine similarity of the f-th class. Let be the cosine similarity of the i-th class.

6. A multi-regularization strategy image classification learning apparatus, used to execute the multi-regularization strategy image classification learning method as described in any one of claims 1-5, characterized in that, include: The first loss function construction module is used to construct the first loss function of the convolutional neural network based on LWF regularization and pooling knowledge distillation. The feature extraction layer and main classification layer training module is used to train the feature extraction layer and main classification layer of the convolutional neural network using the first loss function; the formula of the first loss function is as follows: Loss1=Loss MW (F,M t )+λLoss LWF-Pooling ; Among them, Loss MW The cross-entropy loss function is used for the main classification layer, F is the feature extraction layer, and M is the cross-entropy loss function. t Let λ be the main classification layer for the t-th task, and λ be the pooling adjustment parameter. Loss LWF-Pooling Let LWF be the regularized pooling distillation loss function; The second loss function construction module is used to construct the second loss function of the convolutional neural network based on the cross-entropy loss function of the sub-classification layer and the labeled data, as well as the classification layer difference loss function between the sub-classification layers. The second loss function of the convolutional neural network, constructed based on the cross-entropy loss function of the sub-classification layer and the labeled data, and the classification layer difference loss function between the sub-classification layers, includes: Calculate the cross-entropy loss function of the sub-classification layer, which is the sum of the cross-entropy of the cosine similarity of each sub-classification layer and the sum of the cosine similarities of the sub-classification layers; Based on the differences in classification results across all sub-classification layers, a classification layer difference loss function is constructed. Referring to the playback-based incremental learning method, an additional sample dataset is introduced as the sub-classification layer dataset x. out ∈X out The dataset size is N out The difference loss function between subclassification layers is calculated based on L1 regularization of the output results of each pair of subclassification layers. The formula for the difference loss function between subclassification layers is as follows: Where, p m (y|x out ) and p n (y|x out ) represent the softmax classification results of subclassing layers with serial numbers m and n, respectively, where d represents L1 regularization, m is the number of one subclassing layer, and n is the number of the other subclassing layer; The formula for L1 regularization of the softmax classification results of the subclassing layers with indices m and n is as follows: d(p m (y|x out ),p n (y|x out ))=|p m (y|x out )-p n (y|x out )|; Where || represents the absolute value operation; For k subclassing layers, there are a total of One subcategorization layer pair; The second loss function is constructed based on the cross-entropy loss function of the subclassification layer and the difference loss function of the classification layer; The subclassification layer training module is used to train the subclassification layer using the second loss function.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the multi-regularization strategy image classification learning method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the multi-regularization strategy image classification learning method as described in any one of claims 1 to 5.

Images