A method for training classification networks with the help of heatmap
By introducing heatmap-assisted training mechanism into deep learning convolutional neural networks, sharing weights and weighting loss values, the overfitting problem caused by fewer training samples is solved, improving the generalization ability of the classification network and saving computing resources.
Patent Information
- Application Number
- CN202010662452.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-07-10
- Publication Date
- 2024-12-17
- Estimated Expiration
- 2040-07-10
AI Technical Summary
Existing deep learning convolutional neural networks are prone to overfitting when there are fewer training samples, resulting in poor generalization capabilities of the model.
A method of assisting in training classification networks with heatmap is adopted. By setting up a network structure of two branches, where the left branch is a fully connected network to predict classification results, and the right branch is used to generate heatmap, heatmap and classification network share weights, during the training process Weighted addition of loss values to assist in training, and only the network used for training is removed during inference to save calculations.
It effectively improves the generalization ability of the classification network, improves the generalization ability of the model with fewer training samples, and saves calculation amount and improves efficiency in the inference process.
Smart Images
Figure CN113919491B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of convolutional neural networks, and in particular to a method for training a classification network with the aid of heatmap. Background Art
[0002] With the development of science and technology, deep learning of convolutional neural networks is increasingly being used. Existing deep learning convolutional neural networks include classification networks with fully connected layers and fully convolutional networks for image segmentation without fully connected layers.
[0003] However, existing deep learning neural network training is highly dependent on training samples. When there are fewer training samples, overfitting is prone to occur, resulting in poor model generalization ability.
[0004] Common terms in the prior art are as follows:
[0005] 1. Convolutional neural network, a feature extractor consisting of convolutional layers and pooling layers.
[0006] 2. Heatmap, heat map, here refers to the manually calibrated picture that corresponds to the training samples.
[0007] 3. Fully connected layer, each neuron is connected to all neurons in the previous layer. Summary of the invention
[0008] In order to solve the above problems, the purpose of the present invention is to propose a new deep learning training method, which can improve the generalization ability of the classification network when the training samples are limited.
[0009] Specifically, the present invention provides a method for training a classification network with the aid of a heatmap, the method comprising the following steps:
[0010] Step 1: The network sets two branches in the middle, where the left branch uses a fully connected network to predict the classification results;
[0011] Step 2: The right branch is designed as a network used only for training;
[0012] Step 3: Create a heatmap in the network used only for training on the right branch, mark the locations of the common feature areas in the sample images, and map them to a heatmap;
[0013] Step 4, make output layer labels in the left branch network;
[0014] Step 5: Train the network. After weighted addition of the heatmap loss value and the output layer loss value, train the network. Because the heatmap network and the classification network share weights in the first half of the network before branching, the image features learned by the heatmap will benefit the classification network.
[0015] Step 6: When using the trained network for inference, remove the network used only for training to save computational effort.
[0016] The heatmap network is the right branch after the branch, and the classification network is the left branch after the branch. The output of the first half of the network is used as the input of the latter two networks, the heatmap network and the classification network. The heatmap network and the classification network are in a parallel relationship and do not depend on each other and do not affect each other. In the process of network learning, the heatmap network and the classification network jointly affect the weight data of the first half of the network.
[0017] The method generates a heatmap having a corresponding relationship with the training samples through manually determining and calibrating the required learning image features, so as to assist the training.
[0018] The fully connected network including the left branch in step 1 further includes: Input layer, C1 convolution layer, S2 pooling layer, C3 convolution layer, S4 pooling layer, C5 convolution layer, F6 fully connected layer, output layer.
[0019] The right branch of step 2 uses upsampling and convolutional networks to predict the heatmap of the image, and there is a corresponding relationship between the heatmap and the training sample image.
[0020] There is a corresponding relationship between the heatmap and the training sample image, and the size is 1:1 or geometrically mapped according to the ratio actually required.
[0021] The step 3 further comprises:
[0022] When the classification network detects whether there is a specific area in the image, the edge information of the specific area is important information, while the background and other content except the edge of the specific area are information that can be ignored. The area at the edge of the specific area is defined as a common feature area, which is the pattern feature that the neural network needs to learn. Therefore, the edge area of the specific area in the sample image is mapped to the heatmap, that is, the area containing important information is highlighted. If there is no specific area in the image, the heatmap pattern is a completely black image.
[0023] The step 4 of making the output layer label further includes:
[0024] To determine whether a picture contains a specific area, the pictures containing the specific area can be classified into one category, and the pictures not containing the specific area can be classified into another category.
[0025] Therefore, the advantages of this application are: by manually determining and calibrating the required learning image features, a heatmap with a corresponding relationship with the training samples is generated for auxiliary training, which effectively improves the generalization ability of the classification network when there are fewer training samples. In addition, an auxiliary feature learning network is added during the training process to assist the network in learning image features, and the corresponding calculations are removed during the reasoning process, saving calculations and improving efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention.
[0027] Figure 1 It is a schematic flow chart of the method of the present invention.
[0028] Figure 2 The figure is a schematic diagram of a network structure to which the method of the present invention is applied.
[0029] Figure 3 This is a picture example of a situation in step 3 of the specific implementation example.
[0030] Figure 4 This is a picture example of another situation in step 3 of the specific embodiment. DETAILED DESCRIPTION
[0031] In order to more clearly understand the technical content and advantages of the present invention, the present invention is now further described in detail in conjunction with the accompanying drawings.
[0032] like Figure 1 As shown, the present invention relates to a method for training a classification network with the aid of heatmap, the method comprising the following steps:
[0033] Step 1: The network sets two branches in the middle, where the left branch uses a fully connected network to predict the classification results;
[0034] Step 2: The right branch is designed as a network used only for training;
[0035] Step 3: Create a heatmap in the network used only for training on the right branch, mark the locations of the common feature areas in the sample images, and map them to a heatmap;
[0036] Step 4, make output layer labels in the left branch network;
[0037] Step 5: Train the network. After weighted addition of the heatmap loss value and the output layer loss value, train the network. Because the heatmap network and the classification network share weights in the first half of the network before branching, the image features learned by the heatmap will benefit the classification network.
[0038] Step 6: When using the trained network for inference, remove the network used only for training.
[0039] like Figure 2 As shown, it can be seen from the network structure of the method of the present invention that the network has a branch in the middle. The left branch uses a fully connected network to predict the classification result, and the right branch uses upsampling and convolutional networks to predict the heatmap of the image. There is a corresponding relationship between the heatmap and the training sample image, and the size can be a geometric mapping of 1:1 or other ratios.
[0040] Furthermore, in step 3, a heatmap is created to mark the locations of the common feature areas in the sample images and map them to a heatmap. Figure 3 As shown in the figure, for example, a classification network detects whether there is paper in an image. The edge information of the paper is important information, while the background and the person in the paper are negligible information. The area around the edge of the paper is defined as a common feature area, which is the pattern feature that the neural network needs to learn. Figure 3 The edge area of the paper in the sample image on the middle left is mapped to the heatmap, that is Figure 3 The right half contains the highlighted area containing important information. If there is no paper in the pattern, the heatmap pattern will be a completely black picture. Figure 4 shown.
[0041] Furthermore, in step 4, an output layer label is produced. For example, if we determine whether this image contains a piece of paper, the image containing paper can be classified into one category, and the image not containing paper can be classified into another category.
[0042] Furthermore, in step 5, the network is trained, and the loss value of the heatmap and the loss value of the output layer are weighted and added, and then the network is trained. Because in the first half, the heatmap network and the classification network share weights, the image features learned by the heatmap will benefit the classification network, preventing the network from learning other irrelevant features and enhancing the generalization ability of the network.
[0043] Finally, in step 6, when using the trained network for inference, remove Figure 2 Only the network used in training is used to save computation.
[0044] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A method for training a classification network with the aid of a heatmap, characterized in that: The method comprises the following steps: Step 1: The network sets two branches in the middle, where the left branch uses a fully connected network to predict the classification results; Step 2: The right branch is designed as a network used only for training; Step 3: Create a heatmap in the network used only for training on the right branch, mark the locations of the common feature areas in the sample images, and map them to a heatmap; When the classification network detects whether there is a specific area in the image, the edge information of the specific area is important information, while the background and other content except the edge of the specific area are information that can be ignored. The area at the edge of the specific area is defined as a common feature area, which is the pattern feature that the neural network needs to learn. Therefore, the edge area of the specific area in the sample image is mapped to the heatmap, that is, the area containing important information is highlighted. If there is no specific area in the image, the heatmap pattern is a completely black image. Step 4, make output layer labels in the left branch network; Step 5: Train the network. After weighted addition of the heatmap loss value and the output layer loss value, train the network. Because the heatmap network and the classification network share weights in the first half of the network before branching, the image features learned by the heatmap will benefit the classification network. The heatmap network is the right branch after the branch, and the classification network is the left branch after the branch. The output of the first half of the network is used as the input of the latter two networks, the heatmap network and the classification network. The heatmap network and the classification network are in a parallel relationship and do not depend on each other and do not affect each other. In the process of network learning, the heatmap network and the classification network jointly affect the weight data of the first half of the network; Step 6: When using the trained network for inference, remove the network used only for training.
2. The method of claim 1, wherein heatmap is used to assist in training a classification network. The method generates a heatmap having a corresponding relationship with the training samples through manually determining and calibrating the required learning image features, so as to assist the training.
3. The method for training a classification network with the aid of heatmap according to claim 1, characterized in that: The fully connected network including the left branch in step 1 further includes: Input layer, C1 convolution layer, S2 pooling layer, C3 convolution layer, S4 pooling layer, C5 convolution layer, F6 fully connected layer, output layer.
4. The method of claim 1, wherein heatmap is used to assist in training a classification network. The right branch of step 2 uses upsampling and convolutional networks to predict the heatmap of the image, and there is a corresponding relationship between the heatmap and the training sample image.
5. The method of training a classification network with the aid of heatmap according to claim 4, characterized in that: There is a corresponding relationship between the heatmap and the training sample image, and the size is 1:1 or geometrically mapped according to the ratio actually required.
6. The method of claim 1, wherein heatmap is used to assist in training a classification network. The step 4 of making the output layer label further includes: To determine whether a picture contains a specific area, the pictures containing the specific area can be classified into one category, and the pictures not containing the specific area can be classified into another category.