A deep learning model training method considering multiple feature regions

Abstract

The application discloses a deep learning model training method considering multiple feature regions, which comprises the following steps: loading an image I into a deep learning model, initially training the model and updating the model parameters; stacking the multi-dimensional feature maps output by the feature extractor in the model training process to form a stacked heat map; generating a mask based on the stacked heat map and shielding the key attention area in the image I by using the mask; replacing the original image I with the shielded image I and loading the image I into the deep learning model with updated parameters to retrain the model and update the model parameters; repeating the above steps once or multiple times, switching another image for the same training, until all the images in the training set image library are input, the model training is completed, and the model parameters are stored; the method aims to improve the generalization ability and robustness of the deep learning model based on the attention mechanism, considers multiple regional features, and avoids missing identification.

Description

Technical Field

[0001] This invention relates to the field of deep learning, and more specifically to a method for training deep learning models that takes into account multiple feature regions. Background Technology

[0002] Attention mechanisms originate from attentional thinking, which refers to the human ability to quickly sift through a large amount of information using limited attentional resources. Currently, attention mechanisms are incorporated into deep learning models, drawing on human attentional thinking methods, and are widely used in various deep learning tasks such as image classification and speech recognition. Attention mechanisms can quickly help models identify key areas of interest, leading to faster conclusions. This type of model has achieved significant results in single feature recognition or key feature recognition, improving recognition accuracy. However, deep learning models based on attention mechanisms have the following problems:

[0003] 1) Prone to feature dependence: Attention-based models are prone to over-reliance on a certain type of feature (e.g., color, shape, or texture) during training. They may ignore other equally important features or even develop a preference for a certain feature, leading to overfitting and recognition errors.

[0004] 2) Imbalanced attention allocation: When the object to be identified occupies a large proportion of the image, or when the key features are distributed in a relatively scattered manner, deep learning models based on attention mechanisms often only focus on the most discriminative feature regions, without giving sufficient attention to other feature regions, thus affecting the overall accuracy of recognition. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a deep learning model training method that considers multiple feature regions. This method aims to improve the generalization ability and robustness of attention-based deep learning models, avoiding missed recognitions.

[0006] The technical solution is as follows:

[0007] A deep learning model training method that takes into account multiple feature regions includes the following steps:

[0008] S1. Load the labeled image I into the deep learning model, perform initial training on the model, and update the model parameters;

[0009] S2. Stack the multi-dimensional feature maps output by the feature extractor during model training to form a stacked heatmap;

[0010] S3. Generate a mask based on the stacked heatmap, and use the mask to mask the key areas of interest in image I;

[0011] Replace the original image I with the masked image I and load it into the deep learning model that has completed parameter updates. Then, retrain the model and update the model parameters.

[0012] S4. Repeat steps S2 to S3 once or more, then proceed to step S5.

[0013] S5. Switch to another labeled image from the training set, denoted as image I, and use it to execute step S1 until all images in the training set image library have been input. The model training is then complete, and the model parameters are stored.

[0014] Furthermore, in step S2, the multi-dimensional feature maps output by the feature extractor during model training are stacked to form a stacked heatmap in the following way:

[0015]

[0016] A(i,j) represents the stacked heatmap, F k (i,j) represents the k-th dimension feature map output by any convolutional layer of the feature extractor, where k = 1, 2, ..., n, and n represents the dimension of the feature map output by the feature extractor. This means accumulating the pixel values ​​at the same position (i,j) in the 1-dimensional to n-dimensional feature maps.

[0017] Preferred, F k (i,j) represents the k-th dimension feature map output by the last convolutional layer of the feature extractor.

[0018] Furthermore, in step S3, a mask is generated based on the stacked heatmap, and the key areas of interest in image I are masked using the mask as follows:

[0019] ① Take the average value of each pixel in the stacked heatmap and denote it as the threshold.

[0020] ② Pixels with values ​​greater than a threshold in the stacked heatmap The pixel values ​​of certain pixels are set to 0, and the pixel values ​​of other pixels are set to 1 to generate a mask;

[0021] ③ Upsample the mask to the size of image I, and then multiply it by the size of image I to mask the key areas of interest in image I.

[0022] In order to constrain the model’s focus area to a more precise region, more preferably, in step S2, the dimensional feature maps output by different convolutional layers of the feature extractor are stacked to form multiple stacked heatmaps.

[0023] In step S3, masks are generated based on multiple stacked heatmaps to obtain multiple masks. The multiple masks are unified to the same size, and their intersection is taken to generate the final mask. The final mask is used to mask the key areas of interest in image I.

[0024] Preferably, steps S2 to S3 are performed 2 to 4 times.

[0025] This method has the following characteristics:

[0026] This method is designed for deep learning models based on attention mechanisms, such as E-Net, SE-ResNet, and SE-ResNeXt, or networks that embed SE (Squeeze and Excitation) and CBAM (Convolutional Block Attention Module) modules.

[0027] By masking the key areas of interest in the input image and then re-inputting the masked image into the model for training, the model is forced to focus on other potentially overlooked areas, thus suppressing the model's excessive focus on the most distinctive local features in the image.

[0028] The model parameters trained using this method are more able to focus on global image features and take into account features from multiple regions, thereby improving the model's generalization ability and robustness and avoiding missed recognition. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of the hand bone image training process in a specific implementation method;

[0030] Figure 2 This is a schematic diagram of stacking multidimensional feature maps to form a stacked heatmap. Detailed Implementation

[0031] The technical solution of the present invention will be described in detail below with reference to the embodiments.

[0032] The attention mechanism in deep learning allows neural networks to focus their attention on relevant parts of the input data. In traditional neural networks, the output of each neuron depends only on the outputs of all neurons in the previous layer. However, in the attention mechanism, the output of each neuron not only depends on the outputs of all neurons in the previous layer but can also be weighted according to different parts of the input data, i.e., different weights are assigned to different parts. This allows the model to pay more attention to key information in the input sequence, thereby improving the model's accuracy and efficiency.

[0033] However, deep learning models based on attention mechanisms often focus too much on a certain region in an image, thus easily ignoring other feature regions and leading to missed recognition, especially in the following situations:

[0034] Scenario 1: The object to be identified in the image has obvious feature preferences. For example, the color feature of the target to be identified is more obvious. In this case, the model may pay too much attention to the color feature information and ignore other feature information. When other targets with the same color feature appear, misidentification will easily occur.

[0035] Scenario 2: The key features of the object to be identified are relatively scattered. For example, large images often contain many features, and these features are distributed in multiple regions of the image. Deep learning models based on attention mechanisms often only focus on the few most discriminative regions and do not give enough attention to other important regions, which leads to a significant reduction in recognition accuracy.

[0036] To address the above-mentioned issues, this invention proposes a deep learning model training method that takes into account multiple feature regions. The model trained by this method is better able to focus on global features of the image, avoiding missed or false recognition.

[0037] This method first acquires images of the test object at different times and angles (sample images are acquired for the same type of test object, such as images of hand bones from different people) and labels the features in the images to obtain a training set;

[0038] The specific steps for model training are as follows:

[0039] S1. Load the labeled image I into the deep learning model, perform initial training on the model, and update the model parameters;

[0040] S2. Stack the multi-dimensional feature maps output by the feature extractor during model training to form a stacked heatmap;

[0041] In practice, the stacking method is as follows:

[0042]

[0043] A(i,j) represents the stacked heatmap, F k (i,j) represents the k-th dimension feature map output by any convolutional layer of the feature extractor, where k = 1, 2, ..., n, and n represents the dimension of the feature map output by the feature extractor. This means accumulating the pixel values ​​at the same position (i,j) in the 1-dimensional to n-dimensional feature maps.

[0044] S3. Generate a mask based on the stacked heatmap, and use the mask to mask the key areas of interest in image I; replace the original image I with the masked image I and load it into the deep learning model that has completed parameter updates, retrain the model, and update the model parameters.

[0045] In practice, step S3 is as follows:

[0046] ① Take the average value of each pixel in the stacked heatmap and denote it as the threshold.

[0047] ② Pixels with values ​​greater than a threshold in the stacked heatmap The pixel value of the i-th pixel is set to 0, and the pixel value of the other pixels is set to 1, generating a mask M(i,j):

[0048]

[0049] ③ Upsample the mask M(i,j) to the size of image I, and then multiply it by image I to mask the key areas of interest in image I.

[0050] S4. After repeating steps S2 to S3 once or more (preferably, repeating steps S2 to S3 1 to 4 times), proceed to step S5.

[0051] S5. Switch to another labeled image from the training set, denoted as image I, and use it to execute step S1. Continue this process until all images in the training set image library have been input, at which point the model training is complete, and the model parameters are stored. Note that when step S1 is executed again, the model parameters have already been updated during the training process and are no longer the initial model parameters.

[0052] In feature extractors, deeper network layers result in higher feature map dimensions and larger receptive fields, allowing each point to "see" a larger area of ​​the input image. To constrain the model's focus area to a more precise region, a preferred implementation method is to generate a mask using the following approach:

[0053] In step S2, the dimensional feature maps output by different convolutional layers of the feature extractor are stacked to form multiple stacked heatmaps. In step S3, masks are generated based on the multiple stacked heatmaps to obtain multiple masks. The multiple masks are unified to the same size, and their intersection is taken to generate the final mask. The final mask is used to mask the key areas of interest in image I.

[0054] The following example uses hand bone feature recognition. Figure 1 As shown, an illustrative example will be provided:

[0055] S1. Load the labeled hand bone image I into the deep learning model. The model performs data augmentation, which includes: flipping, rotating, deforming and resizing the image, adding noise and / or affine transformation, and normalizing and standardizing the image. Finally, the image is used as the input image of the model.

[0056] Perform initial training on the model and update the model parameters;

[0057] S2, such as Figure 2 As shown, during the initial training process, the multidimensional feature maps output by the last convolutional layer from the feature extractor are selected and stacked to form a stacked heatmap; where the feature extractor is the convolutional layer before the pooling layer and the classification layer.

[0058] In this embodiment, in order to constrain the model's focus area to a more precise region, the mask is generated using the following method:

[0059] The dimensional feature maps output by the penultimate and last convolutional layers of the feature extractor are stacked to form two stacked heatmaps, A1(i,j) and A2(i,j).

[0060]

[0061] In practice, the feature maps output by different convolutional layers can be stacked according to the actual situation. For example, the feature maps output by the 2nd, 3rd and 6th convolutional layers can be stacked to form 3 stacked heatmaps.

[0062] S3. Generate masks based on multiple stacked heatmaps to obtain multiple masks:

[0063] like Figure 2 As shown, the specific steps are as follows:

[0064] ① Take the average value of each pixel in the stacked heatmaps A1(i,j) and A2(i,j) respectively, and denote it as the threshold.

[0065] ② In the stacked heatmap A1(i,j), pixels with values ​​greater than the threshold are... Set the pixel value of the i-th pixel to 0 and the pixel value of the other pixels to 1 to generate a mask M1(i,j):

[0066]

[0067] Pixels with values ​​greater than the threshold in the stacked heatmap A2(i,j) are considered. Set the pixel value of the i-th pixel to 0 and the pixel value of the other pixels to 1 to generate a mask M2(i,j):

[0068]

[0069] Unify the two masks M1(i,j) and M2(i,j) to the same size (downsampling / upsampling), and then take their intersection to generate the final mask M(i,j):

[0070] M(i,j)=M1(i,j)∩M2(i,j)

[0071] ③ Upsample the mask M(i,j) to the size of image I, and then multiply it by image I to mask the key areas of interest in image I;

[0072] Replace the original image I with the masked image I and load it into the deep learning model that has completed parameter updates (initial training is complete). Perform a second training on the model and update the model parameters.

[0073] S4. Steps S2 to S3 are performed again (in this embodiment, they are executed twice in total; in actual applications, the number of executions can be set according to actual needs).

[0074] The specific implementation is as follows:

[0075] S2. During the second model training process, the multi-dimensional feature maps output by the feature extractor are stacked to form a stacked heatmap.

[0076] S3. Generate a mask based on the stacked heatmap, and use the mask to mask the key areas of interest in the new image I;

[0077] Replace the original image I with the masked image I and load it into the deep learning model that has completed parameter updates (second model training is complete). Perform a third training on the model and update the model parameters.

[0078] After the third training session is completed, proceed to step S5;

[0079] S5. Switch to another labeled image from the training set, denoted as image I, and use it to execute step S1 until all images in the training set image library have been input. The model training is then complete, and the model parameters are stored.

[0080] like Figure 1As can be seen, after initial training, the model only focuses on the bones of the palm and some fingers (local region features), without covering the entire hand bone region, resulting in inaccurate hand bone recognition. In this method, during training, the feature maps output by the model's feature extractor from the previous training are stacked to form a stacked heatmap. The key areas of interest in the input image are masked and then trained again. That is, a hand bone image is trained three times: the input for the first training is the original image, the input for the second training is the masked image, and the input for the third training is the image after being masked again based on the previous mask. This training method forces the model to pay attention to other feature regions that are not necessarily of interest, making the final model parameters more reasonable and able to pay attention to the features of the entire hand bone.

[0081] The foregoing description of specific exemplary embodiments of the present invention is for illustrative and descriptive purposes. It is not intended to be exhaustive, nor to limit the invention to the precise forms disclosed; obviously, many changes and variations are possible in accordance with the foregoing teachings. The exemplary embodiments were chosen and described to explain the specific principles of the invention and its practical application, thereby enabling others skilled in the art to implement and utilize various exemplary embodiments of the invention, as well as their different alternatives and modifications. The scope of the invention is intended to be defined by the appended claims and their equivalents.

Claims

1. A deep learning model training method that takes into account multiple feature regions, characterized in that, Includes the following steps: S1. Load the labeled image I into the deep learning model, perform initial training on the model, and update the model parameters; S2. During model training, the multidimensional feature maps output by the last convolutional layer of the feature extractor are stacked to form a stacked heatmap; or, the multidimensional feature maps output by different convolutional layers of the feature extractor are stacked separately to form multiple stacked heatmaps. Among them, stacked heatmap The calculation method is as follows: Let represent the k-th dimension feature map output by any convolutional layer of the feature extractor, where k = 1, 2, ..., n, and n represents the dimension of the feature map output by the feature extractor. The same position in the 1D to nD feature map Accumulate the pixel values ​​at each location; S3. Generate a mask based on the stacked heatmap, and use the mask to mask the key areas of interest in image I. When there are multiple stacked heatmaps, generate masks based on the multiple stacked heatmaps respectively to obtain multiple masks. Unify the multiple masks to the same size, and then take their intersection to generate the final mask. Use the final mask to mask the key areas of interest in image I. Replace the original image I with the masked image I and load it into the deep learning model that has completed parameter updates. Then, retrain the model and update the model parameters. S4. Repeat steps S2 to S3 once or more, then proceed to step S5. S5. Switch to another labeled image from the training set, denoted as image I, and use it to execute step S1 until all images in the training set image library have been input. The model training is then complete, and the model parameters are stored.

2. The deep learning model training method that takes into account multiple feature regions as described in claim 1, characterized in that: In step S3, a mask is generated based on the stacked heatmap. The mask is used to mask the key areas of interest in image I as follows: ① Take the average value of each pixel in the stacked heatmap and denote it as the threshold. ; ② Pixels with values ​​greater than a threshold in the stacked heatmap The pixel values ​​of certain pixels are set to 0, and the pixel values ​​of other pixels are set to 1 to generate a mask; ③ Upsample the mask to the size of image I, and then multiply it by the size of image I to mask the key areas of interest in image I.

3. The deep learning model training method that takes into account multiple feature regions as described in claim 1, characterized in that: Perform steps S2 and S3 2 to 4 times.

4. The deep learning model training method that takes into account multiple feature regions as described in claim 1, characterized in that: Before step S1, images of the object to be tested collected at different times and from different angles are compiled and the features in the images are labeled to obtain a training set.

Images