Deep learning model training method considering multiple feature regions

By generating key areas of interest in the masked image within the deep learning model, the model is forced to focus on multi-feature regions, thus solving the feature dependency and attention allocation imbalance problems of attention mechanism models and improving recognition accuracy and robustness.

CN119027783AActive Publication Date: 2024-11-26EASY THINKING HANGZHOU TECH CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202411124278.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-16
Publication Date
2024-11-26
Estimated Expiration
2044-08-16

AI Technical Summary

Technical Problem

Deep learning models based on attention mechanisms are prone to feature dependence and imbalance in attention allocation, leading to recognition errors and decreased accuracy.

Method used

By generating masks to mask out key areas of interest in an image, the model is forced to focus on other potentially overlooked areas. By using stacked heatmaps and masks to train the image multiple times, the model's attention to multi-feature regions is improved.

Benefits of technology

This improved the model's generalization ability and robustness, avoided missed identifications, and enhanced recognition accuracy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027783A_ABST
    Figure CN119027783A_ABST
Patent Text Reader

Abstract

The invention discloses a deep learning model training method considering multiple feature regions, and the method comprises the steps: loading an image I into a deep learning model, carrying out the initial training of the model, and updating the parameters of the model; stacking the multi-dimensional feature maps output by the feature extractor in the model training process to form a stacked thermodynamic diagram; generating a mask on the basis of the stacked thermodynamic diagram, and performing shielding processing on a key region of interest in the image I by using the mask; replacing the original image I with the shielded image I, loading the image I into the deep learning model of which the parameters are updated, training the model again, and updating the model parameters; repeating the above steps for one or more times, switching another image to carry out the same training until all the images in the training set image library are input, completing model training, and storing model parameters; the method aims at improving generalization ability and robustness of a deep learning model based on an attention mechanism, multi-region features are considered, and missed recognition is avoided.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of deep learning, and in particular to a deep learning model training method that takes into account multiple feature areas. Background Art

[0002] The attention mechanism originates from attention thinking, which refers to the ability of humans to quickly filter out high-value information from a large amount of information using limited attention resources. At present, the attention mechanism is combined with deep learning models. It draws on the processing method of human attention thinking and is widely used in various types of deep learning tasks such as image classification and speech recognition. The attention mechanism can quickly help the model filter out key areas of focus and then quickly draw conclusions. The model has achieved remarkable results in single feature recognition or key feature recognition, and has improved recognition accuracy. However, the deep learning model based on the attention mechanism has the following problems:

[0003] 1) Prone to feature dependence: Models based on the attention mechanism tend to be overly dependent on a certain type of feature (for example, color, shape, or texture) during training. They may ignore other equally important features or even develop a preference for a certain feature, causing the model to overfit and lead to recognition errors.

[0004] 2) Imbalanced attention allocation: When the object to be identified occupies a large proportion of the image or the key features are dispersed, the deep learning model based on the attention mechanism will often only focus on the most discriminative feature areas, and does not pay enough attention to other feature areas, thus affecting the overall recognition accuracy. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a deep learning model training method that takes into account multiple feature regions. This method aims to improve the generalization ability and robustness of the deep learning model based on the attention mechanism and avoid missed recognition.

[0006] The technical solution is as follows:

[0007] A deep learning model training method taking into account multiple feature regions includes the following steps:

[0008] S1, load the labeled image I into the deep learning model, perform initial training on the model and update the model parameters;

[0009] S2, stacking the multi-dimensional feature maps output by the feature extractor during model training to form a stacked heat map;

[0010] S3, generating a mask based on the stacked heat map, and using the mask to shield the key focus area in the image I;

[0011] The masked image I replaces the original image I and is loaded into the deep learning model with updated parameters, the model is trained again, and the model parameters are updated;

[0012] S4, after repeating steps S2 to S3 once or multiple times, proceed to step S5;

[0013] S5. Switch another labeled image from the training set and record it as image I, and use it to execute step S1. After all images in the training set image library are input, the model training is completed and the model parameters are stored.

[0014] Furthermore, in step S2, the multi-dimensional feature graphs output by the feature extractor during the model training process are stacked to form a stacked heat map in the following manner:

[0015]

[0016] A(i,j) represents the stacked heat map, F k (i, j) represents the k-th dimension feature map output by any convolutional layer of the feature extractor, k = 1, 2...n, n represents the dimension of the feature map output by the feature extractor, It means to accumulate the pixel values ​​at the same position (i, j) in the 1D to nD feature map.

[0017] Preferred, F k (i,j) represents the k-th dimension feature map output by the last convolutional layer of the feature extractor.

[0018] Further, in step S3, a mask is generated based on the stacked heat map, and the method of using the mask to shield the focus area in the image I is as follows:

[0019] ① Take the mean of the pixel values ​​in the stacked heat map and record it as the threshold

[0020] ② Stack the pixels in the heat map whose values ​​are greater than the threshold The pixel value of the pixel point is set to 0, and the pixel values ​​of other pixels are set to 1 to generate a mask;

[0021] ③ Upsample the mask to the size of image I, then multiply it by image I to mask the focus area in image I.

[0022] In order to constrain the model's focus area to a more precise area, it is more preferred that, in step S2, the dimensional feature maps output by different convolutional layers of the feature extractor are stacked respectively to form multiple stacked heat maps;

[0023] In step S3, masks are generated based on the multiple stacked heat maps to obtain multiple masks, the multiple masks are unified into the same size, and then their intersection is taken to generate a final mask, and the final mask is used to shield the key focus area in image I.

[0024] Preferably, steps S2 to S3 are performed 2 to 4 times.

[0025] This method has the following characteristics:

[0026] This method targets deep learning models based on attention mechanisms, such as E-Net, SE-ResNet, and SE-ResNeXt, or networks embedded with SE (Squeeze and Excitation) and CBAM (Convolutional Block Attention Module) modules.

[0027] By masking the focused area in the input image and inputting the masked image into the model again for training, the model is forced to pay attention to other areas that may be ignored, thus suppressing the model's excessive attention to the most distinguishable local features in the image.

[0028] The model parameters trained by this method can pay more attention to the global features of the image and take into account the characteristics of multiple regions, thereby improving the generalization ability and robustness of the model and avoiding missed recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a schematic diagram of the hand bone image training process in a specific implementation manner;

[0030] Figure 2 Schematic diagram of stacking multi-dimensional feature maps to form a stacked heat map. DETAILED DESCRIPTION

[0031] The technical solution of the present invention is described in detail below in conjunction with embodiments.

[0032] The attention mechanism in deep learning allows the neural network to focus on the relevant parts when processing input data. In traditional neural networks, the output of each neuron depends only on the output of all neurons in the previous layer, while in the attention mechanism, the output of each neuron not only depends on the output of all neurons in the previous layer, but also can be weighted according to different parts of the input data, that is, different weights are given to different parts. This allows the model to pay more attention to the key information in the input sequence, thereby improving the accuracy and efficiency of the model.

[0033] However, deep learning models based on the attention mechanism often focus too much on a certain area in the image, which easily ignores other feature areas and leads to missed recognition, especially in the following situations:

[0034] Scenario 1: The object to be identified in the image has obvious feature preferences. For example, the color features of the target to be identified are relatively obvious. In this case, the model may pay too much attention to the color feature information and ignore other feature information. When other targets with the same color features appear, misidentification will easily occur.

[0035] Scenario 2: The key features of the object to be identified are relatively scattered. For example, large-size images often contain more features, and the features are distributed in multiple areas of the image. Deep learning models based on the attention mechanism often only focus on the most discriminative areas, and do not pay enough attention to other important areas, which greatly reduces the recognition accuracy.

[0036] In view of the above phenomenon, the present invention proposes a deep learning model training method that takes into account multiple feature areas. The model trained by this method can pay more attention to the global features of the image and avoid missed recognition and misrecognition.

[0037] This method first collects images of the object to be tested at different times and angles (collecting sample images for the same type of object to be tested, such as collecting hand bone images of different people) and labels the features in the images to obtain a training set;

[0038] Next, the specific steps of model training are as follows:

[0039] S1, load the labeled image I into the deep learning model, perform initial training on the model and update the model parameters;

[0040] S2, stacking the multi-dimensional feature maps output by the feature extractor during model training to form a stacked heat map;

[0041] In specific implementation, the stacking method is as follows:

[0042]

[0043] A(i,j) represents the stacked heat map, F k (i, j) represents the k-th dimension feature map output by any convolutional layer of the feature extractor, k = 1, 2...n, n represents the dimension of the feature map output by the feature extractor, It means to accumulate the pixel values ​​at the same position (i, j) in the 1D to nD feature map.

[0044] S3, generating a mask based on the stacked heat map, and using the mask to shield the focus area in the image I; replacing the original image I with the shielded image I and loading it into the deep learning model that has completed parameter update, retraining the model, and updating the model parameters;

[0045] In specific implementation, the process of step S3 is:

[0046] ① Take the mean of the pixel values ​​in the stacked heat map and record it as the threshold

[0047] ② Stack the pixels in the heat map whose values ​​are greater than the threshold The pixel values ​​of the pixels are set to 0, and the pixel values ​​of other pixels are set to 1 to generate the mask M(i,j):

[0048]

[0049] ③ Upsample the mask M(i,j) to the size of image I, then multiply it by image I to mask the focus area in image I.

[0050] S4, after repeating steps S2 to S3 once or multiple times (preferably, steps S2 to S3 are repeated 1 to 4 times), proceed to step S5;

[0051] S5, switch another labeled image from the training set and record it as image I, and use it to execute step S1, until all images in the training set image library are input, the model training is completed, and the model parameters are stored. Wherein, when step S1 is executed again, the model parameters have been updated in the above training process and are no longer the initial model parameters.

[0052] In the feature extractor, the deeper the network layer, the higher the dimension of the feature map, the larger the receptive field, and the larger the input image area that each point can "see". In order to constrain the model's focus area to a more precise area, as a more preferred implementation, the mask is generated using the following method:

[0053] In step S2, the dimensional feature maps output by different convolutional layers of the feature extractor are stacked respectively to form multiple stacked heat maps; in step S3, masks are generated based on the multiple stacked heat maps to obtain multiple masks, the multiple masks are unified into the same size, and then their intersection is taken to generate a final mask, and the final mask is used to shield the key focus area in image I.

[0054] Let’s take hand bone feature recognition as an example. Figure 1 As shown, an exemplary explanation is given:

[0055] S1, loading the labeled hand bone image I into the deep learning model, and the model performs data augmentation, which includes: flipping, rotating, deforming and resizing the image, adding noise and / or affine transformation, and performing normalization and standardization processing, and finally serving as the input image of the model;

[0056] Perform initial training on the model and update model parameters;

[0057] S2, such as Figure 2 As shown in the figure, during the initial training process, the multi-dimensional feature map output by the last convolutional layer in the feature extractor is selected and stacked to form a stacked heat map; wherein the feature extractor is the convolutional layer before the pooling layer and the classification layer.

[0058] In this embodiment, in order to constrain the focus area of ​​the model to a more precise area, the mask is generated in the following manner:

[0059] The dimensional feature maps output by the penultimate layer and the last convolutional layer of the feature extractor are stacked to form two stacked heat maps A1(i,j) and A2(i,j);

[0060]

[0061] In specific implementation, the dimensional feature maps output by different convolutional layers can be selected and stacked respectively according to the actual situation. For example, the dimensional feature maps output by the 2nd, 3rd, and 6th convolutional layers can be selected and stacked respectively to form 3 stacked heat maps.

[0062] S3. Generate masks based on multiple stacked heat maps to obtain multiple masks:

[0063] like Figure 2 As shown, the specific steps are:

[0064] ① Take the mean of each pixel value in the stacked heat map A1(i,j) and A2(i,j) respectively, and record it as the threshold

[0065] ② Stack the pixels in the heat map A1(i,j) whose values ​​are greater than the threshold The pixel values ​​of the pixels are set to 0, and the pixel values ​​of other pixels are set to 1 to generate the mask M1(i,j):

[0066]

[0067] The pixel values ​​in the stacked heat map A2(i,j) that are greater than the threshold The pixel values ​​of the pixels are set to 0, and the pixel values ​​of other pixels are set to 1 to generate the mask M2(i,j):

[0068]

[0069] Unify the two masks M1(i,j) and M2(i,j) to the same size (downsampling / upsampling), and then take their intersection to generate the final mask M(i,j):

[0070] M(i,j)=M1(i,j)∩M2(i,j)

[0071] ③ Upsample the mask M(i,j) to the size of image I, then multiply it by image I to mask the focus area in image I;

[0072] The masked image I replaces the original image I and loads it into the deep learning model that has completed parameter update (initial training completed), trains the model for the second time, and updates the model parameters;

[0073] S4, steps S2 to S3 are performed again (in this embodiment, the execution is performed twice in total, and the number of executions can be set according to actual needs in actual application);

[0074] The specific implementation is as follows:

[0075] S2. During the second model training process, the multi-dimensional feature maps output by the feature extractor are stacked to form a stacked heat map;

[0076] S3, generating a mask based on the stacked heat map, and using the mask to shield the focus area in the new image I;

[0077] The masked image I replaces the original image I and is loaded into the deep learning model that has completed parameter update (the second model training is completed), the model is trained for the third time, and the model parameters are updated;

[0078] After the third training is completed, proceed to step S5;

[0079] S5. Switch another labeled image from the training set and record it as image I, and use it to execute step S1. After all images in the training set image library are input, the model training is completed and the model parameters are stored.

[0080] like Figure 1It can be seen that after the initial training, the model only focused on the bones of the palm and some fingers (local area features), and did not cover the entire hand bone area, resulting in inaccurate hand bone recognition. During the training process, this method stacks the feature maps output by the model feature extractor during the last training to form a stacked heat map, masks the key focus areas of the input image, and then conducts training again, that is, a hand bone image is trained three times, the input for the first training is the original image, the input for the second training is the masked image, and the input for the third training is the image after masking again on the basis of the previous mask; this training method forces the model to pay attention to other feature areas that may not necessarily be paid attention to, so that the model parameters obtained by the final training are more reasonable and can pay attention to the entire hand bone features.

[0081] The foregoing description of the specific exemplary embodiments of the present invention is presented for the purpose of illustration and description. The foregoing description is not intended to be exhaustive, nor is it intended to limit the present invention to the precise form disclosed, and it is apparent that many changes and variations are possible in light of the above teachings. The exemplary embodiments are selected and described to explain the specific principles of the present invention and its practical application, so that other technicians in the field can realize and utilize the various exemplary embodiments of the present invention and its different selected forms and modified forms. The scope of the present invention is intended to be limited by the appended claims and their equivalents.

Claims

1. A deep learning model training method taking into account multiple feature regions, characterized in that: The steps include: S1, load the labeled image I into the deep learning model, perform initial training on the model and update the model parameters; S2, stacking the multi-dimensional feature maps output by the feature extractor during model training to form a stacked heat map; S3, generating a mask based on the stacked heat map, and using the mask to shield the key focus area in the image I; The masked image I replaces the original image I and is loaded into the deep learning model with updated parameters, the model is trained again, and the model parameters are updated; S4, after repeating steps S2 to S3 once or multiple times, proceed to step S5; S5. Switch another labeled image from the training set and record it as image I, and use it to execute step S1. After all images in the training set image library are input, the model training is completed and the model parameters are stored.

2. The deep learning model training method taking into account multiple feature areas as claimed in claim 1, characterized in that: In step S2, the multi-dimensional feature graphs output by the feature extractor during the model training process are stacked to form a stacked heat map in the following manner: A(i,j) represents the stacked heat map, F k (i, j) represents the k-th dimension feature map output by any convolutional layer of the feature extractor, k = 1, 2...n, n represents the dimension of the feature map output by the feature extractor, It means to accumulate the pixel values ​​at the same position (i, j) in the 1D to nD feature map.

3. The deep learning model training method taking into account multiple feature areas as claimed in claim 2, characterized in that: F k (i,j) represents the k-th dimension feature map output by the last convolutional layer of the feature extractor.

4. The deep learning model training method taking into account multiple feature areas as claimed in claim 1, characterized in that: In step S3, a mask is generated based on the stacked heat map, and the method of using the mask to shield the focus area in the image I is as follows: ① Take the mean of the pixel values ​​in the stacked heat map and record it as the threshold ② Stack the pixels in the heat map whose values ​​are greater than the threshold The pixel value of the pixel point is set to 0, and the pixel values ​​of other pixels are set to 1 to generate a mask; ③ Upsample the mask to the size of image I, then multiply it by image I to mask the focus area in image I.

5. The deep learning model training method taking into account multiple feature areas as claimed in claim 1, characterized in that: In step S2, the dimensional feature maps output by different convolutional layers of the feature extractor are stacked respectively to form multiple stacked heat maps; In step S3, masks are generated based on the multiple stacked heat maps to obtain multiple masks, the multiple masks are unified into the same size, and then their intersection is taken to generate a final mask, and the final mask is used to shield the key focus area in image I.

6. The deep learning model training method taking into account multiple feature regions as claimed in claim 1, characterized in that: Steps S2 to S3 are performed 2 to 4 times.

7. The deep learning model training method taking into account multiple feature areas as claimed in claim 1, characterized in that: Before step S1, the images of the object to be tested collected at different times and angles are aggregated and the features in the images are labeled to obtain a training set.

Citation Information

Patent Citations

  • Earth observation image semantic segmentation method based on self-supervised learning

    CN112308860A

  • Semantic segmentation network training method and device, equipment and storage medium

    CN113780532A

  • Zero sample semantic segmentation method based on thermodynamic diagram and self-attention module

    CN116486078A

  • AU2020103905A4