Target augmentation and multi-region selection network method for alien invasive plant recognition
Patent Information
- Application Number
- CN202311158825.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-08
- Publication Date
- 2026-10-30
- Estimated Expiration
- 2043-09-08
AI Technical Summary
[0007]本发明的目的是为了解决现有的网络存在的在进行细粒度分类时，只关注区分性区域的定位，忽略了网络学习目标整体结构的特征对于结果的影响以及区分性区域定位的作用的技术问题；以及这些方法生成的区别性部位不一定是最有区分性的那些，从而造成分类的结果不佳的技术问题
[0049]1)本发明提出了一种目标增强和多区域选择网络构建方法，它具有三个分支，三个分支可以分别针对性的学习原始图像信息，目标整体结构信息和区分性区域的信息；
Smart Images
Figure CN117409235B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, and more specifically to a target enhancement and multi-region selection network method for identifying invasive alien plants. Background Technology
[0002] Invasive alien species are a global environmental problem. They not only cause severe ecological damage and biological pollution, leading to species extinction and biodiversity loss, but also easily trigger ecological disasters, endangering human and livestock health and causing significant economic losses. Currently, invasive alien plants and native plants may be so similar in appearance that they are difficult to distinguish. Moreover, the same species of plant can exhibit differences at different growth stages, while two different species of plant may appear similar at the same growth stage. Therefore, the classification of invasive plants mainly relies on manual judgment by domain experts, resulting in a large workload and low efficiency for image-based species identification tasks.
[0003] Since the rise of deep learning algorithms in the image processing field, automatic classification based on computer vision has seen significant development. Currently, there are two main types of fine-grained classification methods: one uses feature encoding, which enhances object classification by performing bilinear pooling on feature maps generated by two networks; the other uses localization, which has the advantage of capturing the distinguishing features between different targets, a crucial guarantee for differentiating them. Strongly supervised learning, which involves labeling additional distinguishing regions for classification, is not only computationally intensive but also subject to personal subjectivity, as different people may perceive different distinguishing features differently. To avoid these issues, current fine-grained classification primarily uses weakly supervised learning, employing only image-level labels to complete the localization of distinguishing features and the identification of target categories.
[0004] PMG uses a jigsaw puzzle approach, feeding in a shuffled image to disrupt its overall structure and force the network to learn local region information. HERBS learns different detailed feature maps by suppressing background regions and enhancing target features. Despite these significant advancements, these weakly supervised fine-grained classification networks still face some challenges:
[0005] 1) Much research simply inputs the original image directly into the network to learn the overall structural information of the target, focusing more on how to better capture the distinctive parts in local regions. This approach ignores the positive role of locating the overall structure of the target in the network's learning of the target's structural information.
[0006] 2) Existing studies often locate detailed regions directly through attention or other methods, indirectly judging the discriminativeness of the generated regions based on the quality of the results, rather than selecting the most important regions by measuring their relative importance. This results in low accuracy for classifying invasive species using existing algorithms, failing to meet practical needs. Summary of the Invention
[0007] The purpose of this invention is to address the technical problems of existing networks that, when performing fine-grained classification, only focus on the localization of discriminative regions, ignoring the influence of the overall structural features of the network's learning target on the results and the role of discriminative region localization; and the technical problem that the discriminative parts generated by these methods are not necessarily the most discriminative, resulting in poor classification results.
[0008] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:
[0009] A target enhancement and multi-region selection network method for identifying invasive alien plants includes the following steps:
[0010] Step 1: Design a target augmentation and multi-region selection network with three branches: the first branch - image branch, which learns the relationship information between the image and the background; the second branch - target branch, which learns the overall structural information of the target; and the third branch - region branch, which learns fine-grained information of multiple discriminative regions.
[0011] Step 2: Construct a target enhancement and localization module to locate the target based on the input image feature map, obtain the target's bounding box, and crop and enlarge it to obtain the target image;
[0012] Step 3: Construct a multi-region selection module to convert the feature map of the target image into a channel relationship matrix. Use an index to measure the importance of different channels and select the regions contained in the most important channels for learning.
[0013] Step 4: First, the original image is fed into the image branch to generate features of the original image and obtain the classification result of the original image; at the same time, the image feature map is input into the target enhancement and localization module to obtain the location information of the target, and the original image is cropped and enlarged to obtain the target image; then the target image is input into the target branch to extract features and obtain the classification result of the target image; in addition, the target feature map will pass through the multi-region selection module to select multiple distinctive regions to obtain region images, which are fed into the region branch to learn detailed information and obtain the classification result. The results output by the three branches are used to calculate the loss and constrain the training of the network.
[0014] The target augmentation and multi-region selection networks are constructed and trained using the above steps.
[0015] like Figure 2 As shown, in step 1, when establishing the target enhancement and multi-region selection network, the image branch input is the original image, the target branch input is the target image after the original image has been cropped by the target enhancement and localization module, and the target branch input is multiple partial region images located by the multi-region selection module. The feature extraction network uses ResNet50, and the three branches share the model parameters.
[0016] In step 1, the target enhancement and multi-region selection network is established as follows: Figure 2 As shown:
[0017] Original branch feature extractor → Original branch fully connected layer;
[0018] Original branch feature extractor → Target enhancement and localization module → Target branch feature extractor → Target branch fully connected layer;
[0019] Target branch feature extractor → Multi-region selection module → Region branch feature extractor → Region branch fully connected layer.
[0020] In step 2, the constructed target enhancement and localization module is as follows: Figure 3 As shown:
[0021] Original branch feature extractor → Calculate attention map → Generate original attention mask → Erosion and dilation module;
[0022] When using the constructed target enhancement and localization module, follow these steps:
[0023] 2-1: Given an image X, feed it into the image branch to extract features, and take the feature map F∈R from the last layer of the network. C×H×W C describes the channels of the feature map, and H×W are the height and width of the feature map;
[0024] 2-2: The feature maps F are summed along the channel dimension to obtain the activation map A∈R 1×H×W ,
[0025]
[0026] F i The feature represents the i-th channel of the feature map, and C is the number of channels;
[0027] 2-3: Identify the target regions using activation map A. Specifically, first, take the average value z of activation map A. Points in activation map A with values greater than z are considered part of the target and assigned a value of 1. Points with values less than z are assigned a value of 0, representing background regions unrelated to the target. This results in an attention map containing multiple activation regions.
[0028]
[0029] (x, y) represents the coordinates of a feature point on the feature map; then select The largest connected region in the bounding box is used as the region where the target exists, and the coordinates of the top-left and bottom-right corners of the target bounding box are returned for positioning; although The largest region in the map contains the most target structure information, but in many cases, even the largest region does not cover the entire target, but only a large part of it. Since the real target is a continuous and complete structure, the connected regions containing information about different parts of the target are also close together. We only need to connect the close regions containing target information, and finally select the largest connected region to form a bounding box, which can locate a more complete target structure. Specifically, we enhance the feature map by simulating erosion and dilation operations. Different regions containing information about each part of the target are connected to form a complete target region. Finally, the region with the largest connected region is selected to form the target bounding box M as the final localization result.
[0030] 2-3: The erosion operation simulated using Torch mainly consists of minimum pooling, while the dilation operation mainly consists of max pooling. During minimum pooling, single noise points within the window region are filtered out. During max pooling, discontinuous ROI regions within the window region are connected by their maximum values. Since the model processes a large amount of data in this part, compared to normal erosion and dilation algorithms, implementing this architecture using Torch and leveraging the GPU improves computational speed, mitigating the slow training speed caused by CPU limitations. The simulated dilation operation can... By connecting similar regions (multiple regions containing target information) in the image, the problem that a single connected region cannot contain complete target information can be effectively alleviated. At the same time, the erosion operation can effectively remove small background regions in the image. After performing this erosion and dilation operation, and then taking the minimum bounding box of the connected regions, we can obtain the attention map A containing the target information. o The target image is obtained by locating the position in the original image based on the coordinates of the attention map, cropping and enlarging it, and then feeding the target image into the target network to learn the target structural features and perform classification.
[0031] In step 3, the constructed multi-region selection module is as follows: Figure 4 As shown:
[0032] Target branch feature extractor → Inter-channel self-attention → Region selection module → Generate mask region;
[0033] The structure of the region selection module is as follows:
[0034] Inter-channel self-attention → Entropy calculation → Sorting;
[0035] In step 3, when the feature map of the target image is input into the multi-region selection module, the following sub-steps are used:
[0036] 3-1: In order for the multi-region selection module to select the most important regions from the target image, a value is defined for each channel using self-attention to reflect the importance of each channel;
[0037] 3-2: This module converts the target feature map F∈R extracted by the target network. C×H×W As input; first, the feature map Q∈
[0038] R C×HW and the transposed feature map K∈R HW×C Multiplication yields the relationship matrix M between channels. qk ∈R C×C Then, each row of the matrix is assigned a softmax function to obtain the global attention matrix M∈R on the channel. C×c The calculation formula used is as follows:
[0039] M(i,j)=softmax(QK)(i,j) (3)
[0040] The value of M(i,j) represents the similarity measure between the i-th channel and the j-th channel, and the value in the i-th row represents the similarity between the i-th channel and all channels globally.
[0041] 3-3: To calculate the importance of each channel, an entropy formula is used to evaluate the importance of each channel and to find channels that contain information about discriminative parts; the entropy formula used is as follows:
[0042]
[0043] Here, i and j represent the row and column of M, respectively, and H(i) represents the importance of the i-th channel. The rows of M are sorted in ascending order based on their entropy values, and the first N rows are selected as the final component attention A. n ∈R N×C , put A n Multiplying by V yields the region feature map F. n ∈R N×HW N is the number of channels in the region feature map, which contains the N most important regions; then it is fed into the region branch to learn the information of the discriminative parts.
[0044] In step 4:
[0045] Use cross-entropy loss as the classification loss for the three branches;
[0046]
[0047] P image (Y) is the classification result of the image branch for the image, P object (Y) is the classification result of the target branch for the captured target image, P part (r) represents the classification results of the component branch for all discriminative regions. The total loss is the sum of the losses of the three branches, which are used together to optimize the network.
[0048] Compared with the prior art, the present invention has the following technical effects:
[0049] 1) This invention proposes a target enhancement and multi-region selection network construction method, which has three branches. The three branches can respectively learn the original image information, the overall structure information of the target, and the information of the discriminative regions;
[0050] 2) This invention proposes a target enhancement and localization module, which can accurately locate the required target position, enabling the network to better learn the global structural features of the target, while eliminating background interference, which is beneficial for the next step of locating important areas;
[0051] 3) This invention proposes a multi-region selection attention module, which uses the global relationship of self-attention to select multiple important regions in the global structure and locate these regions so that the network can effectively learn fine-grained features. Attached Figure Description
[0052] The present invention will be further described below with reference to the accompanying drawings and embodiments:
[0053] Figure 1 This is a flowchart of the training strategy of the present invention;
[0054] Figure 2 This is a diagram of the target enhancement and multi-region selection network structure of the present invention;
[0055] Figure 3 This is a structural diagram of the target enhancement and positioning module of the present invention;
[0056] Figure 4 This is a structural diagram of the multi-region selection module of the present invention. Detailed Implementation
[0057] A target enhancement and multi-region selection network method for identifying invasive alien plants includes the following steps:
[0058] Step 1: First, design a target enhancement and multi-region selection network with three branches: the first branch - image branch, which learns the relationship information between the image and the background; the second branch - target branch, which learns the overall structural information of the target; and the third branch - region branch, which learns fine-grained information of multiple discriminative regions.
[0059] Step 2: Construct a target enhancement and localization module, which locates the target based on the input image feature map, obtains the target's bounding box, and crops and enlarges it to obtain the target image;
[0060] Step 3: Construct a multi-region selection module, which converts the feature map of the target image into a channel relationship matrix, uses an index to measure the importance of different channels, and selects the regions contained in the most important channels for learning;
[0061] Step 4: First, the original image is fed into the image branch to generate features for the original image, resulting in a classification result. Simultaneously, the image feature map is input into the target enhancement and localization module to obtain the target's location information. The original image is then cropped and enlarged to obtain the target image. Next, the target image is input into the target branch to extract features and obtain a classification result. Furthermore, the target feature map passes through a multi-region selection module, selecting multiple distinctive regions to obtain region images. These region images are then fed into the region branch to learn detailed information and obtain a classification result. The results from these three branches are used to calculate the loss and constrain the network's training.
[0062] The target augmentation and multi-region selection networks are constructed and trained using the above steps.
[0063] like Figure 2 As shown, in step 1, when establishing the target enhancement and multi-region selection network, the image branch input is the original image, the target branch input is the target image after the original image has been cropped by the target enhancement and localization module, and the target branch input is multiple partial region images located by the multi-region selection module. The feature extraction network uses ResNet50, and the three branches share the model parameters.
[0064] In step 1, the target enhancement and multi-region selection network is established as follows:
[0065] Original branch feature extractor 1 → Original branch fully connected layer 2;
[0066] Original branch feature extractor 1 → Target enhancement and localization module 3 → Target branch feature extractor 4 → Target branch fully connected layer 5;
[0067] Target branch feature extractor 4 → Multi-region selection module 6 → Region branch feature extractor 7 → Region branch fully connected layer 8.
[0068] like Figure 3 As shown, in step 2, the target enhancement and localization module constructed is as follows:
[0069] Original branch feature extractor 1 → Calculate attention map 9 → Generate original attention mask 10 → Erosion and dilation module 11;
[0070] In step 2, when inputting the image feature map into the target enhancement and localization module, the following sub-steps are used:
[0071] 2-1: Given an image X, feed it into the image branch to extract features, and take the feature map F∈R from the last layer of the network. C×H×W C describes the channels of the feature map, and H×W are the height and width of the feature map;
[0072] 2-2: The feature maps F are summed along the channel dimension to obtain the activation map A∈R 1×H×W ,
[0073]
[0074] F i The feature represents the i-th channel of the feature map, and C is the number of channels;
[0075] 2-3: Identifying the target regions using activation map A. Specifically, the mean value z of activation map A is calculated. Points in activation map A with a value greater than z are considered part of the target region and assigned a value of 1. Points with a value less than z are assigned a value of 0, representing background regions unrelated to the target. This results in an attention map containing multiple activation regions.
[0076]
[0077] (x, y) represents the coordinates of a feature point on the feature map. Then select... The largest connected region in the bounding box is used as the region where the target exists, and the coordinates of the top-left and bottom-right corners of the target bounding box are returned for positioning. Although The largest region in the map contains the most target structural information, but in many cases, even the largest region does not cover the entire target, but only a large portion of it. Since a real target is a continuous and complete structure, the connected regions containing information about different parts of the target are also close together. We only need to connect the close regions containing target information, and finally select the area with the largest connected region to form a bounding box, thus locating a more complete target structure. Specifically, we enhance the feature map by simulating erosion and dilation operations. Different regions containing information about each part of the target are connected to form a complete target region. Finally, the region with the largest connected region is selected to form the target bounding box M as the final localization result.
[0078] 2-3: The erosion operation simulated using Torch mainly consists of minimum pooling, while the dilation operation mainly consists of max pooling. During minimum pooling, single noise points within the window region are filtered out. During max pooling, discontinuous ROI regions within the window region are connected by their maximum values. Since the model processes a large amount of data in this part, compared to normal erosion and dilation algorithms, implementing this architecture using Torch and leveraging the GPU improves computational speed, mitigating the slow training speed caused by CPU limitations. Through the simulated dilation operation, [the model can...] By connecting different regions that are close to each other (multiple regions containing target information), the problem that a single connected region cannot contain complete target information can be effectively alleviated. At the same time, the erosion operation can effectively remove small background regions in the image. After performing this erosion and dilation operation, and then taking the minimum bounding box of the connected regions, we can obtain the attention map A containing the target information. o The target image is obtained by locating the position in the original image based on the coordinates of the attention map, cropping and enlarging it, and then feeding the target image into the target network to learn the target structural features and perform classification.
[0079] like Figure 4 As shown,
[0080] In step 3, the multi-region selection module constructed is as follows:
[0081] Target branch feature extractor 4 → Inter-channel self-attention 12 → Region selection module 13 → Generate mask region 14;
[0082] The structure of the region selection module is as follows:
[0083] Channel self-attention 12 → Entropy calculation 15 → Sorting 16;
[0084] In step 3, when the feature map of the target image is input into the multi-region selection module, the following sub-steps are used:
[0085] 3-1: To enable the multi-region selection module to select the most important regions from the target image, a value is defined for each channel to reflect the importance of each channel. A self-attention approach is chosen, which allows us to obtain a similarity metric for each channel with all other channels.
[0086] 3-2: This module converts the target feature map F∈R extracted by the target network. C×H×W As input, C, H, and W represent the channel, height, and width, respectively. First, the feature map F is reshaped into Q∈R. C×HW , K∈R HW×CV∈R C×HW , let Q∈R C×HW K∈R obtained by transposing the feature map HW×C Multiplication yields the relationship matrix M between channels. qk ∈R C×C Then, each row of the matrix is assigned a softmax function to obtain the global attention matrix M∈R on the channel. C×C The calculation formula is as follows:
[0087] M(i,j)=softmax(QK)(i,j) (3)
[0088] The value of M(i,j) represents the similarity measure between the i-th channel and the j-th channel, and the value in the i-th row represents the similarity between the i-th channel and all channels globally.
[0089] 3-3: To calculate the importance of each channel, we introduce an entropy formula. The importance of each channel is evaluated based on the entropy value of each row in M, selecting the most discriminative regions. The entropy formula is as follows:
[0090]
[0091] Here, i and j represent the row and column of M, respectively, and H(i) represents the importance of the i-th channel. When it is close to 0, it means that there are few other channels in the same region as the channel, which are generally background or unimportant regions. To select all important regions, the rows of M are sorted in ascending order according to their entropy values, and the first N rows are selected as the final component attention A. n ∈R N×C , put A n Multiplying by V yields the region feature map F. n ∈R N×HW N represents the number of channels in the region feature map, which contains the N most important regions. The information contained in each of the N channels is mapped to the original image, and the images are cropped and enlarged to obtain N detailed regions, which are then fed into the component network to learn fine-grained information.
[0092] In step 4, the network uses cross-entropy loss as the classification loss for the three branches during training.
[0093]
[0094] P image (Y) is the classification result of the image branch for the image, P object (Y) is the classification result of the target branch for the captured target image, P part (Y) represents the classification results of the component branch for all discriminative regions. The total loss is the sum of the losses of the three branches, which are used together to optimize the network.
[0095] Example:
[0096] 1. Dataset Information
[0097] To investigate the performance of the network in this invention for identifying invasive alien plants, experiments and evaluations were conducted on three fine-grained classification datasets: 21-Plant, 18-Plant, and FGVC-Aircraft, as shown in Table 1. The 21-plant and 18-plant datasets were taken from the plant categories of iNaturalist-2021Mini and iNaturalist-2018 to facilitate evaluation of the network's performance in plant classification. Simultaneously, the robustness of the proposed model was verified on FGVC-Aircraft, a commonly used public dataset for fine-grained classification.
[0098] Table 1 shows information about the three datasets used in this invention.
[0099]
[0100] 2. Parameter Settings
[0101] The equipment used in the experiments of this invention consisted entirely of NVIDIA GeForce RTX 3090 graphics cards with 24GB of dedicated video memory. This invention uses a ResNet-50 network as the backbone network and employs an ImageNet pre-trained model. The input images for the image network and target network are 448×448, while the input image for the component network is 224×224. The batch size was set to 12 for plants in iNaturalist-21 and iNaturalist-18, and to 6 for FGVC-Aircraft. This invention uses image-level labels, employs SGD as the optimizer, and the model is built using PyTorch.
[0102] 3. Comparative Experiment
[0103] This invention was compared with state-of-the-art methods on three datasets. The invention was first compared on 21-mini-plant and 18-plant plant datasets, and the invention achieved the best classification performance among all the networks compared.
[0104] Table 2 shows the comparative experiments of this invention on plant datasets from 2018 and 2021.
[0105]
[0106] Furthermore, this invention was compared with some state-of-the-art networks on the public dataset FGVC-Aircraft, and this invention also achieved the best results.
[0107] Table 3 shows the comparative experiments of this invention on FGVC-Aircraft.
[0108]
[0109]
[0110] 4. Ablation test
[0111] To verify the effectiveness of the modules designed in this invention, ablation experiments were conducted on the three datasets used in this invention. The backbone of this invention uses ResNet50. The experimental results demonstrate that the OEAL and MRSA modules of this invention can effectively improve the performance of the model on different datasets.
[0112] Table 4 shows the ablation experiments conducted on the three datasets used in this invention.
[0113]
[0114] 5. Comparison with other location-classification based models:
[0115] Unlike RA-CNN and NTS-Net, which capture different regions of an image by designing an additional network, this invention does not introduce additional parameters. The OELA and MRSA of this invention directly generate accurate target regions and partial regions based on feature maps. For target localization, MMAL directly uses spatial attention AOLM to locate the target, which often locates only a part of the target, not the whole target. The OEAL of this invention introduces erosion and dilation operations to connect different parts belonging to the target, thereby giving the target more complete structural information. For multi-region localization, TASN obtains attention maps through a trilinear sampling module, randomly selecting only one channel's attention map as a partial region each time, iterating multiple times to learn different regions. Learning through random selection is inefficient; the MRSA module of this invention can filter and locate the most important regions from all attention maps for learning. Experiments on 18-plant demonstrate that the method of this invention performs best among these networks, as shown in the table.
[0116] Table 5 compares the models with other location-classification network models.
[0117]
[0118] This invention provides a target enhancement and multi-region selection network based on invasive alien plant identification. Unlike existing localization-based methods, the network of this invention can accurately locate the complete target structure and distinguishable parts. It first learns the overall structural information of the target, and then learns the information of the distinguishable regions, progressively learning target features from coarse to fine. This effectively improves the network's ability to identify invasive alien plants.
Claims
1. A target enhancement and multi-region selection network method for identifying invasive alien plants, characterized in that, It includes the following steps: Step 1: Design a target augmentation and multi-region selection network with three branches; the first branch - image branch, learns the relationship information between the image and the background; The second branch – the target branch – learns the overall structural information of the target; the third branch – the region branch – learns fine-grained information of multiple discriminative regions. Step 2: Construct a target enhancement and localization module to locate the target based on the input image feature map, obtain the target's bounding box, and crop and enlarge it to obtain the target image; Step 3: Construct a multi-region selection module to convert the feature map of the target image into a channel relationship matrix. Use an index to measure the importance of different channels and select the regions contained in the most important channels for learning. Step 4: First, the original image is fed into the image branch to generate the features of the original image and obtain the classification result of the original image; at the same time, the image feature map is input into the target enhancement and localization module to obtain the location information of the target, and the original image is cropped and enlarged to obtain the target image; then the target image is input into the target branch to extract features and obtain the classification result of the target image. In addition, the target feature map will go through a multi-region selection module to select multiple distinctive regions to obtain region images, which are then fed into the region branch to learn detailed information and obtain classification results. The results output by the three branches are used to calculate the loss and constrain the training of the network. The target augmentation and multi-region selection network is constructed and trained through the above steps; In step 2, the target enhancement and localization module is constructed as follows: The original branch feature extractor (1), the attention map calculation unit (9), the original attention mask generation unit (10), and the erosion and dilation module (11) are connected in sequence; When using the constructed target enhancement and localization module, follow these steps: 2-1: Given an image X, feed it into the image branch to extract features, and take the feature map from the last layer of the network. We use C to describe the channels of the feature map. It is the height and width of the feature map; 2-2: The activation map is obtained by summing the feature maps F along the channel dimension. : Representative feature map Characteristics of each channel It is the number of channels; 2-3: Identify the regions containing the target using region A; specifically, first obtain the average value of the activation map A. It is believed that the activation graph A is greater than The point is part of the target, and its corresponding position is assigned a value of 1, which is less than 1. Points with activation values of 0 are considered background regions unrelated to the target; this results in an attention map containing multiple activation regions. : Represents the coordinates of feature points on the feature map; then select The largest connected region in the map is used as the region where the target exists, and the coordinates of the top-left and bottom-right corners of the target bounding box are returned for localization; the feature map is enhanced by simulating erosion and dilation operations. Different regions containing information about each part of the target are connected to form a complete target region. Finally, the region with the largest connected region is selected to form the target bounding box M as the final localization result. 2-3: The erosion operation simulated using Torch consists of a minimum pooling operator, and the dilation operation consists of a maximum pooling operator. During minimum pooling, single noise points within the window region are filtered out. During maximum pooling, discontinuous ROI regions within the window region are connected by their maximum values. Since the model processes a large amount of data in this part, compared to normal erosion and dilation algorithms, Torch is used to implement this architecture, and the GPU is used to improve computational speed. The simulated dilation operation... Connecting different regions that are close to each other; After performing this erosion and dilation operation, and then taking the minimum bounding box of the connected regions, an attention map containing target information can be obtained. The target image is obtained by locating the position in the original image based on the coordinates of the attention map, cropping and enlarging it, and then feeding the target image into the target network to learn the target structure features and classify it. In step 3, the multi-region selection module constructed is as follows: The target branch feature extractor (4), the inter-channel self-attention unit (12), the region selection module (13), and the mask region generation unit (14) are connected in sequence; In the region selection module, the processing flow is as follows: The process involves sequentially performing inter-channel self-attention, entropy calculation, and sorting. In step 3, when the feature map of the target image is input into the multi-region selection module, the following sub-steps are used: 3-1: In order for the multi-region selection module to select the most important regions from the target image, a value is defined for each channel using self-attention to reflect the importance of each channel; 3-2: This module extracts the target feature map from the target network. As input; first, the feature map and the transposed feature map Multiplication yields the relationship matrix between channels. Then, each row of the matrix is assigned a softmax function to obtain the global attention matrix for each channel. The calculation formula used is as follows: The value represents the first The first channel and the first The similarity metric for the nth channel, the th The value of the row represents the first row. The similarity between each channel and all channels globally; 3-3: Use the entropy formula to evaluate the importance of each channel and find channels that contain information about discriminative parts; the entropy formula used is as follows: here and They represent Rows and columns in Represents the first The importance of each channel, based on entropy value. The rows are sorted in ascending order; select the first row. Line as the final component attention ,Bundle Multiply Obtain the region feature map , The number of channels in the region feature map contains the most important... Each region is then fed into a regional branch to learn information about the distinguishing parts.
2. The method according to claim 1, characterized in that, In step 1, when establishing the target enhancement and multi-region selection network, the image branch input is the original image, the target branch input is the target image after cropping by the target enhancement and localization module, and the target branch input is multiple partial region images localized by the multi-region selection module. The feature extraction network uses ResNet50, and the three branches share the model parameters.
3. The method according to claim 2, characterized in that, In step 1, the target enhancement and multi-region selection network is established as follows: The output of the original branch feature extractor (1) is connected to the input of the original branch fully connected layer (2) to form the first branch; The original branch feature extractor (1), the target enhancement and localization module (3), the target branch feature extractor (4), and the target branch fully connected layer (5) are connected in sequence to form the second branch; The target branch feature extractor (4), the multi-region selection module (6), the region branch feature extractor (7), and the region branch fully connected layer (8) are connected in sequence to form the third branch.
4. The method according to any one of claims 1 to 3, characterized in that, In step 4, cross-entropy loss is used as the classification loss for the three branches: It represents the classification result of the image branch for the image. This is the classification result of the target branch for the captured target image. The component branch represents the classification results for all discriminative regions. The total loss is the sum of the losses of the three branches, which are used together to optimize the network.