Multi-branch image semantic segmentation method and system based on AM and feature fusion

Abstract

The application discloses a multi-branch image semantic segmentation method and system based on AM and feature fusion, and the method comprises the following steps: step 1, inputting an input image to be processed into a learning down-sampling network for three times of feature extraction to obtain a low-resolution feature map; step 2, inputting the low-resolution feature map into a deep semantic feature extraction branch, a spatial detail feature enhancement branch and a spatial detail branch respectively, performing semantic feature extraction, performing feature enhancement on spatial feature information, and performing feature fusion to obtain fused features; step 3, performing feature enhancement on the fused features based on an improved adaptive attention module; and step 4, up-sampling the low-resolution feature map to a high-resolution image, restoring the image size, and realizing semantic segmentation.

Description

Technical Field

[0001] This invention belongs to the field of semantic segmentation in computer vision, specifically involving a multi-branch image semantic segmentation method and system based on AM and feature fusion. Background Technology

[0002] Semantic segmentation plays a crucial role in computer vision and has facilitated several real-world applications, such as satellite remote sensing, intelligent robotics, and autonomous driving. The significance of semantic segmentation lies in processing an input image using algorithms to obtain a series of related image regions of different categories, each with a corresponding label, such as sky, road, power grid equipment, and pedestrians.

[0003] In recent years, semantic segmentation algorithms have developed rapidly, achieving significant research results both domestically and internationally. Current mainstream segmentation models primarily utilize Convolutional Neural Networks (CNNs) and encoder-decoder structures. For example, Fully Convolutional Networks (FCNs) primarily replace the fully connected layers of CNNs with convolutional layers and use deconvolution for upsampling. Based on FCNs, Ronneberger proposed the U-Net network structure, characterized by a novel U-shaped symmetric structure and the fusion of feature maps extracted by the encoder and decoder. While the U-Net model improves accuracy, its computational efficiency is very low, and it doesn't fully capture contextual information. To further improve segmentation accuracy, Google proposed DeepLabV1, DeepLabV2, DeepLabV3, and DeepLabV3+ networks. These networks improve segmentation accuracy with increased network depth. Although they achieve significant improvements in segmentation accuracy, their excessive depth typically requires substantial computational resources. However, mobile and embedded devices, such as intelligent robots and autonomous vehicles, have limited computational resources, making them difficult to embed. In the field of autonomous driving, not only are computing resources limited, but high demands are also placed on the real-time performance of algorithms. To improve the real-time performance of algorithms, one approach is to compress the network model, such as by using pruning techniques to remove redundancy in the network. However, when the model parameters are severely reduced, the algorithm performance will drop sharply. Another approach is to start from the network structure itself and design a lightweight network model with low computational cost.

[0004] BiSeNet, a bilateral segmentation network, consists of three main parts: a spatial path component, a feature extraction component, and a feature fusion component. The spatial path component preserves high-resolution spatial information, the feature extraction component has a large receptive field and good semantic segmentation information, and the feature fusion component aggregates spatial and contextual information to generate features rich in both semantic and spatial information, thus improving accuracy. However, the spatial path component and the feature extraction component are independent, limiting the model's learning ability. ICNet's three-branch structure meets real-time requirements, but its network model is not lightweight enough. Fast-SCNN, a fast segmentation convolutional neural network designed for high-resolution images, does not achieve ideal segmentation accuracy, failing to achieve an effective balance between real-time efficiency and accuracy. In summary, current real-time semantic segmentation has achieved substantial results, but some shortcomings still exist, especially in power systems. Faced with a large number of grid devices of various models and complex obstruction relationships, the extreme reduction of network parameters in pursuit of lightweight networks leads to insufficient semantic information extraction. Excessive reduction of feature extraction parameters for spatial information branches results in blurred segmentation boundaries and partial loss of global information. When using a multi-branch structure, the aggregation of spatial and semantic information acquired from each branch fails to achieve optimal results. Existing technologies often fuse feature branches of different resolutions using an additive approach each time, failing to maximize the utilization of each branch's features. This results in slow response times and unbalanced traffic loads for various needs, such as automatic feature data extraction, intelligent analysis, and decision support from IoT sensing terminals of power equipment. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a multi-branch image semantic segmentation method and system based on AM and feature fusion (AM-MulFSNet), which is used to solve the problem of embedding in low-end embedded devices and mobile hardware devices and achieving an effective balance between real-time performance and accuracy. The network model mainly consists of four parts: a semantic feature extraction branch, a spatial detail branch, a spatial detail feature enhancement branch, and an enhanced fusion feature attention module, achieving an effective balance between real-time performance and accuracy.

[0006] The present invention adopts the following technical solution.

[0007] A multi-branch image semantic segmentation method based on AM and feature fusion, characterized in that the method includes: Step 1: Input the image to be processed into the learning downsampling network to perform three feature extractions to obtain a low-resolution feature map; Step 2: Input the low-resolution feature map into the deep semantic feature extraction branch, the spatial detail feature enhancement branch, and the spatial detail branch respectively to extract semantic features and enhance spatial feature information, and then perform feature fusion to obtain fused features; Step 3: Enhance the fused features based on the improved adaptive attention module; Step 4: Upsample the low-resolution feature map to the high-resolution image, restore the image size, and perform semantic segmentation.

[0008] Preferably, step 1 includes: the input image is sequentially passed through a standard convolutional network, a first depthwise separable convolutional network, and a second depthwise separable convolutional network to extract features, resulting in a low-resolution feature map. Among them, low-resolution feature maps The resolution is 1 / 8 of the input image.

[0009] Preferably, step 2 includes inputting the processed low-resolution feature map into the deep semantic feature extraction branch to obtain deep semantic features; upsampling the deep semantic features to the same resolution as the spatial detail feature enhancement branch, and then fusing them with the spatial detail feature enhancement branch using the concat concatenation method to obtain features with the same resolution and number of channels as the spatial detail branch; finally, fusing them with the features of the spatial detail branch using the feature addition method; wherein a convolutional feature extraction is performed after each fusion.

[0010] Preferably, step 2 specifically includes: Step 201: Input the low-resolution feature map into the deep semantic feature extraction branch to obtain the first output feature. : Step 202: Input the low-resolution feature map into the spatial detail feature enhancement branch to obtain the second output feature. : Step 203: Use the concat concatenation method to extract the first output feature of the semantic feature extraction branch. The second output feature of the spatial detail feature enhancement branch The first fusion feature is obtained through fusion. and the first fusion feature Feature extraction is performed to obtain the second fusion feature. ; Step 204: Input the low-resolution feature map into the spatial detail branch, and combine the resulting third branch feature map with the second fused feature map. The final third fusion feature is obtained by fusion. .

[0011] Preferably, step 201 specifically includes: processing the low-resolution feature map Three sets of linear bottleneck reverse residual blocks are input sequentially for deep feature extraction to obtain feature maps. ; feature map Input to the pyramid pooling module, fuse multi-scale information, and retain global information at different scales to obtain feature maps. ; for feature maps Perform 4x upsampling and depthwise convolution to obtain the feature map. ; for feature maps Perform standard convolution to obtain the first output feature. .

[0012] Preferably, in step 202, the spatial detail feature enhancement branch is used to collect the pixel relationships between spatial detail features, and inputs the low-resolution feature map into the convolutional attention module to obtain the second output feature. The second output feature has the same size as the first output feature.

[0013] Preferably, the convolutional attention module includes a channel attention submodule and a spatial attention submodule. The channel attention submodule performs feature compression on the low-resolution feature map in the spatial dimension through max pooling and average pooling, extracts weights, and obtains two different features. The two features are then input into the multilayer perceptron and added in the spatial dimension to obtain attention weights. The channels of the initial input image are weighted to obtain a feature map with enhanced features. The spatial attention module takes the output of the channel attention submodule as input, performs average pooling and max pooling in the channel dimension, compresses the data, extracts spatial weights, and obtains two types of feature maps. The two feature maps with different representations are concatenated in the channel dimension. Convolutional layers are used to fuse and extract the concatenated features to obtain weighted spatial information. After activation by an activation function, a weighted feature map is obtained. The obtained weighted feature map is multiplied with the initial input image to obtain an attention-guided feature map.

[0014] Preferably, step 3 includes: inputting the fused features into the query branch, key branch, and value branch of the adaptive attention module respectively for 1×1 convolution mapping, wherein after the 1×1 convolution of the key branch and the value branch, adaptive pooling is added to obtain the key branch pooling feature map and the value branch pooling feature map respectively; The feature map obtained after convolving the key branch pooling feature map with the query branch feature map is multiplied and then input. Softmax The function is used to obtain the first attention weights; The output attention weight is obtained by multiplying the first attention weight with the value branch pooling feature map. The obtained output attention weights are added to the fused features to obtain the weighted output features.

[0015] Preferably, the query branch is converted into a feature of size N×C after a 1×1 convolution, where N=H×W; the key branch is downsampled to size C×M after a 1×1 convolution; the value branch is downsampled to size M×C after a 1×1 convolution, where C is the number of channels, H is the height of the fused feature, and W is the width of the fused feature, where M≤N / 512.

[0016] Preferably, step 4 includes: enhancing the feature map. After two depthwise separable convolutions by the decoder, classification is performed using convolutional layers, followed by an 8x upsampling to obtain an output of the same size as the original input image, thus achieving semantic segmentation.

[0017] A second aspect of the present invention provides a multi-branch image semantic segmentation system based on AM and feature fusion. The system includes a learning downsampling module, a feature fusion module, an adaptive global feature attention module (AGFAM), and an upsampling classifier; among which... The learning downsampling module is used to perform three feature extractions on the input image to be processed in order to obtain a low-resolution feature map; The feature fusion module includes a deep semantic feature extraction branch, a spatial detail feature enhancement branch, and a spatial detail branch, which are used to extract semantic features from low-resolution feature maps, enhance spatial feature information, and fuse features to obtain fused features. The Adaptive Global Feature Attention (AGFAM) module is used to enhance the fused features, resulting in an enhanced feature map. An upsampling classifier is used to classify and upsample feature maps that have been enhanced with features, restore the image size, and achieve semantic segmentation. The beneficial effects of this invention are that, compared with existing technologies, it solves the embedding problem of model algorithms in embedded devices, especially low-memory embedded devices and mobile hardware devices. While ensuring segmentation accuracy, it also guarantees computational efficiency and real-time performance, achieving low computational cost and lightweight model. This invention introduces a spatial feature detail enhancement branch and a convolutional attention module to enhance the network's global feature representation and better utilize shallow features. Simultaneously, to compensate for the loss of detailed information in its deep semantic features, the shallow features of the spatial feature detail enhancement branch are aggregated with the deep semantic features, and the shallow features are weighted and fused together. Furthermore, to better capture the interdependencies between long-distance pixels, an adaptive attention module is introduced after feature fusion, thereby improving the overall expressive power of the network. This invention achieves high accuracy with a relatively small number of parameters while maintaining a fast segmentation speed, achieving a reasonable balance between real-time efficiency and accuracy.

[0018] The beneficial effects of the present invention also include: 1. This invention aims to balance the real-time performance and accuracy of network models. It introduces a spatial feature detail enhancement module into the Fast_SCNN network. The spatial feature detail enhancement module includes an attention module, which captures the interrelationships between spatial features and further enhances the feature extraction capability of the spatial detail branch.

[0019] 2. Through reasonable design, this invention creates an aggregation module with three branches, maximizing the use of the feature information of each branch to achieve better segmentation accuracy.

[0020] 3. After feature aggregation, this invention introduces an improved adaptive attention module to add weighted attention to the original fused feature map, obtaining the dependency relationship between any two pixels in the feature map, further enhancing the features. The segmentation method provided by this invention is suitable for situations with a large number of power grid devices, especially in power systems, where there are many types of power grid devices and complex occlusion relationships. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating a multi-branch image semantic segmentation method based on AM and feature fusion according to the present invention. Figure 2 This is a model architecture diagram of a multi-branch image semantic segmentation method based on AM and feature fusion in this invention; Figure 3 This invention relates to a CBAM network structure in a multi-branch image semantic segmentation method based on AM and feature fusion. Figure 4 This invention relates to a channel attention submodule network structure in a multi-branch image semantic segmentation method based on AM and feature fusion. Figure 5 This invention relates to a spatial attention submodule network structure in a multi-branch image semantic segmentation method based on AM and feature fusion. Figure 6 This is a diagram illustrating the feature fusion process in a multi-branch image semantic segmentation method based on AM and feature fusion in this invention. Figure 7 This is a structural diagram of the AGFAM principle in a multi-branch image semantic segmentation method based on AM and feature fusion in this invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of this invention. The embodiments described in this application are merely some embodiments of this invention, and not all embodiments. Based on the spirit of this invention, other embodiments obtained by those skilled in the art without creative effort are all within the protection scope of this invention.

[0023] Figure 1 This is a flowchart illustrating a multi-branch image semantic segmentation method based on AM and feature fusion according to the present invention, where AM stands for attention mechanism. Figure 1 As shown, the first aspect of this invention relates to a multi-branch image semantic segmentation method based on AM and feature fusion. A multi-branch image semantic segmentation method based on AM and feature fusion, characterized in that the method includes: Step 1 involves feeding the input image to the learning downsampling network for three feature extraction steps to obtain a low-resolution feature map. Specifically, Step 1 includes: the input image sequentially passing through a standard convolutional network, a first depthwise separable convolutional network, and a second depthwise separable convolutional network in the learning downsampling module to extract features, resulting in a low-resolution feature map with a resolution of 1 / 8 of the input image. .

[0024] Step 2: Input the low-resolution feature map into the deep semantic feature extraction branch, the spatial detail feature enhancement branch, and the spatial detail branch respectively to extract semantic features and enhance spatial feature information, and then perform feature fusion to obtain fused features; Step 2 includes inputting the processed low-resolution feature map into the deep semantic feature extraction branch to obtain deep semantic features; upsampling the deep semantic features to the same resolution as the spatial detail feature enhancement branch, and then fusing them with the spatial detail feature enhancement branch using a concat concatenation method to obtain features with the same resolution and number of channels as the spatial detail branch; finally, fusing them with the spatial detail branch features using feature addition; wherein a convolution is performed after each fusion to extract features again. Step 2 specifically includes: Step 201: Input the low-resolution feature map into the deep semantic feature extraction branch to obtain the first output feature. : Specifically, step 201 includes: transferring the low-resolution feature map... Three sets of linear bottleneck reverse residual blocks are input sequentially for deep feature extraction to obtain feature maps. ; feature map Input to the pyramid pooling module, fuse multi-scale information, and retain global information at different scales to obtain feature maps. ; for feature maps Perform 4x upsampling and depthwise convolution to obtain the feature map. ; for feature maps Perform standard convolution to obtain the first output feature. .

[0025] Step 202, convert the low-resolution feature map The input space detail feature enhancement branch yields the second output feature. The spatial detail feature enhancement branch is used to collect pixel relationships between spatial detail features. It inputs the low-resolution feature map into the convolutional attention module to obtain the second output feature. The second output feature has the same size as the first output feature.

[0026] The convolutional attention module includes a channel attention submodule and a spatial attention submodule. The channel attention submodule performs feature compression on the low-resolution feature map in the spatial dimension through max pooling and average pooling, extracts weights, and obtains two different features. The two features are then input into the multilayer perceptron and added in the spatial dimension to obtain attention weights. The channels of the initial input image are weighted to obtain a feature map with enhanced features. The spatial attention module takes the output of the channel attention submodule as input, performs average pooling and max pooling in the channel dimension, compresses the data, extracts spatial weights, and obtains two types of feature maps. The two feature maps with different representations are concatenated in the channel dimension. Convolutional layers are used to fuse and extract the concatenated features to obtain weighted spatial information. After activation by an activation function, a weighted feature map is obtained. The obtained weighted feature map is multiplied with the initial input image to obtain an attention-guided feature map.

[0027] Step 203: Use the concat concatenation method to extract the first output feature of the semantic feature extraction branch. The second output feature of the spatial detail feature enhancement branch The first fusion feature is obtained through fusion. and the first fusion feature Feature extraction is performed to obtain the second fusion feature. ; Step 204: Input the low-resolution feature map into the spatial detail branch, and combine the resulting third branch feature map with the second fused feature map. The final third fusion feature is obtained by fusion. .

[0028] Step 3: Enhance the fused features based on the improved adaptive attention module; Step 3 includes: inputting the fused features into the query branch, key branch, and value branch of the adaptive attention module respectively and performing 1×1 convolution mapping, wherein after the 1×1 convolution of the key branch and the value branch, adaptive pooling is added to obtain the key branch pooled feature map and the value branch pooled feature map respectively; The features obtained after convolving the pooled feature map of the key branch with the features obtained after convolving the query branch are multiplied and then input into the Softmax function to obtain the first attention weights; The output attention weight is obtained by multiplying the first attention weight with the value branch pooling feature map. The obtained output attention weights are added to the fused features to obtain the weighted output features.

[0029] The query branch is converted into an N×C feature after a 1×1 convolution, where N = H×W; the key branch is downsampled to a C×M size after a 1×1 convolution; the value branch is downsampled to a M×C size after a 1×1 convolution, where C is the number of channels, H is the height of the fused feature, and W is the width of the fused feature. Preferably, M ≤ N / 512.

[0030] Step 4: Upsample the low-resolution feature map to the high-resolution image, restore the image size, and perform semantic segmentation.

[0031] Step 4 includes: enhancing the feature map. After two depthwise separable convolutions by the decoder, classification is performed using convolutional layers, followed by an 8x upsampling to obtain an output of the same size as the original input image, thus achieving semantic segmentation step 4. The low-resolution feature map is upsampled to a high-resolution image to restore the image size and achieve semantic segmentation.

[0032] A second aspect of this invention provides a multi-branch image semantic segmentation system based on AM and feature fusion, the network structure of which is as follows: Figure 2 As shown. The system includes a learning downsampling module, a feature fusion module, an adaptive attention module, and an upsampling classifier; among which... The learning downsampling module is used to perform three feature extractions on the input image to be processed in order to obtain a low-resolution feature map; The feature fusion module includes a deep semantic feature extraction branch, a spatial detail feature enhancement branch, and a spatial detail branch, which are used to extract semantic features from low-resolution feature maps and enhance spatial feature information to obtain fused features. An improved adaptive attention module is used to enhance the fused features; Upsampling classifiers are used to upsample low-resolution feature maps to high-resolution images, restore the image size, and achieve semantic segmentation.

[0033] The feature fusion module comprises three branches: a semantic feature extraction branch, a spatial detail branch, and a spatial detail enhancement branch. The semantic feature extraction and spatial detail branches are identical to those in the Fast-SCNN network. This invention adds a third branch—the spatial detail enhancement branch—at the end of the spatial feature extraction branch, incorporating a spatial branch feature attention module to suppress invalid information, enhance more effective spatial detail information, and guide the fusion of semantic features during feature fusion. To further improve the global feature dependency, an adaptive attention module is added after feature fusion, which helps improve image segmentation accuracy.

[0034] To capture the pixel relationships between spatial detail features and further enhance the feature extraction capability of the spatial detail branch, this invention incorporates a Convolutional Block Attention Module (CBAM) into the newly added third branch, forming a spatial detail feature enhancement module. The attention mechanism's starting point is feature enhancement; by performing weighted operations on the feature map, it obtains the mapping relationships between pixel features, strengthens the effective connections between features, and suppresses invalid feature information, thus achieving the effect of feature enhancement. For example... Figure 3As shown, CBAM is a dual attention mechanism that filters effective information in both channel and spatial dimensions, collaboratively learning key details and global information in feature images. It assigns larger weights to important classes in the image and smaller weights to the image background and less important classes, thereby increasing the neural network's attention to the global context and image boundaries, thus improving the network's learning ability and expressive performance. CBAM consists of two parts: a channel attention submodule and a spatial attention submodule.

[0035] The channel attention mechanism models the importance of each channel, differentiating the role of each channel in a given pixel. The channel attention submodule network structure is as follows: Figure 4 As shown. Let the input features be... , , , These represent the number of channels, height, and width of the input feature map, respectively. First, input the feature map... In the spatial dimension, feature compression is performed using max pooling and average pooling to extract weights, resulting in two distinct features. These features are then fed into a Multi-Layer Perceptron (MLP) to obtain two 1×1×C feature maps. Finally, the two distinct feature maps are added together in the spatial dimension to obtain... We use the sigmoid function for activation to obtain the attention weights we need. The final step is the weighting process—weighting the channels of the initial input image to obtain a feature map for feature enhancement. The entire process can be represented by the following formula: (1) (2) (3) (4) in, This represents the Sigmoid activation function.

[0036] The essence of spatial attention mechanisms is to extract pixels from important spatial regions and obtain weight vectors for those regions. The network structure of the spatial attention submodule is as follows: Figure 5 As shown.

[0037] First, take the output of the channel attention submodule Using the input as input, average pooling and max pooling are performed along the channel dimension to compress the data, extract spatial weights, and obtain two types of feature maps. That is: (5) (6) Secondly, the two feature maps with different representations obtained are concatenated along the channel dimension. A convolutional layer is used to fuse and extract the concatenated features to obtain weighted spatial information. This information is then activated by an activation function to obtain a spatial attention feature map. : (7) In the formula, This indicates a convolutional layer with a kernel size of 7×7.

[0038] Finally, the obtained weighted feature map With input feature map Multiplying them together yields a feature map with attention guidance. ,Right now: (8) The feature fusion module first concatenates the extracted semantic features with the enhanced spatial detail features channel by channel to obtain features rich in both semantic and simplified information. Then, it performs a convolution operation to further extract feature information from this feature map. Finally, it adds and fuses the feature map with the spatial detail features to obtain the final fused feature information. The process is as follows: Figure 6 As shown.

[0039] Finally, this invention incorporates an adaptive attention module designed to capture the dependencies between arbitrary pixels in the global feature map. To reduce the computational cost of the original non-local module, this paper proposes an improved adaptive attention module—the Adaptive Global Feature Attention Module (AGFAM), whose structure is as follows: Figure 7 As shown.

[0040] Assuming the input feature signal is X and the output feature is Y, the core of nonlocal operations is: (9) Where Y and X have the same size. It outputs the feature pixel index. It is the input feature pixel index. The binary function f calculates the relationship between any two pixels in the input feature map, while the unary function g calculates the relationship between the input signal at position j. Normalization is performed.

[0041] In the standard Non-local module, the Query branch, Key branch, and Value branch are linearly mapped to features of size C×H×W through 1×1 convolutions respectively, and then converted into features of sizes N×C, C×N, and N×C respectively, where N = H×W. When performing matrix multiplication between the Query branch and the Key branch and then with the Value branch, the computational intensity is high and the efficiency is low. To address this drawback, the AGFAM module adds adaptive pooling after the 1×1 convolutions in the Key branch and the Value branch to perform downsampling to extract some representative features, which are downsampled to sizes C×M and M×C respectively, where M << N. This significantly reduces the computational amount during matrix multiplication. The finally obtained output attention weights are added to the original input features to obtain weighted output features, which enhance the global dependency between any two pixels in the feature map and further strengthen the features.

[0042] Embodiment 1: Multi-branch image semantic segmentation method based on AM and feature fusion: Step 1: Input the input image to be processed into the learning downsampling network for three times of feature extraction to obtain a low-resolution feature map; Specifically, the input image is , and the input image First passes through the first convolution module in the learning downsampling module for the first initial feature extraction, and reduces the resolution to 1 / 2 of the original to obtain the first-resolution feature map , that is (10) Among them, the convolution used for the first initial feature extraction is a standard convolution with a convolution kernel of 3×3, a stride of 2, and an output channel number of 32. The feature map obtained from the first feature extraction is .

[0043] Then, the feature map of the first feature extraction is fed into the depthwise separable convolution for the second feature extraction to obtain the second-resolution feature map, that is (12) Among them, the second feature extraction uses a depthwise separable convolution with a convolution kernel of 3×3, a stride of 2, and an output channel number of 48. The resolution is 1 / 2 of the feature map , that is, 1 / 4 of the input image.

[0044] Continue to perform the third feature extraction of the learning downsampling module to obtain. The feature map passes through the depthwise separable convolution again to extract features and obtain the third-resolution feature map , that is (13) The method used is a depthwise separable convolution with a kernel size of 3×3, a stride of 2, and 64 output channels. The resolution is reduced to half of the original resolution, and the feature map resolution is now 1 / 8 of the input image resolution.

[0045] Step 2: Input the low-resolution feature map into the deep semantic feature extraction branch, the spatial detail feature enhancement branch, and the spatial detail branch respectively for semantic feature extraction and spatial feature enhancement, followed by feature fusion. Existing technologies commonly fuse feature branches of different resolutions using an "+" approach each time. This invention fuses features of the same resolution. For the deep semantic feature extraction branch, the obtained deep semantic features are first upsampled to the same resolution as the spatial detail feature enhancement branch, then fused with the spatial detail feature enhancement branch using a concat concatenation method to obtain features with the same resolution and number of channels as the spatial detail branch. Finally, it is fused with the spatial detail branch features using an "+" approach. Each fusion is followed by a convolution for further feature extraction. This fusion method, designed based on the network in this paper, maximizes the utilization of features from each branch.

[0046] Specifically, step 2 includes: Step 201: Input the low-resolution feature map, i.e., the third-resolution feature map, into the deep semantic feature extraction branch to obtain the first output feature. : In this invention, the deep semantic feature extraction branch consists of a linear bottleneck inverse residual block and a pyramid pooling module. The linear bottleneck inverse residual block is used to continue extracting deep features on the existing feature map, and the pyramid pooling module integrates information at different resolutions.

[0047] Specifically, step 201 includes: Step 2011, the third resolution feature map The input convolutional kernel size is 3×3, the stride is 2, and the output channel number is 64. The output feature map is repeated 3 times, that is, it goes through three identical linear bottleneck inverse residual blocks. The formula is expressed as: (14) in, This indicates that after passing through 3 Botleneck modules, the output feature map is... Its size is 64×128×64.

[0048] Step 2012, feature map Input three bottleneck modules to extract features and obtain feature maps. Its convolution kernel size remains 3×3, stride remains 2, and number of output channels remains 96. The formula is expressed as: (15) in, This indicates that after passing through 3 Botleneck modules, the output feature map is... Its size is 32×64×96.

[0049] Step 2013, feature map Inputting three Botleneck modules for deep feature extraction yields feature maps. It is understandable that the low-resolution feature map, after passing through three sets of linear bottleneck reverse residual blocks, acquires deeper semantic information. At this point, the convolutional kernel is 3×3, the stride is 1, and the number of output channels is 128. (16) in, This indicates that after passing through 3 identical Botleneck modules, the output feature map is... Its size is 32×64×128.

[0050] Step 2014, feature map Input to the pyramid pooling module, fuse multi-scale information, and retain global information at different scales to obtain feature maps. ,Right now (17) Here, PPM represents the pyramid pooling module, and the output feature map size is 32×64×128.

[0051] Step 2015, for the feature map Perform 4x upsampling and depthwise convolution to obtain the feature map. ,Right now (18) Here, Upsample represents the upsampling operation. Preferably, a 4x upsampling is performed, using bilinear interpolation to obtain the feature map. The size is 128×256×128.

[0052] After upsampling, to better represent the features, a convolutional layer is needed to extract more useful features. (19) Here, DWConv represents depthwise convolution, and in this case, a 3×3 depthwise convolution is used with a stride of 1, which does not change the size of the feature map and reduces the amount of computation.

[0053] Step 2016, for the feature map Perform standard convolution to obtain the first output feature. ; It is understood that the first output feature extracted through the deep semantic feature extraction branch in this invention... This needs to be fused with the spatial detail feature enhancement branch, so the number of channels is halved here, also achieved through convolution. To improve accuracy, a standard 3×3 convolution is chosen. (20) Where Conv2D represents standard convolution, and the resulting feature map The size is 128×256×64.

[0054] Step 202: Input the low-resolution feature map, i.e., the third-resolution feature map, into the spatial detail feature enhancement branch to obtain the second output feature. : The second branch in this invention is the spatial detail feature enhancement branch. To capture the pixel relationships between spatial detail features and further enhance the feature extraction capability of the spatial detail branch, a convolutional attention mechanism is added to the spatial detail feature enhancement branch, forming a spatial detail feature enhancement module. The input is fed into a convolutional attention module to enhance the effective connections between features, suppress invalid feature information, and obtain the second output feature. ,Right now (twenty one) Wherein, CBAM represents the convolutional attention module, and the resulting feature map Size and They are the same, both being 128×256×64.

[0055] Step 203: Use the concat concatenation method to extract the first output feature of the semantic feature extraction branch. The second output feature of the spatial detail feature enhancement branch The fusion yields the first fusion feature. The first fusion feature is then used to extract features to obtain the second fusion feature. ,Right now (twenty two) Here, Concat represents the Concatenate cascading operation. This represents the fused feature map.

[0056] Then, through convolution operations, further feature information is extracted from the feature map, i.e. (twenty three) Where Conv2D represents standard convolution, and the resulting feature map The size is 128×256×128.

[0057] Step 204: Input the low-resolution feature map, i.e., the third-resolution feature map, into the spatial detail branch and then fuse the resulting third-branch feature map with the first fused feature map. The final third fusion feature is obtained by fusion. In this step, the fusion method is addition: Specifically, the third-resolution feature map is subjected to standard convolution to obtain the third-branch spatial detail feature map. The third branch spatial detail feature map With the second fusion feature The features are added together and fused to obtain the final feature fusion information. ,Right now (twenty four) (15) Here, Conv2D represents a standard 1×1 convolution, which obtains the spatial detail feature map of the third branch. The feature map obtained from the first fusion With the third branch feature map The final fusion feature is obtained by fusion through addition. Its size is 128×256×128.

[0058] Step 3: Enhance the fused features based on the improved adaptive attention module; This invention incorporates an adaptive attention module after feature fusion to capture the dependencies between arbitrary pixels in the global feature map. Understandably, the non-local module calculates the similarity between the current point and other points, normalizes this similarity to obtain the weight of each point, multiplies this weight by the corresponding feature mapping value, and sums the results to obtain the output of the current point. The output contains global information, where similarity represents the contribution of other points to this point; the greater the connection between other points and this point, the greater the similarity and the greater the impact on the output.

[0059] The improved adaptive attention module proposed in this invention enhances the global dependency between any two pixels in the feature map, further strengthening the features.

[0060] In the multi-branch feature extraction method in the prior art, feature enhancement is not performed on all branches. Compared with the prior art, in this article, for the third branch, the spatial detail feature enhancement branch, CBAM is used to further enhance the obtained spatial detail features, capture the mutual connections between spatial features, further enhance the feature extraction ability of the spatial detail branch, and compensate for the partial loss of spatial information features.

[0061] In the standard Non-local module, the Query branch, Key branch, and Value branch are linearly mapped to features of size C×H×W through 1×1 convolutions respectively, and then are respectively converted into features of size N×C, C×N, and N×C, where N = H×W. When performing matrix multiplication between the Query branch and the Key branch and then with the Value branch, the computational effort is large and the efficiency is low. As Figure 7 shown, to address this drawback, the present invention proposes adding adaptive pooling after the 1×1 convolution of the Key branch and the Value branch to perform downsampling to extract some representative features, which are respectively downsampled to sizes of C×M and M×C, where M << N. The computational amount will be greatly reduced during matrix multiplication. The finally obtained output attention weights are added to the original input features to obtain the weighted output features, which enhance the global dependence relationship between any two pixels in the feature map and further strengthen the features. Here, C is the number of channels, H is the height of the input feature map, and W is the width of the input feature map; the Query branch, Key branch, and Value branch respectively represent the query branch, key branch, and value branch. Common pooling operations include max pooling, average pooling, L2 pooling, and adaptive pooling. In addition to the above common pooling methods, there are also some less used pooling methods, such as random pooling and min pooling, etc. Adaptive pooling has the advantages of flexibility, adaptability, and high precision. Since this application needs to dynamically adjust the pooling window size according to the size of the input feature map, adaptive pooling is adopted.

[0062] To reduce the computational amount of the existing Non-local module, the present invention proposes an improved adaptive attention module - Adaptive Global Feature Attention Module (AGFAM). The fused features are sent into the AGFAM module, that is (25) where AGFAM represents the feature map with enhanced features obtained after the operation of this module .

[0063] Specifically, the fused features of size 128×256×128 The 1×1 convolutional linear mapping through the Query branch, Key branch, and Value branch respectively yields features of size 128×256×128. The mapped features are then transformed into N×128 features through the Query branch, where N = 256×128. After passing through the Key and Value branches, the mapped features undergo adaptive pooling, and representative features are extracted through downsampling. Preferably, the 128×256×128 features are downsampled to 128×M and M×128 sizes respectively, where C is the number of channels, H is the height of the fused feature, and W is the width of the fused feature. Preferably, M ≤ N / 512. For example, if N = 256×128 = 32768, take M = 8×8 = 64, where M is 1 / 512 of N, which greatly reduces the computational cost during matrix multiplication.

[0064] Step 4: Perform convolution and classification on the feature maps that have been enhanced with features, and then output the semantic segmentation results.

[0065] Specifically, step 4 includes: Step 401, Enhance the feature map The first depthwise separable convolution after the decoder, i.e. (26) Where DSConv represents a depthwise separable convolution with a kernel size of 3×3 and a stride of 1, and the output is... .

[0066] Subsequently, the same module is used to perform the same operation, that is... (27) Where DSConv represents a depthwise separable convolution with a kernel size of 3×3 and a stride of 1, and the output is... .

[0067] Step 402: Use convolutional layers to perform classification, dividing the data into 19 categories. (28) Where Conv2D represents a standard convolution with a kernel size of 3×3, a stride of 1, and 19 output channels, outputting a feature map. The size is 128×256×19.

[0068] Step 403, process the input feature map By performing an 8x upsampling, an output image of the same size as the original input image is obtained, thus achieving semantic segmentation. (29) in, This indicates an 8x upsampling, and the output is... .

[0069] To verify the method proposed in this invention, the following specific embodiments are adopted, and the detailed parameters of the embodiments are shown in Table 1.

[0070] Table 1 Experimental Detail Parameters

[0071] This embodiment uses the original Fast-SCNN network as a benchmark, and adds CBAM and AGFAM modules for training and testing. The results are shown in Table 2. The evaluation metrics are: Mean Intersection over Union (MIoU), frame rate (FPS), prediction time per image (Time), and number of network structural parameters (Params).

[0072] Table 2 Ablation Experiment Results

[0073] The results show that each part of the network structure in this paper improves the accuracy of the overall network model. The model achieves a trade-off between speed and accuracy, improving accuracy without sacrificing the requirements of real-time and lightweight network. It is an algorithm model for semantic segmentation of high-resolution images.

[0074] Experiments were conducted on the Cityscapes dataset and compared with typical real-time lightweight semantic segmentation networks ERFNet, ENet, ICNet, DABNet, and Fast-SCNN. The results are shown in Table 3.

[0075] Table 3 Performance Comparison of Different Semantic Segmentation Models

[0076] As shown in Table 3, with an input image size of 1024×2048, this paper achieves an accuracy of 71.55% MIoU without any pre-training or other assistance, a segmentation speed of 97.6 frames / s, and a model parameter count of only 1.39M. This is the best performing model among the listed methods, showing improvements of 3.55%, 13.25%, 2.05%, 1.45%, and 3.55% respectively compared to the comparison methods, and achieving a runtime of 97.6 FPS. While not as fast as the original Fast-SCNN network in inference speed, it is faster than the other comparison methods and far exceeds real-time requirements.

[0077] The beneficial effects of this invention are that, compared with existing technologies, it solves the embedding problem of model algorithms in embedded devices, especially low-memory embedded devices and mobile hardware devices. While ensuring segmentation accuracy, it also guarantees computational efficiency and real-time performance, with low computational cost and a lightweight model. It introduces a spatial feature detail enhancement branch and a convolutional attention module to enhance the global feature representation of the network and better utilize shallow features. Simultaneously, to compensate for the loss of detailed information in deep semantic features, the shallow features of the spatial feature detail enhancement branch are aggregated with the deep semantic features, and the shallow features are weighted and fused together. Furthermore, to better capture the interdependencies between long-distance pixels, an adaptive attention module is introduced after feature fusion, thereby improving the overall expressive power of the network. This invention achieves high accuracy with a relatively small number of parameters while maintaining a fast segmentation speed, achieving a reasonable balance between real-time efficiency and accuracy. The segmentation method provided by this invention is suitable for situations with a large number of power grid devices, especially in power systems, where the number of devices is large, the types are diverse, and the occlusion relationships are complex.

[0078] The beneficial effects of the present invention also include: 1. This invention aims to balance the real-time performance and accuracy of network models. It introduces a spatial feature detail enhancement branch into the Fast_SCNN network. The spatial feature detail enhancement branch includes an attention module to capture the interrelationships between spatial features and further enhance the feature extraction capability of the spatial detail branch.

[0079] 2. Through reasonable design, this invention creates an aggregation module with three branches, maximizing the use of the feature information of each branch to achieve better segmentation accuracy.

[0080] 3. After feature aggregation, this invention introduces an improved adaptive attention module to add weighted attention to the original fused feature map, thereby obtaining the dependency relationship between any two pixels in the feature map and further enhancing the features. This disclosure can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this disclosure.

[0081] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0082] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0083] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0084] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the protection scope of the claims of the present invention.

Claims

1. A multi-branch image semantic segmentation method based on AM and feature fusion, characterized in that, The method includes: Step 1: Input the image to be processed into the learning downsampling network to perform three feature extractions to obtain a low-resolution feature map; Step 2 involves inputting the low-resolution feature map into the deep semantic feature extraction branch, the spatial detail feature enhancement branch, and the spatial detail branch respectively for semantic feature extraction and spatial feature enhancement. Feature fusion is then performed to obtain fused features. The deep semantic feature extraction branch and the spatial detail branch are the same as those in the Fast-SCNN network. The spatial detail feature enhancement branch includes a convolutional block attention module (CBAM). Step 2 specifically includes: Step 201: Input the low-resolution feature map into the deep semantic feature extraction branch to obtain the first output feature. : Step 202: Input the low-resolution feature map into the spatial detail feature enhancement branch to obtain the second output feature. ; Step 203: Use the concat concatenation method to extract the first output feature of the deep semantic feature extraction branch. The second output feature of the spatial detail feature enhancement branch The first fusion feature is obtained through fusion. and the first fusion feature Feature extraction is performed to obtain the second fusion feature. ; Step 204: Input the low-resolution feature map into the spatial detail branch, and combine the resulting third branch feature map with the second fused feature map. The final third fusion feature is obtained by fusion. ; Step 3: Enhance the fused features based on the Adaptive Global Feature Attention (AGFAM) module to obtain enhanced feature maps. Step 3 includes: inputting the fused features into the query branch, key branch, and value branch of the AGFAM module for 1×1 convolution mapping. After the 1×1 convolution of the key branch and value branch, adaptive pooling is added to obtain the key branch pooled feature map and the value branch pooled feature map, respectively. After the 1×1 convolution of the query branch, it is converted into a feature of size N×C, where N=H×W. After the 1×1 convolution of the key branch, adaptive pooling is added, and it is downsampled to size C×M. After the 1×1 convolution of the value branch, adaptive pooling is added, and it is downsampled to size M×C. Where C is the number of channels, H is the height of the fused feature, and W is the width of the fused feature, where M≤N / 512. Step 4: Classify and upsample the feature maps of the feature enhancement to restore the image size and achieve semantic segmentation.

2. The multi-branch image semantic segmentation method based on AM and feature fusion according to claim 1, characterized in that, Step 1 includes: the input image is sequentially passed through a standard convolutional network, a first depthwise separable convolutional network, and a second depthwise separable convolutional network to extract features, resulting in a low-resolution feature map. Among them, low-resolution feature maps The resolution is 1 / 8 of the input image.

3. The multi-branch image semantic segmentation method based on AM and feature fusion according to claim 1, characterized in that, Step 2 includes inputting the processed low-resolution feature map into the deep semantic feature extraction branch to obtain deep semantic features; upsampling the deep semantic features to the same resolution as the spatial detail feature enhancement branch, and then fusing them with the spatial detail feature enhancement branch using the concat concatenation method to obtain features with the same resolution and number of channels as the spatial detail feature enhancement branch; finally, fusing them with the features of the spatial detail branch using the feature addition method.

4. The multi-branch image semantic segmentation method based on AM and feature fusion according to claim 3, characterized in that, Step 201 specifically includes: converting the low-resolution feature map... Three sets of linear bottleneck reverse residual blocks are input sequentially for deep feature extraction to obtain feature maps. ; feature map Input to the pyramid pooling module, fuse multi-scale information, and retain global information at different scales to obtain feature maps. ; for feature maps Perform 4x upsampling and depthwise convolution to obtain the feature map. ; for feature maps Perform standard convolution to obtain the first output feature. .

5. The multi-branch image semantic segmentation method based on AM and feature fusion according to claim 1, characterized in that, In step 202, the spatial detail feature enhancement branch is used to collect the pixel relationships between spatial detail features. The low-resolution feature map is input into the convolutional block attention module (CBAM) to obtain the second output feature. The second output feature has the same size as the first output feature.

6. The multi-branch image semantic segmentation method based on AM and feature fusion according to claim 1, characterized in that, The Convolutional Block Attention Module (CBAM) includes a channel attention submodule and a spatial attention submodule. The channel attention submodule performs feature compression on the low-resolution feature map in the spatial dimension through max pooling and average pooling, extracts weights, and obtains two different features. The two features are then input into the multilayer perceptron and added together in the spatial dimension to obtain the attention weights. The channels of the initial input image are then weighted to obtain the feature map with enhanced features. The spatial attention module takes the output of the channel attention submodule as input, performs average pooling and max pooling in the channel dimension, compresses the data, extracts spatial weights, and obtains two types of feature maps. The two feature maps with different representations are concatenated in the channel dimension. Convolutional layers are used to fuse and extract the concatenated features to obtain weighted spatial information. After activation by an activation function, a weighted feature map is obtained. The obtained weighted feature map is multiplied with the initial input image to obtain an attention-guided feature map.

7. The multi-branch image semantic segmentation method based on AM and feature fusion according to claim 1, characterized in that, Step 3 also includes: After the fused features are respectively input into the query branch, key branch, and value branch of the Adaptive Global Feature Attention Module (AGFAM) and mapped by 1×1 convolution, the feature obtained by convolving the pooled feature map of the key branch with the feature obtained by the query branch is multiplied and then input into the Softmax function to obtain the first attention weight; The output attention weight is obtained by multiplying the first attention weight with the value branch pooling feature map. The obtained output attention weights are added to the fused features to obtain the weighted output features.

8. The multi-branch image semantic segmentation method based on AM and feature fusion according to claim 1, characterized in that, Step 4 includes: enhancing the feature map. After two depthwise separable convolutions by the decoder, classification is performed using the convolutional layer, followed by an 8x upsampling to obtain an output of the same size as the original input image, thus achieving semantic segmentation.

9. A multi-branch image semantic segmentation system based on AM and feature fusion, used to implement the multi-branch image semantic segmentation method based on AM and feature fusion as described in any one of claims 1-8, characterized in that: The system includes a learning downsampling module, a feature fusion module, an adaptive global feature attention module (AGFAM), and an upsampling classifier. in The learning downsampling module is used to perform three feature extractions on the input image to be processed in order to obtain a low-resolution feature map; The feature fusion module includes a deep semantic feature extraction branch, a spatial detail feature enhancement branch, and a spatial detail branch, which are used to extract semantic features from low-resolution feature maps, enhance spatial feature information, and fuse features to obtain fused features. The Adaptive Global Feature Attention (AGFAM) module is used to enhance the fused features, resulting in an enhanced feature map. Upsampling classifiers are used to classify and upsample feature maps with enhanced features, restore image size, and achieve semantic segmentation.

10. A terminal, comprising a processor and a storage medium; characterized in that: The storage medium is used to store instructions; The processor is configured to operate according to the instructions to perform the steps of the method according to any one of claims 1-8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method according to any one of claims 1-8.

Images