Lightweight high-precision semantic segmentation method and system

Abstract

The invention discloses a lightweight high-precision semantic segmentation method and system, and relates to the technical field of image segmentation. According to the method, a lightweight feature extraction backbone network is constructed, an edge enhanced convolution pooling pyramid is provided, gating attention is mixed, a multi-branch cascade up-sampling decoder is constructed, four-layer features of a backbone network MobileNetv2 are extracted in an encoder stage, and an edge guided cavity space convolution pooling pyramid is designed. Effective features obtained from a backbone network are fully utilized, the ability of a model to capture and highlight object boundaries is improved, in a decoder stage, a hybrid gating attention module is designed, complementary important features of space, channels and contexts are dynamically integrated, the ability of network global interaction information is enhanced, and the network performance is improved. A multi-branch cascade up-sampling decoder is constructed, layer-by-layer up-sampling is accurately and efficiently carried out, the high precision of semantic segmentation is kept, meanwhile, the parameter quantity of the model is greatly reduced, and deployment on mobile equipment is facilitated.

Description

Technical Field

[0001] The present invention relates to the technical field of image segmentation, and particularly to a lightweight and high-precision semantic segmentation method and system. Background Art

[0002] In the field of image semantic segmentation, traditional network models have significant defects, such as SegNet, Unet, FCN, etc. These models have complex structures and often have billions of parameters. Taking the medical image segmentation model as an example, its calculation consumes a large amount of memory and computing power. This makes it difficult to achieve rapid deployment and real-time processing in mobile devices with small memory (such as mobile medical diagnosis APPs) and embedded devices with limited computing power (such as in-vehicle vision modules). With the popularization of 5G and the development of the Internet of Things, the demand for real-time semantic segmentation in edge devices has increased sharply. The semantic segmentation network based on Deeplabv3+ has broad engineering application prospects due to its simple network structure, small number of parameters, and high accuracy. Developing a lightweight and high-precision semantic segmentation network has become the key direction to solve the current problems and meet the needs of various scenarios.

[0003] The feature extraction network MobileNet series is widely used for its lightweight and powerful feature extraction ability. MobileNetv1 introduced depthwise separable convolution, which decomposes the standard convolution into depth convolution and pointwise convolution, significantly reducing the amount of computation and the number of parameters, but no additional non-linear activation layer was added. MobileNetv2 proposed the inverted residual structure and linear bottleneck layer on this basis, significantly improving the feature expression ability and providing higher accuracy under similar computing resources. Although introducing the MobileNet series as the backbone network of Deeplabv3+ improves the application potential of the network in real-time and low-resource environments, the segmentation accuracy of small object edge features and the overall segmentation accuracy need to be improved.

[0004] Currently, the Deeplabv3+ semantic segmentation network based on the feature extraction backbone network Xception or ResNet101 with a large number of parameters has a high segmentation accuracy, but its large number of parameters leads to inference latency on mobile devices, limiting its application potential in real-time and low-resource environments; moreover, the upsampling method of the Deeplabv3+ network is single and rough, resulting in feature loss, making it difficult to capture detailed features and unable to efficiently fuse shallow information and high-level semantic information. Therefore, while significantly compressing the number of parameters and enhancing the generalization ability of the model on mobile devices, it is also necessary to fully extract feature information, enhance the ability of the network to globally interact with information, and ensure high accuracy of the model in complex scene segmentation, which is the core technical problem that those skilled in the art urgently need to solve. Summary of the Invention

[0005] The object of the present invention is to provide a lightweight high-precision semantic segmentation method and system to solve the technical problems raised in the background art.

[0006] To achieve the above object, the present invention provides the following technical solutions: A lightweight high-precision semantic segmentation method, at least including the following steps:

[0007] S1: Construct a lightweight feature extraction network, namely MobileNetv2, and use the lightweight feature extraction network MobileNetv2 as the backbone network to replace the original backbone network Xception. While deeply extracting features, the number of parameters is greatly reduced, and the deployment ability on mobile devices is improved;

[0008] S2: Propose an edge-guided atrous spatial pyramid, and fuse the extracted multi-scale edge information with the deep feature branch passing through the backbone network to strengthen edge perception.

[0009] S3: Design a hybrid gating attention mechanism to dynamically integrate complementary spatial, channel, and context important features.

[0010] S4: Construct a multi-branch cascaded up-attention decoder, introduce a hybrid gating attention mechanism into the deep feature branch, and efficiently and accurately upsample step by step to restore the feature map.

[0011] Further, the lightweight feature extraction network is used to extract four layers of deep features and shallow features to form multi-scale capture of global features;

[0012] The application of the lightweight feature extraction network at least includes the following steps:

[0013] Perform data augmentation operations on the input RGB image to expand sample diversity and improve the generalization of the model. The data augmentation operations include but are not limited to rotation, translation, photometric transformation, and Gaussian model processing.

[0014] Crop a template image with a size of 512×512×3 on the template frame from the data-augmented RGB image as the input of the network;

[0015] Extract the 1 / 2, 1 / 4, 1 / 8, and 1 / 16 downsampled feature maps of MobileNetv2, and denote them as features 1 / 2 、features 1 / 4 、features 1 / 8 、features 1 / 16 .

[0016] Further, the application process of the edge-guided atrous spatial pyramid at least includes the following steps:

[0017] First, generate the feature map. Based on ASPP, add a dilated convolution with a dilation rate of 3 to enhance the perception ability of the detailed features of small objects. Through four convolution operations with different dilation rates, obtain receptive fields of different scales, and introduce an edge enhancement module to expand the network, retaining more edge and texture features and enhancing the multi-scale edge information perception ability;

[0018] Secondly, perform edge enhancement. Send the outputs of the four dilated convolution blocks with dilation rates of 3, 6, 12, and 18 respectively in the edge-guided atrous spatial pyramid pooling into Edge Enhancers (EE) to highlight the object edges in the feature map;

[0019]

[0020] Among them is the average pooling operation performed on to extract global information, thereby reducing the influence of the background;

[0021] Then, perform 1x1 convolution for edge enhancement. Send the edge-enhanced feature map into a 1x1 convolution layer. The role of the 1x1 convolution layer is to reduce the number of channels of the feature map, and then add it to the original dilated convolution feature map to strengthen the detailed information of each scale and obtain edge-enhanced feature maps of different scales Refer to the following formula:

[0022]

[0023] Then, perform multi-scale concatenation. Concatenate and fuse the four edge-enhanced feature maps of different scales to obtain the multi-scale edge feature F mef , refer to the following formula:

[0024]

[0025] F mef includes both fine-grained detailed information and multi-scale edge information;

[0026] Finally, perform global feature and edge feature fusion. Refer to the following formula:

[0027]

[0028] Concatenate F mef and the global average pooling global_feature along the channel dimension, and use a 1×1 convolution layer to reduce the dimension of the fused feature to obtain the output feature F ma_out .

[0029] Furthermore, the hybrid gated attention mechanism in S3 enables the model to dynamically adjust the feature combination according to the importance of different attention mechanisms through split convolution and gated weight multiplication operations, thereby flexibly adjusting the weights of each feature, enhancing the model's expressive ability, and improving the decision-making accuracy of global feature extraction. The hybrid gated attention mechanism specifically includes the following steps:

[0030] First, CoT is used to enhance the feature context information, and at the same time, ResNeSt is used to strengthen the channel information. After the feature map Fea is input into the CoT block, a k×k grouped convolution is first performed on the input key K through a 3×3 convolution to obtain the local context information representation K of the input, and the static context representation of the input is obtained. The dynamic multi-head attention matrix is learned through two consecutive 1×1 convolutions. See the following formula:

[0031] A = [K 1 , Q]W θ W δ

[0032] where W θ has an activation function, and W δ does not have an activation function;

[0033] This attention matrix A is learned based on the query feature Q and the contextualized static key feature K 1 That is, A is the local attention matrix for each spatial position learned based on the query feature and the contextualized key feature. This method enhances self-attention learning under the additional guidance of the mined static context K 1 ;

[0034] The learned attention matrix is multiplied by the input value to achieve the dynamic context representation K 2 of the input. See the following formula:

[0035] K 2 = V * A

[0036] where V is the values;

[0037] The fusion of the static and dynamic context representations is used as the output Y1;

[0038] Y1 = K 1 + K 2

[0039] The ResNeSt branch splits the input feature map Fea into k groups, each group is denoted as Cardinal, and then each group Cardinal is split into R Split blocks. There are a total of G = KR groups, increasing the network width;

[0040] Then, a 3×3 global average pooling across spatial dimensions is used to collect global context information with embedded channel statistics. The calculation formula for the c-th component is as follows:

[0041]

[0042] Where represents the value of the c-th channel at the spatial position (i, j), and H and W are the height and width of the input image respectively;

[0043] The weighted fusion represented by the cardinality array is performed using channel-wise soft attention aggregation. Each feature map channel is generated using a weighted combination after splitting. The calculation formula for the c-th component is as follows:

[0044]

[0045] Where, U R(k-1)+i represents the intermediate feature generated by independent convolution operations on the R branches of features within the k-th cardinality array, is the assignment weight of the soft attention, which is expressed as:

[0046]

[0047] Where, represents determining the weight of each segmentation of the c-th channel according to the global context representation s k ;

[0048] Finally, the cardinality array representations are linearly concatenated along the channel dimension, and the output is:

[0049] Y2 = V + X

[0050] For dynamic gated feature fusion, the output Y1 of the CoT block and the output Y2 of ResNeSt are concatenated together by multiplying with gated weights to achieve dynamic fusion of features in multiple channels and at multiple levels. Y1 and Y2 are concatenated along the channel dimension and activated by the non-linear function Sigmoid to obtain the gated weight w:

[0051] w = Sigmoid(Y1 + Y2)

[0052] By linearly concatenating the results of multiplying Y1 and Y2 with the corresponding gated weight w, a feature map Y that fuses different scales is obtained feature :

[0053] Y feature = Y1 × w + Y2 × (1 - w)

[0054] Finally, permutation attention is performed on the gated fusion feature map Y featureIntroduce permutation attention to inhibit the propagation of irrelevant information and enhance the representation of important features;

[0055] Divide the input image Y feature into G groups, Y feature = [Y f1 , Y f2 , … Y fG , and each group constructs channel attention and spatial attention with one SA unit simultaneously;

[0056] Y fk forms two branches along the channel dimension, namely Y fk1 and Y fk2 , where Y fk1 utilizes the mutual relationship of channels to generate a channel attention map, and Y fk2 utilizes the spatial relationship of features to generate a spatial attention map;

[0057] Embed global information using global average pooling and generate channel weights s:

[0058]

[0059] where, F gp (Y fk1 ) is the average value of all positions of the feature map Y fk1 in the spatial dimension;

[0060] The channel attention mechanism Y′ fk1 is:

[0061] Y′ fk1 = Sigmoid(W1s + b1) · Y fk1

[0062] where, W1 is the weight matrix of channel attention; b1 is the bias term; Sigmoid() is the activation function for generating normalized weights;

[0063] The spatial attention mechanism Y′ fk2 is:

[0064] Y′ fk2 = Sigmoid(W2 · GN(Y fk2 ) + b2) · Y fk2

[0065] where, W2 is the weight matrix of spatial attention; b2 is the bias term; GN(Y fk2 ) is used to normalize each group of channels of Y fk2 to stabilize the training process; Sigmoid() is the activation function of spatial attention;

[0066] where \(W1, b1, W2, b2\in R\) C / 2G×1×1 ;

[0067] Then connect the two branches so that the number of channels is the same as the number of input channels, i.e.:

[0068] Y′ fk =[Y′ fk1 , Y′ fk2

[0069] Finally, aggregate the cross-group information transfer along the channel dimension between different sub-features through the "channel shuffle" operator to obtain the feature map Y after multi-scale feature fusion output ;

[0070] Y output =channel shuffle([Y′ f1 , Y′ f2 ,…Y′ fG )

[0071] Y′ f1 , Y′ f2 ,..., Y′ fG represents that each branch Y′ f is aggregated into a large feature map. The channel shuffle operation helps the network share information between multiple scales and multiple branches by rearranging and mixing the channel information between different groups, thus achieving a richer feature representation.

[0072] Furthermore, the construction of the multi-branch cascaded up-attention decoder at least includes the following steps:

[0073] In the MobileNetV2 backbone network, the original feature map is successively decomposed into four feature maps features of different scales of 1 / 2, 1 / 4, 1 / 8, and 1 / 16 i , i = 1 / 2, 1 / 4, 1 / 8, 1 / 16, and both the 1 / 8 feature map and the 1 / 16 feature map contain rich and diverse semantic features and global features;

[0074] Send the 1 / 16 feature map containing the richest semantic features into the edge-guided dilated spatial convolutional pooling pyramid to extract features, and then send its output into the hybrid gating attention mechanism for dynamic multi-scale edge feature extraction to obtain the high-level feature map fea′ 1 / 16 , see the following formula:

[0075] fea′ 1 / 16 =h(m(features 1 / 16 ))

[0076] fea′ 1 / 16 ​Upsampling 2x and features 1 / 8 Splicing, apply a hybrid gated attention operation to dynamically integrate spatial, channel, and contextual feature information, see the following formula:

[0077] fea′ 1 / 8 =h(Concat(features 1 / 8 ,Up2(fea′ 1 / 16 )))

[0078] Among them, m is the edge-guided atrous spatial convolution pooling pyramid operation, and h is the hybrid gated attention operation;

[0079] features 1 / 2 、features 1 / 4 It is a shallow feature map, which contains features such as color and texture, but has low semantics and contains more noise. It is directly upsampled step by step to save computing resources;

[0080] fea′ 1 / 8 Upsampling 2x and features 1 / 4 Perform splicing and fusion, continue upsampling by 2 times, and combine with features 1 / 2 Get the multi-scale cascade feature enhancement feature map fea mcfe , see the following formula:

[0081] fea mcfe =Concat(Up2(Concat(Up2(fea′ 1 / 8 ),features 1 / 4 )))

[0082] fea mcfe It is an enhanced map of multi-scale features that integrates information at different scales, including low-level features and high-level semantic information.

[0083] A lightweight and high-precision semantic segmentation system, comprising at least a data acquisition module, a feature extraction module, a feature enhancement module, a feature fusion module, a feature cascade upsampling module and a semantic segmentation prediction module;

[0084] The data acquisition module is used to acquire and preprocess original data;

[0085] The feature extraction module is used to obtain high-level semantic features and low-level semantic features through the backbone network;

[0086] The feature enhancement module is used for fine-grained edge detail feature enhancement of high-level semantic features;

[0087] The feature fusion module is used to dynamically enhance the ability of the network to globally interact with multi-scale feature information;

[0088] The feature cascaded upsampling module is used for progressive interactive restoration of the feature map;

[0089] The semantic segmentation prediction module is used to map the class labels through a pseudo-color mask after predicting the pixel categories of the input image through the model, and superimpose them on the original image to generate a visual result of pixel-level semantic annotation.

[0090] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0091] The present invention constructs a lightweight feature extraction backbone network, and proposes an edge-enhanced convolutional pooling pyramid, hybrid gated attention, and constructs a multi-branch cascaded upsampling decoder. Four layers of features of the backbone network MobileNetv2 are extracted in the encoder stage, and an edge-guided atrous spatial convolutional pooling pyramid is designed to make full use of the effective features obtained from the backbone network and improve the model's ability to capture the boundaries of prominent objects. In the decoder stage, a hybrid gated attention module is designed to dynamically integrate complementary spatial, channel, and context important features, enhance the network's ability to interact globally, and construct a multi-branch cascaded upsampling decoder to accurately and efficiently upsample layer by layer. While maintaining high-precision semantic segmentation, the present invention significantly reduces the number of model parameters, facilitates deployment on mobile devices, and has high application value in real-time and low-resource environments. Description of the Drawings

[0092] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0093] Figure 1 It is the overall flowchart of the method of the present invention;

[0094] Figure 2 It is the overall flowchart of the segmentation system of the present invention;

[0095] Figure 3 It is the visualization result diagram of the present invention. Detailed Embodiments

[0096] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments.

[0097] Embodiment 1:

[0098] Refer to Figure 1 and Figure 3, A lightweight high-precision semantic segmentation method, at least including the following steps:

[0099] S1: Construct a lightweight feature extraction network, namely MobileNetv2. Use the lightweight feature extraction network MobileNetv2 as the backbone network to replace the original backbone network Xception. While deeply extracting features, significantly reduce the number of parameters and improve the deployment ability on mobile devices;

[0100] S2: Propose an edge-guided atrous spatial pyramid. Fuse the extracted multi-scale edge information with the deep feature branch passing through the backbone network to strengthen edge perception.

[0101] S3: Design a hybrid gating attention mechanism to dynamically integrate complementary spatial, channel, and context important features.

[0102] S4: Construct a multi-branch cascaded up-attention decoder. Introduce the hybrid gating attention mechanism into the deep feature branch to efficiently and accurately upsample and recover the feature map step by step.

[0103] The lightweight feature extraction network is used to extract four layers of deep features and shallow features to form multi-scale capture of global features;

[0104] The application of the lightweight feature extraction network at least includes the following steps:

[0105] Perform data augmentation operations on the input RGB image to expand sample diversity and improve model generalization. The data augmentation operations include but are not limited to rotation, translation, photometric transformation, and Gaussian model processing.

[0106] Crop a template image with a size of 512×512×3 from the data-augmented RGB image on the template frame as the input of the network;

[0107] Extract the 1 / 2, 1 / 4, 1 / 8, and 1 / 16 downsampled feature maps of MobileNetv2, denoted as features 1 / 2 、features 1 / 4 、features 1 / 8 、features 1 / 16 .

[0108] The application process of the edge-guided atrous spatial pyramid at least includes the following steps:

[0109] First, generate the feature map. Based on ASPP, add a dilated convolution with a dilation rate of 3 to enhance the perception ability of small object detail features. Through 4 convolution operations with different dilation rates, obtain receptive fields of different scales, and introduce an edge enhancement module to expand the network, retain more edge and texture features, and enhance the multi-scale edge information perception ability;

[0110] Secondly, edge enhancement is performed. The outputs of the atrous convolution blocks with dilation rates of 3, 6, 12, and 18 respectively in all the atrous spatial pyramid pooling with edge guidance are sent into Edge Enhancers (EE) to highlight the object edges in the feature map;

[0111]

[0112] where is the average pooling operation performed on to extract global information and thus reduce the influence of the background;

[0113] Then, 1x1 convolution is performed for edge enhancement. The edge-enhanced feature map is sent into a 1x1 convolution layer. The role of the 1x1 convolution layer is to reduce the number of channels of the feature map, and then it is added to the original atrous convolution feature map to strengthen the detailed information of each scale and obtain edge-enhanced feature maps of different scales See the following formula:

[0114]

[0115] Then, multi-scale splicing is performed. The 4 edge-enhanced feature maps of different scales are spliced and fused to obtain the multi-scale edge feature F mef , see the following formula:

[0116]

[0117] F mef includes both fine-grained detail information and multi-scale edge information;

[0118] Finally, global feature and edge feature fusion is performed. See the following formula:

[0119]

[0120] F mef is concatenated with the global average pooling global_feature along the channel dimension, and a 1×1 convolution layer is used for dimensionality reduction on the fused feature to obtain the output feature F ma_out .

[0121] The hybrid gated attention mechanism in S3 enables the model to dynamically adjust the feature combination according to the importance of different attention mechanisms through the operations of split convolution and gated weight multiplication, thereby flexibly adjusting the weights of each feature, enhancing the expressive power of the model, and improving the decision-making accuracy of global feature extraction. The hybrid gated attention mechanism specifically includes the following steps:

[0122] First, CoT is used to enhance the feature context information, while ResNeSt is used to strengthen the channel information. After the feature map Fea is input into the CoT block, a grouped convolution of k×k is first performed on the input key K through a 3×3 convolution to obtain the local context information representation K of the input, and the static context representation of the input is obtained. The dynamic multi-head attention matrix is learned through two consecutive 1×1 convolutions. See the following formula:

[0123] A = [K 1 , Q]W θ W δ

[0124] where W θ has an activation function, and W δ has no activation function;

[0125] This attention matrix A is learned based on the query feature Q and the contextualized static key feature K 1 That is, A is the local attention matrix for each spatial position learned based on the query feature and the contextualized key feature. This method enhances self-attention learning under the additional guidance of the mined static context K 1 ;

[0126] The learned attention matrix is multiplied by the input value to achieve the dynamic context representation K 2 of the input. See the following formula:

[0127] K 2 = V * A

[0128] where V is the values;

[0129] The fusion of the static and dynamic context representations is used as the output Y1;

[0130] Y1 = K 1 + K 2

[0131] The ResNeSt branch splits the input feature map Fea into k groups, each group is denoted as Cardinal, and then each group of Cardinal is split into R Split blocks, with a total of G = KR groups, increasing the network width;

[0132] Then, a 3×3 global average pooling across the spatial dimension is used to collect the global context information with embedded channel statistics. Among them, the calculation formula for the c-th component is:

[0133]

[0134] where Denote the value at the spatial position (i, j) of c channels. H and W are the height and width of the input image respectively;

[0135] The weighted fusion represented by the base array is performed using soft attention aggregation in a channel-wise manner. Each feature map channel is generated using a weighted combination after splitting. Among them, the calculation formula for the c-th component is:

[0136]

[0137] Among them, U R(k-1)+i Denote the intermediate features generated by independent convolution operations on the R branch features within the k-th base array. Is the assignment weight of the soft attention, which is expressed as:

[0138]

[0139] Among them, Denote the weight of each segmentation of the c-th channel determined according to the global context representation s k ;

[0140] Finally, the base array representations are linearly concatenated along the channel dimension, and the output is:

[0141] Y2 = V + X

[0142] For dynamic gated feature fusion, the output Y1 of the CoT block and the output Y2 of ResNeSt are concatenated together by multiplying with gated weights to achieve dynamic fusion of features in multiple channels and multiple levels. Y1 and Y2 are concatenated along the channel dimension and activated by the non-linear function Sigmoid to obtain the gated weight w:

[0143] w = Sigmoid(Y1 + Y2)

[0144] By linearly concatenating the results of multiplying Y1 and Y2 by the corresponding gated weight w, the feature map Y that fuses different scales is obtained feature :

[0145] Y feature = Y1 × w + Y2 × (1 - w)

[0146] Finally, for permutation attention, the gated fusion feature map Y feature Is fed into the permutation attention to suppress the propagation of irrelevant information and enhance the representation of important features;

[0147] The input image Y feature Is divided into G groups, Y feature = [Y f1 , Y f2 , … Y fG, one SA unit in each group constructs channel attention and spatial attention simultaneously;

[0148] Y fk Form two branches along the channel dimension, namely Y fk1 and Y fk2 , where Y fk1 Utilize the mutual relationship of channels to generate a channel attention map, Y fk2 then utilize the spatial relationship of features to generate a spatial attention map;

[0149] Embed global information using global average pooling and generate channel weights s:

[0150]

[0151] where, F gp (Y fk1 ) is the average value of all positions of the feature map Y fk1 in the spatial dimension;

[0152] The channel attention mechanism Y′ fk1 is:

[0153] Y′ fk1 = Sigmoid(W1s + b1)·Y fk1

[0154] where, W1 is the weight matrix of channel attention; b1 is the bias term; Sigmoid() is the activation function for generating normalized weights;

[0155] The spatial attention mechanism Y′ fk2 is:

[0156] Y′ fk2 = Sigmoid(W2·GN(Y fk2 ) + b2)·Y fk2

[0157] where, W2 is the weight matrix of spatial attention; b2 is the bias term; GN(Y fk2 ) is used to normalize each group of channels of Y fk2 to stabilize the training process; Sigmoid() is the activation function of spatial attention;

[0158] where, W1, b1, W2, b2 ∈ R C / 2G×1×1 ;

[0159] Then connect the two branches to make the number of channels consistent with the input channel number, that is:

[0160] Y′ fk = [Y′ fk1 , Y′fk2

[0161] Finally, cross-group information transfer along the channel dimension between different sub-features is aggregated through the "channel shuffle" operator to obtain the feature map Y after multi-scale feature fusion output ;

[0162] Y output = channel shuffle([Y′ f1 , Y′ f2 ,…Y′ fG )

[0163] Y′ f1 , Y′ f2 ,..., Y′ fG represents each branch Y′ f aggregated into a large feature map. The channel shuffle operation helps the network share information between multiple scales and multiple branches by rearranging and mixing channel information between different groups, thus achieving a richer feature representation

[0164] Constructing a multi-branch cascaded up-attention decoder includes at least the following steps

[0165] In the MobileNetV2 backbone network, the original feature map is sequentially decomposed into four feature maps features with different scales of 1 / 2, 1 / 4, 1 / 8, and 1 / 16 i , i = 1 / 2, 1 / 4, 1 / 8, 1 / 16. Both the 1 / 8 feature map and the 1 / 16 feature map contain rich and diverse semantic features and global features

[0166] The 1 / 16 feature map containing the richest semantic features is sent to the edge-guided atrous spatial convolutional pooling pyramid to extract features, and then its output is sent to the hybrid gating attention mechanism for dynamic multi-scale edge feature extraction to obtain the high-level feature map fea′ 1 / 16 , see the following formula

[0167] fea′ 1 / 16 = h(m(features 1 / 16 ))

[0168] fea′ 1 / 16 is upsampled by 2 times and concatenated with features 1 / 8 , and the hybrid gating attention operation is applied to dynamically integrate spatial, channel, and context feature information. See the following formula

[0169] fea′ 1 / 8 = h(Concat(features 1 / 8 , Up2(fea′​1 / 16 )))

[0170] Among them, m is the edge-guided atrous spatial convolutional pooling pyramid operation, and h is the hybrid gated attention operation;

[0171] features 1 / 2 、features 1 / 4 are shallow feature maps, which contain features such as color and texture, but have low semanticity and contain more noise. Directly performing progressive upsampling on them saves computing resources;

[0172] fea′ 1 / 8 Upsample by a factor of 2 and concatenate and fuse with features 1 / 4 Continue to upsample by a factor of 2 and concatenate with features 1 / 2 to obtain the multi-scale cascaded feature enhanced feature map fea mcfe , see the following formula:

[0173] fea mcfe = Concat(Up2(Concat(Up2(fea′ 1 / 8 ), features 1 / 4 )))

[0174] fea mcfe is an enhanced map of multi-scale features, which fuses information of different scales, including low-level features and high-level semantic information.

[0175] Specifically in implementation, verify the lightweight and high-precision semantic segmentation method of the edge-guided atrous spatial convolutional pooling pyramid combined with the multi-branch cascaded attention upsampling decoder disclosed in the present invention. Compare and verify the segmentation accuracy and the number of model parameters on the open-source dataset Pascal VOC2012. Table 1 shows the comparison results of the segmentation accuracy and the number of model parameters of the method of the present invention and eight other advanced methods under different attributes.

[0176] It can be analyzed from Table 1 that the lightweight and high-precision semantic segmentation method of the edge-guided atrous spatial convolutional pooling pyramid combined with the multi-branch cascaded attention upsampling decoder described in this paper has an MIoU value of 77.57% for the segmentation accuracy on the open-source dataset Pascal VOC2012, and the number of parameters is only 8.034M. Compared with the original DeepLabv3+ semantic segmentation network with Xception as the feature extraction backbone network, the MIoU is increased by 0.98%, but the number of parameters is only 6.81 times that of the original network, and it has strong competitiveness compared with other advanced semantic segmentation methods.

[0177] Table 1 Comparison results of each advanced model on the PASCAL VOC 2012 validation set

[0178]

[0179]

[0180] Embodiment 2:

[0181] See also Figure 2 The present invention also provides a lightweight and high-precision semantic segmentation system, comprising:

[0182] A data acquisition module, used to acquire and pre-process original data;

[0183] Feature extraction module, used to obtain high-level semantic features and low-level semantic features through the backbone network;

[0184] Feature enhancement module, used for fine-grained edge detail feature enhancement of high-level semantic features;

[0185] Feature fusion module, used to dynamically enhance the network's ability to globally interact with multi-scale feature information;

[0186] Feature cascade upsampling module for progressive and interactive restoration of feature maps;

[0187] The semantic segmentation prediction module predicts the pixel-by-pixel category of the input image through the model, maps the category label through a pseudo-color mask, and superimposes it on the original image to generate a pixel-level semantic annotation visualization result.

[0188] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A lightweight and high-precision semantic segmentation method, characterized in that: At least include the following steps: S1: Construct a lightweight feature extraction network, namely MobileNetv2. Use the lightweight feature extraction network MobileNetv2 as the backbone network to replace the original backbone network Xception. While deeply extracting features, significantly reduce the number of parameters and improve the deployment ability on mobile devices; S2: Propose an edge-guided atrous spatial pyramid. Fuse the extracted multi-scale edge information with the deep feature branch passing through the backbone network to strengthen edge perception; S3: Design a hybrid gating attention mechanism to dynamically integrate complementary spatial, channel, and context important features; S4: Construct a multi-branch cascaded up-attention decoder. Introduce the hybrid gating attention mechanism into the deep feature branch to efficiently and accurately gradually upsample and restore the feature map.

2. The lightweight high-precision semantic segmentation method according to claim 1, characterized in that: The lightweight feature extraction network is used to extract four-layer deep features and shallow features to form multi-scale capture of global features; The application of the lightweight feature extraction network at least includes the following steps: Perform data augmentation operations on the input RGB image to expand sample diversity and improve model generalization. The data augmentation operations include but are not limited to rotation, translation, photometric transformation, and Gaussian model processing. Crop a template image with a size of 512×512×3 on the template frame from the data-augmented RGB image as the input of the network; Extract the 1 / 2, 1 / 4, 1 / 8, and 1 / 16 downsampled feature maps of MobileNetv2 respectively, denoted as features 1 / 2 , features 1 / 4 , features 1 / 8 , features 1 / 16 .

3. A lightweight high-precision semantic segmentation method according to claim 1, characterized in that: The application process of the edge-guided atrous spatial pyramid at least includes the following steps: First, generate the feature map. Based on ASPP, add a dilated convolution with a dilation rate of 3 to enhance the perception ability of small object detail features. Through 4 convolution operations with different dilation rates, obtain receptive fields of different scales, and introduce an edge enhancement module to expand the network, retain more edge and texture features, and enhance the multi-scale edge information perception ability; Next, edge enhancement is performed, and the outputs of the atrous convolution blocks with dilation rates of 3, 6, 12, and 18 in all 4 atrous spatial pyramid pooling with edge guidance are sent into the Edge Enhancers to highlight the object edges in the feature map; Among them is the average pooling operation performed on to extract global information and thus reduce the influence of the background; Then perform 1x1 convolution for edge enhancement, and the feature map after edge enhancement is fed into a 1x1 convolutional layer. The role of the 1x1 convolutional layer is to reduce the number of channels of the feature map, and then it is added to the original dilated convolutional feature map to strengthen the detailed information of each scale, obtaining edge-enhanced feature maps of different scales Refer to the following formula: Then, multi-scale splicing is performed to splice and fuse the edge enhancement feature maps of 4 different scales to obtain the multi-scale edge feature F mef , as shown in the following formula: F mef It includes both fine-grained detailed information and multi-scale edge information; Finally, fuse the global feature and the edge feature. Refer to the following formula: Concatenate F mef with the global average pooling global_feature along the channel dimension, and use a 1×1 convolutional layer to reduce the dimension of the fused feature to obtain the output feature F ma_out .

4. A lightweight high-precision semantic segmentation method according to claim 1, characterized in that: The hybrid gating attention mechanism in S3 enables the model to dynamically adjust the feature combination according to the importance of different attention mechanisms through the operation of splitting convolution and multiplying gating weights, thereby flexibly adjusting the weights of each feature, enhancing the expression ability of the model, and improving the decision-making accuracy of global feature extraction. The hybrid gating attention mechanism specifically includes the following steps: First, enhance the feature context information through CoT, and at the same time strengthen the channel information by ResNeSt; after the feature map Fea is input into the CoT block, first perform k×k grouped convolution on the input key K through a 3×3 convolution to obtain the local context information representation K with local context information, obtain the static context representation of the input, and learn the dynamic multi-head attention matrix through two consecutive 1×1 convolutions. Refer to the following formula: A = [K 1 , Q]W θ W δ Among them, W θ has an activation function, and W δ does not have an activation function; This attention matrix A is learned based on the query feature Q and the contextualized static key feature K 1 That is, A is the local attention matrix for each spatial position learned based on the query feature and the contextualized key feature. This way enhances self-attention learning with the additional guidance of the mined static context K 1 ; The learned attention matrix is multiplied by the input value to achieve a dynamic context representation K of the input 2 , as shown in the following formula: K 2 = V * A where V is the values value; Fuse the static and dynamic context representations as the output Y1; Y1 = K 1 + K 2 The ResNeSt branch splits the input feature map Fea into k groups, each group is denoted as Cardinal, and then each group Cardinal is split into R Split blocks, with a total of G = KR groups, increasing the network width; Then, a 3×3 global average pooling across spatial dimensions is used to collect global context information with embedded channel statistics. The calculation formula for the c-th component is as follows: Among them represents the value of c channels at the spatial position (i, j), where H and W are the height and width of the input image respectively; The weighted fusion represented by the cardinality array is performed using channel-wise soft attention aggregation. Each feature map channel is generated using a weighted combination after splitting. The calculation formula for the c-th component is as follows: Among them, U R(k-1)+i represents the intermediate features generated by independent convolution operations on the R branch features within the k-th cardinality group, is the assignment weight of the soft attention, which is expressed as: Among them, represents s according to the global context k Determine the weights of each segmentation of the c-th channel; Finally, the cardinality array representations are linearly concatenated along the channel dimension, and the output is: Y2 = V + X For dynamic gated feature fusion, the output Y1 of the CoT block and the output Y2 of ResNeSt are concatenated together by multiplying with gated weights to achieve dynamic fusion of features across multiple channels and multiple levels. Y1 and Y2 are concatenated along the channel dimension and activated by the non-linear function Sigmoid to obtain the gated weight w: w = Sigmoid(Y1 + Y2) By linearly concatenating the results of multiplying Y1 and Y2 by their corresponding gating weights w, a feature map Y that fuses different scales is obtained. feature : Y feature = Y1 × w + Y2 × (1 - w) Finally, replace the attention and input the gated fusion feature map Y feature into the replaced attention to suppress the propagation of irrelevant information and enhance the representation of important features; Divide the input image Y feature into G groups, Y feature = [Y f1 , Y f2 , … Y fG , and simultaneously construct channel attention and spatial attention for each SA unit in each group; Y fk Form two branches along the channel dimension, namely Y fk1 And Y fk2 , where Y fk1 Utilize the mutual relationship of channels to generate a channel attention map, Y fk2 Then utilize the spatial relationship of features to generate a spatial attention map; Global information is embedded using global average pooling to generate the channel weight s: Among them, F gp (Y fk1 ) is the average value of all positions of the computational feature map Y fk1 in the spatial dimension; Channel attention mechanism Y′ fk1 is as follows: Y′ fk1 = Sigmoid(W1s + b1) · Y fk1 where W1 is the weight matrix of channel attention; b1 is the bias term; Sigmoid() is the activation function used to generate the normalized weight; Spatial attention mechanism Y′ fk2 is as follows: Y′ fk2 = Sigmoid(W2·GN(Y fk2 ) + b2)·Y fk2 Among them, W2 is the weight matrix of spatial attention; b2 is the bias term; GN(Y fk2 ) is used to normalize each group of channels of Y fk2 to stabilize the training process; Sigmoid() is the activation function of spatial attention; where W1, b1, W2, b2 ∈ R C / 2G×1×1 ; Then the two branches are connected to make the number of channels consistent with the input channel number, that is: Y′ fk = [Y′ fk1 , Y′ fk2 ​ Finally, cross-group information transfer along the channel dimension between different sub-features is aggregated through the "channel shuffle" operator to obtain the feature map Y after multi-scale feature fusion output ; Y output = channel shuffle([Y′ f1 , Y′ f2 , … Y′ fG ) Y′ f1 , Y′ f2 ,..., Y′ fG represent each branch Y′ f are aggregated into a large feature map. The channel shuffle operation helps the network share information between multiple scales and multiple branches by rearranging and mixing the channel information between different groups, thereby achieving richer feature expressions.

5. A lightweight high-precision semantic segmentation method according to claim 4, characterized in that: The construction of the multi-branch cascaded up-attention decoder at least includes the following steps: In the MobileNetV2 backbone network, the original feature map is successively decomposed into four feature maps features with different scales of 1 / 2, 1 / 4, 1 / 8, and 1 / 16 i , where i = 1 / 2, 1 / 4, 1 / 8, 1 / 16. Both the 1 / 8 feature map and the 1 / 16 feature map contain rich and diverse semantic features and global features; Send the 1 / 16 feature map containing the richest semantic features into the edge-guided atrous spatial pyramid pooling to extract features, and then send its output into the hybrid gated attention mechanism for dynamic multi-scale edge feature extraction to obtain the high-level feature map fea′ 1 / 16 , see the following formula: fea′ 1 / 16 = h(m(features 1 / 16 )) fea′ 1 / 16 Upsample by 2 times and features 1 / 8 Concatenate, apply the hybrid gated attention operation, and dynamically integrate spatial, channel, and context feature information, as shown in the following formula: fea′ 1 / 8 = h(Concat(features 1 / 8 , Up2(fea′ 1 / 16 ))) where m is the edge-guided atrous spatial convolution pooling pyramid operation, and h is the hybrid gated attention operation; features 1 / 2 、features 1 / 4 It is a shallow feature map, which contains features such as color and texture, but has low semantics and contains more noise. It is directly upsampled step by step to save computing resources; fea′ 1 / 8 Upsample by 2 times and features 1 / 4 Perform splicing and fusion, continue to upsample by 2 times, and features 1 / 2 Obtain the multi-scale cascaded feature enhanced feature map fea mcfe , see the following formula: fea mcfe = Concat(Up2(Concat(Up2(fea′ 1 / 8 ), features 1 / 4 ))) fea mcfe It is an enhanced map of multi-scale features, which integrates information at different scales, including low-level features and high-level semantic information.

6. A lightweight high-precision semantic segmentation system for the lightweight high-precision semantic segmentation method according to any one of the above claims 1-5, characterized in that: It at least includes a data acquisition module, a feature extraction module, a feature enhancement module, a feature fusion module, a feature cascaded upsampling module, and a semantic segmentation prediction module; The data acquisition module is used to acquire and preprocess the original data; The feature extraction module is used to obtain high-level semantic features and low-level semantic features through the backbone network; The feature enhancement module is used for fine-grained edge detail feature enhancement of high-level semantic features; The feature fusion module is used to dynamically enhance the ability of the network to interact globally and multi-scale feature information; The feature cascaded upsampling module is used to progressively interact and restore the feature map; The semantic segmentation prediction module is used to map the class labels through a pseudo-color mask after the input image is predicted pixel-by-pixel by the model, and superimpose it on the original image to generate a visual result of pixel-level semantic annotation.

Images