A real-time semantic segmentation method based on multi-feature reuse

Abstract

The present application relates to the field of image semantic segmentation, and particularly relates to a real-time semantic segmentation method based on multi-feature reuse, comprising: inputting a target image into a trained semantic segmentation model to output a corresponding semantic segmentation image; comprising: extracting initial feature information of the target image to obtain a primary feature map; extracting feature information of the primary feature map to obtain a local feature map; extracting feature information of the local feature map to obtain a global feature map; fusing the global feature map and the local feature map to obtain an intermediate fusion feature map; fusing the intermediate fusion feature map and the primary feature map to obtain a final fusion feature map; and performing classification prediction based on the final fusion feature map by a classifier to output the corresponding semantic segmentation image. The present application can effectively extract feature information of different levels through a lightweight network, simultaneously realize reuse of low-level features and sufficient fusion of features of different levels, and strengthen effective complementation of spatial information and context information.

Description

Technical Field

[0001] This invention relates to the field of image semantic segmentation, and specifically to a real-time semantic segmentation method based on multi-feature reuse. Background Technology

[0002] Semantic segmentation is a computer vision technique that enables image understanding at the pixel level. In recent years, with the widespread application of convolutional neural networks in tasks such as pattern recognition and scene parsing, semantic segmentation technology has developed rapidly. Simultaneously, with the widespread application of semantic segmentation in scenarios such as autonomous driving, medical imaging, and satellite remote sensing, higher demands are being placed on the size of segmentation models, segmentation accuracy, and real-time performance.

[0003] Currently, deep learning-based semantic segmentation network models generally suffer from drawbacks such as large parameter count and high computational cost, making them unsuitable for resource-constrained scenarios with high real-time requirements. Therefore, the application of semantic segmentation technology in real-time scenarios such as autonomous driving and augmented reality faces new challenges. Existing real-time semantic segmentation methods either fail to meet real-time requirements while maintaining sufficient segmentation accuracy, or prioritize real-time inference speed at the expense of segmentation accuracy. Therefore, constructing a lightweight real-time semantic segmentation model that effectively balances segmentation accuracy and inference speed has become one of the current research hotspots in this field.

[0004] With the increasing demand for mobile device deployment, many lightweight real-time semantic segmentation models based on convolutional neural networks have been proposed. According to the network structure, they can be roughly divided into three categories: (1) encoder-decoder structure, in which the encoder part is mainly used to extract image features and the decoder part is mainly used to sequentially restore image details. However, in the encoding process, frequent downsampling will lead to the loss of scene detail information and cannot be fully restored by upsampling, which affects the accuracy of semantic segmentation. At the same time, the complex decoder structure reduces the forward inference speed of the algorithm. (2) multi-branch structure, which uses different branches to extract image feature information and then fuses them. By utilizing the feature expression capabilities of different branches, information interaction can be effectively realized, thereby improving the accuracy of semantic segmentation. This structure still lacks interaction between branches and introduces a large number of parameters, and parallel operation will bring additional computational overhead. (3) feature reuse structure, which can combine high-level and low-level features to enhance the network learning ability, realize feature complementarity, and thus improve the segmentation effect. However, the current network models based on feature reuse structure still have certain limitations. The extracted semantic feature information is mostly local information and fails to make full use of global context information. Therefore, it is necessary to design more effective feature reuse structure network models to extract rich contextual information and key features, and enhance the feature reusability at different levels, so as to effectively balance the accuracy and efficiency of semantic segmentation.

[0005] In summary, achieving an effective balance between semantic segmentation accuracy and real-time performance, and constructing a more effective deep learning-based real-time semantic segmentation network model, has become a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] To address the shortcomings of the existing technologies, the technical problem to be solved by this invention is: how to provide a real-time semantic segmentation method based on multi-feature reuse, which can effectively extract feature information at different levels through a lightweight network, while realizing the reuse of low-level features and the full fusion of features at different levels, and strengthening the effective complementarity of spatial information and contextual information, thereby improving the accuracy and real-time performance of real-time semantic segmentation.

[0007] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0008] A real-time semantic segmentation method based on multi-feature reuse includes:

[0009] S1: Obtain the target image to be segmented;

[0010] S2: Input the target image into the trained semantic segmentation model and output the corresponding semantic segmentation image;

[0011] The semantic segmentation model includes the following processing steps:

[0012] S201: Extract the initial feature information of the target image to obtain the primary feature map;

[0013] S202: Based on the primary feature map, features are further extracted by a first dense residual module composed of multiple stacked asymmetric residual attention modules to obtain a local feature map;

[0014] S203: Based on the local feature map, features are further extracted by a second dense residual module composed of multiple stacked asymmetric residual attention modules to obtain the global feature map;

[0015] S204: The global feature map and the local feature map are fused through the feature aggregation module to obtain an intermediate fused feature map;

[0016] S205: The intermediate fused feature map and the primary feature map are fused by the enhanced feature fusion module to obtain the final fused feature map;

[0017] S206: The classifier performs classification prediction based on the final fused feature map and outputs the corresponding semantic segmentation image;

[0018] S3: Use the semantic segmentation image output by the semantic segmentation model as the real-time semantic segmentation result of the target image to be segmented.

[0019] Preferably, the initial feature information of the target image is extracted through three consecutively stacked convolutional layers to obtain a primary feature map.

[0020] Preferably, the first dense residual module consists of three consecutively stacked asymmetric residual attention modules, each with a void ratio of 2; the output of the last asymmetric residual attention module in the first dense residual module is a local feature map.

[0021] Preferably, the second dense residual module consists of six consecutively stacked asymmetric residual attention modules with void ratios of 4, 4, 8, 8, 16, and 16, respectively; the output of the last asymmetric residual attention module in the second dense residual module is a global feature map.

[0022] Preferably, each asymmetric residual attention module extracts feature information from the input feature map through the following steps:

[0023] 1) Perform channel dimensionality reduction on the input feature map using a 3×3 convolution operation to obtain a dimensionality-reduced feature map;

[0024] 2) Perform 3×1 and 1×3 depthwise convolutions and 3×1 and 1×3 depthwise dilation convolutions on the dimensionality-reduced feature maps in sequence to obtain depthwise convolution feature maps and depthwise dilation convolution feature maps, respectively;

[0025] 3) Input the depthwise convolutional feature map into the channel attention submodule for feature extraction to obtain the channel attention feature map;

[0026] 4) Concatenate the depthwise dilated convolutional feature map and the channel attention feature map through channels, and obtain the fused feature map through a 1×1 convolution operation;

[0027] 5) Perform residual connection between the fused feature map and the input original feature map using a summation operation to obtain the final feature map.

[0028] Preferably, in the channel attention submodule, the input feature map is first processed through a global pooling layer, a fully connected layer, a ReLU function layer, a fully connected layer, and a Sigmoid function layer connected end to end to obtain the channel feature map; then the channel feature map is multiplied with the input feature map to obtain the corresponding channel attention feature map.

[0029] Preferably, the feature aggregation module includes the following processing steps:

[0030] S2041: Use the Add operation to add the global feature map X1 and the local feature map X2 to obtain the added feature map X;

[0031] S2042: Using two one-dimensional average pooling operations (H, 1) and (1, W) to encode spatial information of the additive feature map X along the vertical and horizontal directions respectively, aggregated feature maps in both horizontal and vertical spatial directions are generated. and

[0032] S2043: Merge aggregated feature maps using join operations and Then, after 1×1 convolution, batch normalization, and nonlinear activation function operations, the intermediate aggregated feature map F is output. z ;

[0033] The formula is described as follows:

[0034]

[0035] In the formula: [] represents a join operation; {} represents batch normalization; δ represents a nonlinear activation function; where F z ∈C / r×1×(H+W), where r represents the reduction rate; C represents the number of channels in the feature map; and H and W represent the height and width of the feature map, respectively.

[0036] S2044: Aggregate the intermediate feature map F z Decomposed into two tensors F in the height and width directions h and F w F is recovered by using 1×1 convolution operation respectively. h and F w The number of channels is the same as the summed feature map X, and then two attention weights g are obtained by the Sigmoid function operation. h and g w ;

[0037] The formula is described as follows:

[0038] g h =σ(Conv h 1×1 (F h ));

[0039] g w =σ(Conv w 1×1 (F w ));

[0040] In the formula: σ represents the Sigmoid activation function; where F h ∈C / r×H×1,F w ∈C / r×1×W;

[0041] S2045: Combine the summation feature map X with the attention weight g h and g w Multiplying them together yields the intermediate fused feature map Y;

[0042] The formula is described as follows:

[0043] Y = X × g h ×g w .

[0044] Preferably, the enhanced feature fusion module includes the following processing steps:

[0045] S2051: Connect the intermediate fused feature map Y and the primary feature map through a connection operation, and perform batch normalization and ReLU activation function operation to obtain the initial feature map F;

[0046] S2052: The initial feature map F is subjected to average pooling and max pooling operations in two branches, followed by 1×1 convolution, ReLU activation function and Sigmoid function operations in sequence, to obtain the feature weights F obtained from the corresponding branches of average pooling and max pooling. avg and F max ;

[0047] The formula is described as follows:

[0048] F avg = f[AvgPool(F)];

[0049] F mcx = f[MaxPool(F)];

[0050] In the formula: f represents the sequential execution of 1×1 convolution, ReLU activation function and Sigmoid function operations; AvgPool represents the average pooling operation; MaxPool represents the max pooling operation;

[0051] S2053: Adjust the feature weights F avg and F max After addition, multiply with the initial feature map F to reweight the features, resulting in the final channel-weighted fused feature map F. c ;

[0052] The formula is described as follows:

[0053] F c =(F avg +F max )×F+F.

[0054] Preferably, before extracting the feature information from the primary feature map, the primary feature map is first downsampled.

[0055] Before extracting feature information from the local feature map, the local feature map is first downsampled.

[0056] Before fusing the global and local feature maps, the global feature map is first upsampled and then convolved with a 3×3 convolution.

[0057] Preferably, the number of channels in the final fused feature map is first compressed into the number of categories through convolution operations, then the final fused feature map is upsampled to the same size as the target image, and finally the predicted semantic segmentation image is output.

[0058] Compared with existing technologies, the real-time semantic segmentation method based on multi-feature reuse in this invention has the following advantages:

[0059] This invention inputs a target image into a trained semantic segmentation model to generate a semantically segmented image. On one hand, in the feature extraction stage, the semantic segmentation model of this invention extracts a primary feature map containing spatial information, a local feature map containing contextual information and local features, and a global feature map containing semantic information and global features through dense residual modules stacked with asymmetric residual attention modules (ARAM). Here, ARAM is a lightweight encoder with advantages of small parameter count and low computational cost, and it can effectively extract feature information at different levels. That is, it can effectively extract feature information at different levels through a lightweight network, thereby improving the real-time performance of semantic segmentation. On the other hand, in the feature fusion stage, the semantic segmentation model of this invention introduces a feature aggregation module and an enhanced feature fusion module to fuse the global feature map and local feature map and then further fuse them with the primary feature map. This realizes the reuse of low-level features and the full fusion of features at different levels, and strengthens the effective complementarity of spatial information and contextual information. In turn, it generates a more discriminative high-resolution high-level mapping map (i.e., the final fused feature map), which improves the problems of insufficient balance between accuracy and real-time performance and insufficient interaction and fusion of feature information at different levels in existing semantic segmentation methods in real-time scenarios, thereby improving the accuracy of real-time semantic segmentation.

[0060] To address the problem that existing real-time semantic segmentation methods lack interaction between different feature information, resulting in inefficient extraction of features at different levels, this invention utilizes first and second dense residual modules stacked with ARAM to extract local feature maps containing contextual information and key features, and global feature maps containing semantic information and global feature information, respectively. ARAM effectively combines depthwise separable convolution and dilated convolution, and introduces an attention mechanism to refine and enhance the expression of effective features. This not only reduces the number of parameters but also facilitates the extraction of rich semantic contextual information and key feature information, thereby improving the extraction efficiency of features at different levels.

[0061] To address the problem that existing real-time semantic segmentation methods fail to consider the differences and diversity of features at different stages when fusing features, resulting in inefficient fusion methods and insufficient integration of contextual and spatial information, thus affecting semantic segmentation accuracy, this invention proposes an Enhanced Feature Fusion Module (EFFM) based on the introduction of a Feature Aggregation Module (FAM). By combining FAM and EFFM, features at different high and low levels are fused from top to bottom, thereby better enabling the reuse of low-level features and the full fusion of features at different levels, and better strengthening the effective complementarity of spatial and contextual information. Attached Figure Description

[0062] To make the objectives, technical solutions, and advantages of the invention clearer, the invention will now be described in further detail with reference to the accompanying drawings, wherein:

[0063] Figure 1 This is a logical block diagram of a real-time semantic segmentation method based on multi-feature reuse.

[0064] Figure 2 This is a network structure diagram of a semantic segmentation model;

[0065] Figure 3 This is a network structure diagram of the asymmetric residual attention module;

[0066] Figure 4 This is a network structure diagram of the feature aggregation module;

[0067] Figure 5 To enhance the network structure diagram of the feature fusion module;

[0068] Figure 6 Visual comparison of different algorithms on the Cityscapes validation set;

[0069] Figure 7 A visual comparison of different algorithms on the CamVid validation set. Detailed Implementation

[0070] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but only to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0071] It should be noted that similar reference numerals and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the figures, or the orientation or positional relationship commonly used when the product is in use. They are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third," etc., are only used to distinguish descriptions and should not be construed as indicating or implying relative importance. In addition, the terms "horizontal," "vertical," etc., do not mean that the component is required to be absolutely horizontal or suspended, but can be slightly tilted. For example, "horizontal" only means that its direction is more horizontal than "vertical," and does not mean that the structure must be completely horizontal, but can be slightly tilted. In the description of this invention, it should also be noted that, unless otherwise explicitly specified and limited, the terms "set," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0072] The following detailed explanation illustrates the specific implementation methods:

[0073] Example:

[0074] This embodiment discloses a real-time semantic segmentation method based on multi-feature reuse.

[0075] like Figure 1 As shown, the real-time semantic segmentation method based on multi-feature reuse includes:

[0076] S1: Obtain the target image to be segmented;

[0077] S2: Input the target image into the trained semantic segmentation model and output the corresponding semantic segmentation image;

[0078] Combination Figure 2 As shown, the semantic segmentation model includes the following processing steps:

[0079] S201: Extract the initial feature information of the target image to obtain the primary feature map;

[0080] In this embodiment, initial feature information of the target image is extracted through three consecutively stacked convolutional layers to obtain a primary feature map. The purpose of designing convolutional layers is to extract feature information, and the number of stacked convolutional layers is determined according to the designer's needs.

[0081] S202: Based on the primary feature map, features are further extracted by the first dense residual module (DRM1), which is composed of multiple stacked asymmetric residual attention modules (ARAM), to obtain a local feature map;

[0082] S203: Based on the local feature map, features are further extracted by a second dense residual module (DRM2) composed of multiple stacked asymmetric residual attention modules (ARAM) to obtain the global feature map;

[0083] S204: The global feature map and local feature map are fused through the Feature Aggregation Module (FAM) to obtain an intermediate fused feature map;

[0084] S205: The intermediate fused feature map and the primary feature map (by performing a 3x3 convolution operation on the primary feature map) are fused through the Enhance Feature Fusion Module (EFFM) to obtain the final fused feature map;

[0085] S206: The classifier performs classification prediction based on the final fused feature map and outputs the corresponding semantic segmentation image;

[0086] In this embodiment, the classifier structure is a 1x1 convolutional layer, the purpose of which is to compress the number of channels into the number of categories. Specifically: first, the number of channels in the final fused feature map is compressed into the number of categories through convolution operations; then, the final fused feature map is upsampled (four times upsampled) to the same size as the target image; finally, the predicted semantic segmentation image is output.

[0087] S3: Use the semantic segmentation image output by the semantic segmentation model as the real-time semantic segmentation result of the target image to be segmented.

[0088] This invention inputs a target image into a trained semantic segmentation model to generate a semantically segmented image. On one hand, in the feature extraction stage, the semantic segmentation model of this invention extracts a primary feature map containing spatial information, a local feature map containing contextual information and local features, and a global feature map containing semantic information and global features through a dense residual module stacked with ARAM. ARAM is a lightweight encoder with advantages of small parameter count and low computational cost, and it can effectively extract feature information at different levels, thus improving the real-time performance of semantic segmentation. On the other hand, in the feature fusion stage, the semantic segmentation model of this invention introduces a feature aggregation module and an enhanced feature fusion module to fuse the global and local feature maps and then further fuse them with the primary feature map. This achieves the reuse of low-level features and the full fusion of features at different levels, and strengthens the effective complementarity of spatial and contextual information, thereby generating a more discriminative high-resolution high-level mapping map (i.e., the final fused feature map). This improves the problems of insufficient balance between accuracy and real-time performance in existing semantic segmentation methods in real-time scenarios, as well as insufficient interaction and fusion of feature information at different levels, thereby improving the accuracy of real-time semantic segmentation.

[0089] To address the problem that existing real-time semantic segmentation methods lack interaction between different feature information, resulting in inefficient extraction of features at different levels, this invention utilizes first and second dense residual modules stacked with ARAM to extract local feature maps containing contextual information and key features, and global feature maps containing semantic information and global feature information, respectively. ARAM effectively combines depthwise separable convolution and dilated convolution, and introduces an attention mechanism to refine and enhance the expression of effective features. This not only reduces the number of parameters but also facilitates the extraction of rich semantic contextual information and key feature information, thereby improving the extraction efficiency of features at different levels.

[0090] To address the problem that existing real-time semantic segmentation methods fail to consider the differences and diversity of features at different stages when fusing features, resulting in inefficient fusion methods and insufficient integration of contextual and spatial information, thus affecting semantic segmentation accuracy, this invention proposes EFFM based on the introduction of FAM. By combining FAM and EFFM, it achieves top-down fusion of features at different high and low levels, thereby better enabling the reuse of low-level features and the full fusion of features at different levels, and better strengthening the effective complementarity of spatial and contextual information.

[0091] In specific implementation, the first dense residual module and the second dense residual module are constructed by consecutively stacking ARAMs to densely extract features. Specifically:

[0092] The first dense residual module consists of three consecutively stacked ARAMs, each with a hole ratio of 2, used to extract feature information from the primary feature map; the output of the last ARAM in the first dense residual module is the local feature map. The second dense residual module consists of six consecutively stacked ARAMs with hole ratios of {4, 4, 8, 8, 16, 16}, used to extract feature information from the local feature map; the output of the last ARAM in the second dense residual module is the global feature map.

[0093] This invention utilizes first and second dense residual modules stacked with ARAM to extract contextual information, key features, local feature maps, and semantic and global feature maps, respectively. ARAM effectively combines depthwise separable convolution and dilated convolution, and introduces an attention mechanism to refine and enhance the expression of effective features. This not only results in a small parameter count but also facilitates the extraction of rich semantic contextual and key feature information, thereby improving the feature extraction effect at different levels. Furthermore, the network structure design of the first and second dense residual modules in this invention not only facilitates the extraction of surrounding or global features and enhances the contextual relationships between pixels but also allows for obtaining similar contextual information in shallow layers to deep networks with fewer parameters. Moreover, it enables dense feature extraction on large and dense feature maps, reducing the loss of significant spatial information and thus better preserving most of the image's edge features.

[0094] Currently, commonly used residual feature extraction modules include the Bottleneck module, the one-dimensional non-bottleneck-1D module, and the depth-wise asymmetric bottleneck (DAB) module. To reduce the number of parameters and facilitate network deepening, the Bottleneck module uses a dimensionality reduction followed by dimensionality increase approach to lightweight the network. However, stacking a large number of convolutional layers may cause network degradation. The Non-bottleneck-1D module improves upon the traditional residual structure by splitting the standard 3×3 two-dimensional convolution into two one-dimensional 3×1 and 1×3 spatial decomposition convolutions, significantly reducing the number of parameters. However, this structure uses fixed-size convolutional kernels, limiting the receptive field and hindering network performance improvement. DABNet uses factorized depth-dilated separable convolutions to construct the DAB module, achieving target feature extraction with fewer parameters and a larger receptive field. However, this module ignores the contextual relationships between pixels and lacks information interaction between branches, leading to classification errors.

[0095] Inspired by the residual modules described above, this invention designs a highly efficient asymmetric residual attention module (ARAM). This module is repeatedly stacked during the feature extraction stage, which can fully capture the contextual information of each layer. Its structure is as follows: Figure 3 As shown. Specifically:

[0096] Each ARAM is used to extract more effective and deeper feature information from the input feature map;

[0097] ARAM extracts feature information from the input feature map through the following steps:

[0098] 1) Perform channel dimensionality reduction on the input feature map using a 3×3 convolution operation to obtain a dimensionality-reduced feature map;

[0099] 2) Perform 3×1 and 1×3 depthwise convolutions and 3×1 and 1×3 depthwise dilation convolutions on the dimensionality-reduced feature maps in sequence to obtain depthwise convolution feature maps and depthwise dilation convolution feature maps, respectively;

[0100] This invention fully leverages the advantages of both types of convolution. Building upon depthwise separable convolution, it further splits the 3×3 depthwise convolution into a cascaded structure of 3×1 and 1×3, significantly reducing the number of parameters in the ARAM and improving network speed. Meanwhile, the introduced dilated convolution further expands the receptive field, helping to capture more contextual information. Combining these two approaches achieves better convolutional results, improving the model's efficiency and accuracy to a certain extent.

[0101] 3) Input the depthwise convolutional feature map into the channel attention submodule for feature extraction to obtain the channel attention feature map;

[0102] In this embodiment, the channel attention (Squeeze and Excitation, SE) submodule first processes the input feature map (such as a depthwise convolutional feature map) through a global pooling layer, a fully connected layer, a ReLU function layer, a fully connected layer, and a Sigmoid function layer connected end to end in sequence to obtain the channel feature map; then the channel feature map is multiplied with the input feature map to obtain the corresponding channel attention feature map.

[0103] The SE submodule of this invention uses global context information captured by global average pooling to recalibrate channels, enhances important channel features and suppresses useless channel features in the global view, and establishes dependencies between channel contexts, thereby strengthening the correlation between channel pixels.

[0104] 4) Concatenate the depthwise dilated convolutional feature map and the channel attention feature map through channels, and obtain the fused feature map through a 1×1 convolution operation;

[0105] 5) Perform residual connection between the fused feature map and the input original feature map using a summation operation to obtain the final feature map.

[0106] The ARAM of this invention first applies a convolution decomposition strategy to depthwise convolution, reducing the number of parameters and computational complexity. Then, it effectively combines depthwise separable convolution and dilated convolution to increase the receptive field without reducing feature map resolution. Next, it introduces a channel attention mechanism to redistribute weights, effectively highlighting key channel features. Therefore, ARAM has fewer parameters and stronger feature extraction capabilities, making it more suitable for lightweight networks with fewer layers that require high segmentation accuracy. It satisfies real-time requirements while maximizing semantic segmentation accuracy, facilitating the extraction of rich semantic context and key feature information, thereby improving the effectiveness of feature extraction at different levels.

[0107] In the feature fusion part, the low-level features in the shallow stage of the network contain rich spatial and local feature information, while the global features in the deep stage contain rich semantic and global feature information. One of the reasons why many single-branch semantic segmentation networks have low accuracy is that they have not effectively integrated feature information at different levels.

[0108] To further optimize the process, this invention proposes to employ two efficient feature fusion modules based on attention mechanisms, namely the Feature Aggregation Module (FAM) and the Enhanced Feature Fusion Module (EFFM), to fuse the output features of adjacent stages in the encoder step by step, thereby obtaining higher-precision output features.

[0109] like Figure 4 As shown, FAM is used to fuse local and global features, and includes the following processing steps:

[0110] S2041: Use the Add operation to add the global feature map X1 and the local feature map X2 to obtain the added feature map X;

[0111] S2042: Using two one-dimensional average pooling operations (H,1) and (1,W) to encode spatial information of the additive feature map X along the vertical (height) and horizontal (width) directions respectively, aggregated feature maps in both the horizontal and vertical spatial directions are generated. and

[0112] S2043: Merge aggregated feature maps using the Concat operation. and Then, after 1×1 convolution, batch normalization, and nonlinear activation function operations, the intermediate aggregated feature map F is output. z ;

[0113] The formula is described as follows:

[0114]

[0115] In the formula: [] represents a join operation; {} represents batch normalization; δ represents a nonlinear activation function; where F z ∈C / r×1×(H+W), where r represents the reduction rate, and r (r=32) is used to reduce F. z The number of channels; C represents the number of channels in the feature map; H and W represent the height and width of the feature map, respectively;

[0116] S2044: Aggregate the intermediate feature map F z Decompose into two tensors F in the height (vertical) and width (horizontal) directions. h and F w F is recovered by using 1×1 convolution operation respectively. h and F w The number of channels is the same as the summed feature map X, and then two attention weights g are obtained by the Sigmoid function operation. h and g w ;

[0117] The formula is described as follows:

[0118] g h =σ(Conv h 1×1 (F h ));

[0119] g w =σ(Conv w 1×1 (F w ));

[0120] In the formula: σ represents the Sigmoid activation function; where F h ∈C / r×H×1,F w ∈C / r×1×W;

[0121] S2045: Combine the summation feature map X with the attention weight g h and g w Multiplying them together yields the intermediate fused feature map Y;

[0122] The formula is described as follows:

[0123] Y = X × g h ×g w .

[0124] The features fused by FAM (first-level) possess rich contextual information and close relationships between pixels, carrying abundant category information, which is beneficial for feature reconstruction. However, the fused feature map still lacks the spatial detail information necessary for feature recovery, making it impossible to recognize all pixel information. Therefore, lower-level spatial information is particularly important for outputting a more refined segmented image. To address this, this invention designs an Enhanced Feature Fusion Module (EFFM).

[0125] Combination Figure 5 As shown, EFFM includes the following processing steps:

[0126] S2051: The intermediate fused feature map Y and the primary feature map are connected by a concat operation, and batch normalization and ReLU activation function operation are performed to obtain the initial feature map F;

[0127] S2052: The initial feature map F is subjected to average pooling and max pooling operations in two branches, followed by 1×1 convolution, ReLU activation function and Sigmoid function operations in sequence, to obtain the feature weights F obtained from the corresponding branches of average pooling and max pooling. avg and F max ;

[0128] The formula is described as follows:

[0129] F avg = f[AvgPool(F)];

[0130] F max = f[MaxPool(F)];

[0131] In the formula: f represents the sequential execution of 1×1 convolution, ReLU activation function and Sigmoid function operations; AvgPool represents the average pooling operation; MaxPool represents the max pooling operation;

[0132] S2053: Adjust the feature weights F avg and F max After addition, multiply with the initial feature map F to reweight the features, resulting in the final channel-weighted fused feature map F. c ;

[0133] The formula is described as follows:

[0134] F c =(F avg +F max )×F+F.

[0135] Specifically: Before extracting the feature information from the primary feature map, the primary feature map is first downsampled.

[0136] Before extracting feature information from the local feature map, a downsampling operation is first performed on the local feature map.

[0137] Before fusing the global and local feature maps, the global feature map is first upsampled by four times and then convolved with a 3×3 convolution, so that the size of the global feature map is similar to that of the local feature map. Figure 1 To.

[0138] This invention reduces the dimensionality of primary and local feature maps by performing a 2x downsampling operation, filtering redundant information in the image, thereby improving the data processing efficiency of the model and further enhancing the real-time performance of semantic segmentation.

[0139] To better illustrate the advantages of the technical solution of the present invention, the following experiments are disclosed in this embodiment.

[0140] This experiment uses two commonly used datasets, Cityscapes and CamVid, to conduct experiments and analyze the results, verifying the effectiveness of the algorithm. Cityscapes is a large-scale urban street scene dataset widely used in semantic segmentation. The experiment uses 5000 finely annotated images for training, validation, and testing, with quantities of 2975, 500, and 1525 respectively, covering 19 categories. CamVid is a street scene dataset for driving cars, containing 11 categories and 701 finely annotated images. These images are divided into 367 training samples, 101 validation samples, and 233 test samples.

[0141] This experiment uses Mean Intersection over Union (MIoU) and Frames Per Second (FPS) as metrics for accuracy and inference speed. These two metrics are the main standard measures in the current field of real-time semantic segmentation.

[0142] The experimental environment was based on PyTorch 1.12.1 and Python 3.7, and the experiments were conducted on a single GTX 2080 GPU.

[0143] The experimental setup is as follows: The network is optimized using the mini-batch stochastic gradient descent optimizer (SGD) with a batch size of 6, a maximum training epoch of 1000, momentum and weight decay set to 0.9 and 1e-4 respectively, and a "poly" learning rate strategy is adopted. The adaptive learning rate adjustment after each iteration is shown in the formula:

[0144]

[0145] In the formula: lr represents the learning rate after each iteration; lr initThe initial learning rate is represented by `iter`, the current iteration index is represented by `max_iter`, the maximum number of iterations per epoch is represented by `power`, and the initial learning rate is set to 4.5e-2.

[0146] Regarding data augmentation, during training, random horizontal flipping, mean decay, and random cropping were applied to the input images, with random values ​​using numbers from {0.75, 1.0, 1.25, 1.5, 1.75, 2.0}. The Cityscapes dataset was randomly cropped to a resolution of 512×1024 for training.

[0147] To verify the performance of each module in the semantic segmentation model (hereinafter referred to as MFRNet) of this invention, MIoU and FPS were used as evaluation criteria to conduct comparative experiments on each module. The experimental results are shown in Tables 1, 2, and 3. This includes verification of the effectiveness of the asymmetric residual attention module and the feature fusion module. The feature extraction module was mainly constructed using three heterogeneous networks—MFRNet-DAB, MFRNet-LWNM, and MFRNet-ARM—built with DAB, LWNB, and ARM for comparison. The feature fusion module mainly used Add+Concat, FAM+FAM, EFFM+EFFM, and FAM+EFFM. Furthermore, since the feature extraction module of this invention uses dilated convolution, two additional sets of comparative experiments were added to further verify the impact of different dilation rates.

[0148] Table 1. Comparison of experimental results for different feature extraction methods

[0149]

[0150] To verify the information extraction capability of ARAM, this patent replaced ARAM with existing DAB, LWNB, and ARM to construct three heterogeneous networks: MFRNet-DAB, MFRNet-LWNM, and MFRNet-ARM for comparison. The results are shown in Table 1. LWNB is a lightweight non-bottleneck module (LWNM) formed by embedding a one-dimensional non-bottleneck structure with depthwise separable convolutions. ARM is an asymmetric residual module (ARM) formed by removing SE from ARAM. As shown in Table 1, the accuracy of the three comparison networks is 1-3 percentage points lower than MFRNet. The reason for this is that the ARAM constructed in this invention introduces an attention-based module (SE) to weight important channel features, suppressing interference from irrelevant noise and helping to extract key information useful for recognition. MFRNet's inference speed is only slightly lower than other methods, which does not affect real-time performance, and its overall performance is better. Therefore, the ARAM module has better feature extraction capability in a lightweight network, ensuring high accuracy in semantic segmentation while meeting real-time performance requirements.

[0151] Table 2 Comparative experimental results of the impact of different fusion methods on MFRNet performance

[0152]

[0153] Feature fusion has always been key to improving semantic segmentation accuracy. MFRNet implements two-level feature fusion, such as... Figure 4As shown in Table 2, the FAM module fuses features from the intermediate and high-level stages of the encoder, constituting the first-level fusion. The EFFM module fuses features from the initial stage of the encoder and features obtained from the first fusion, constituting the second-level fusion. To verify the effectiveness of the two-level feature fusion modules FAM and EFFM in improving the segmentation performance of MFRNet, four fusion methods were designed for comparison, as shown in Table 2: Add+Concat, FAM+FAM, EFFM+EFFM, and FAM+EFFM. Table 2 shows that the fusion method using the cross-combination of FAM and EFFM achieves the best segmentation accuracy, reaching 72.6%, and the second best inference speed, at 98.0 FPS. Both the segmentation accuracy and inference speed are higher than the method using only FAM / EFFM for two-level fusion. While Add+Concat improves the inference speed, it does not consider the differences and diversity between features at each stage, resulting in a decrease in segmentation accuracy of approximately 2.8% compared to the FAM+EFFM fusion method in this patent. This demonstrates that, compared to simple fusion methods and single-module combinations, combining FAM and EFFM can better aggregate feature information from different levels semantically and spatially, effectively capturing more small objects and boundaries, and achieving better segmentation performance.

[0154] Table 3. Experimental results comparing different void ratios and decoders.

[0155]

[0156] Observing the first and second rows of Table 3, it can be seen that MFRNet achieves better segmentation results, with segmentation accuracy 3.1% and 0.7% higher than the fixed-dilution-rate model MFRNet (r=4) and the coprime-dilution-rate model MFRNet (r=3,3,7,7,13,13), respectively, while its inference speed is only slightly lower than MFRNet (r=4). The reason for this is that the receptive field of the fixed-dilution-rate model is limited, which is not conducive to capturing large-scale object features or global features; furthermore, using multiple convolutions with the same dilation rate will produce a grid effect, further reducing segmentation accuracy; and the receptive field of the coprime-dilution-rate model is still smaller than that of this scheme. Therefore, the algorithm of this invention uses the {4,4,8,8,16,16} sequence with a larger dilation rate to achieve higher semantic segmentation accuracy.

[0157] Table 4 shows the IoU / % results of MFRNet and other algorithms on the Cityscapes test set for each category.

[0158]

[0159] Note: Bold text represents the optimal value, and "_" represents the suboptimal value.

[0160] To further demonstrate the advantages of the algorithm of this invention, Table 4 lists the specific segmentation accuracy values ​​of MFRNet and some of the aforementioned algorithms on 19 categories of the Cityscapes test set. As can be seen from the table, MFRNet achieved the best segmentation accuracy in 10 categories and the second best in 5 categories. Compared with the algorithms in Table 4, MFRNet significantly improves the segmentation accuracy when segmenting targets with sharp edges and corners, as well as small targets (such as walls, fences, and traffic signs), and also performs better in segmenting large targets with overlapping and mixed features (such as cars, trucks, and buses).

[0161] Figure 6 The image shows the visual segmentation results of DABNet, LDPNet, and MFRNet on the Cityscapes validation set (some areas with significant improvements are highlighted with rectangles for comparison). Compared to DABNet and LDPNet, the MFRNet of this invention exhibits better visual segmentation results. For example, in Scene 1, MFRNet extracts richer pedestrian information and more accurate contour localization; in Scene 2, the segmentation of traffic signs is closer to the label image. This demonstrates that the algorithm fully utilizes low-level spatial information, resulting in more accurate segmentation of details. In Scene 3, MFRNet's segmentation results for multiple large targets in a mixed scene are also closer to the label image, such as trucks and buses, while DABNet and LDPNet both show missegmentation. This shows that the algorithm of this invention has stronger discrimination capabilities and more accurate segmentation between similar targets. However, in complex mixed scenes, MFRNet's segmentation effect on small, distant targets is poor, as seen in Scene 4 where an occluded rider is missegmented. By comprehensively comparing the segmentation results of the patented algorithm MFRNet with DABNet and LDPNet in four scenarios, MFRNet fully integrates high-level features and low-level details, thus segmenting both large and small targets well and achieving better overall performance.

[0162] Table 5. Overall performance comparison of different algorithms on the Cityscapes test set.

[0163]

[0164] Table 5 (continued) Overall performance comparison of different algorithms on the Cityscapes test set

[0165]

[0166] To verify the effectiveness of this invention, MFRNet was compared with other advanced real-time semantic segmentation algorithms on the Cityscapes dataset, and the results are shown in Table 5. In terms of segmentation accuracy, MFRNet achieved the highest accuracy with an mIoU value of 72.6%, outperforming other real-time semantic segmentation algorithms in the table. Regarding inference speed, MFRNet reached 98 FPS, placing it in the upper-middle range among the methods. Although ESPNet was the fastest, its accuracy was only 60.3%, 12.3% lower than MFRNet, a significant difference in the segmentation field. DFANet had similar speed to MFRNet, but its parameter count was 8.6 times that of MFRNet, clearly achieving a better balance between speed and parameter count. In terms of parameters, MFRNet had a relatively small parameter count of 0.91M, approximately 2.5 times that of ENet and ESPNet, but the segmentation accuracy of these two algorithms was far inferior to that of MFRNet. Furthermore, compared to LEDNet, which had a similar parameter count, MFRNet also had a significant advantage in both segmentation accuracy and speed. By combining the three performance metrics, the MFRNet proposed in this invention maintains the model's lightweight design while balancing segmentation accuracy and inference speed, achieving a good balance between accuracy and speed.

[0167] To verify MFRNet's segmentation ability for lower-resolution images, experiments were conducted on the CamVid autonomous driving dataset. In the experiments, the optimizer method was changed to Adam, and the weight decay was set to 0.0002. Table 6 and... Figure 7 This section presents a comparison of MFRNet with several real-time semantic segmentation algorithms on the CamVid test set. In terms of segmentation accuracy, MFRNet achieved the second-best result at 67.3%, 4% lower than the highest-accurate DFANet, but its inference speed was 30 FPS faster than DFANet, and its parameter count was approximately 8.8 times lower. Regarding inference speed, MFRNet was only 3 FPS slower than ERFNet, which achieved the highest segmentation speed, but it had a greater advantage in segmentation accuracy and parameter count. ENet, ESPNet, and DABNet all had lower parameter counts than MFRNet, but MFRNet's segmentation accuracy was 16%, 11.7%, and 0.9% higher than them, respectively, and its inference speed was only slightly lower than ESPNet by 2 FPS, exceeding ENet and DABNet by 69 FPS and 13 FPS, respectively.

[0168] Table 6. Overall performance comparison of different algorithms on the CamVid test set.

[0169]

[0170] This invention constructs a lightweight asymmetric residual attention module (ARAM) by combining depthwise separable convolution and dilated convolution to extract rich contextual information and enhances the extraction of key features using an attention mechanism. Simultaneously, to fully integrate feature information from various stages to improve segmentation accuracy, an efficient fusion module (EFFM) is constructed based on the attention mechanism and combined with feature aggregation mechanism (FAM) to fuse features from different levels in a top-down manner. This achieves effective fusion of local and global information, spatial and contextual information, thus improving and refining the segmentation results.

[0171] The above experimental studies have demonstrated that the design of each module in the MFRNet of this invention improves the overall segmentation accuracy of the model and enhances its robustness, fully illustrating the effectiveness and rationality of the MFRNet network structure design, thus achieving a good balance between accuracy and inference speed.

[0172] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit the technical solutions. Those skilled in the art should understand that any modifications or equivalent substitutions to the technical solutions of the present invention without departing from the spirit and scope of the present invention should be covered within the scope of the claims of the present invention.

Claims

1. A real-time semantic segmentation method based on multi-feature reuse, characterized in that, Comprise: S1: obtaining a target image to be segmented; S2: inputting the target image into a trained semantic segmentation model, and outputting a corresponding semantic segmentation image; The semantic segmentation model comprises the following processing steps: S201: extracting initial feature information of the target image to obtain a primary feature map; S202: further extracting features based on the primary feature map through a first dense residual module composed of multiple asymmetric residual attention modules stacked to obtain a local feature map; In step S202, the first dense residual module is composed of three asymmetric residual attention modules stacked in series and having a cavity rate of 2; the output of the last asymmetric residual attention module in the first dense residual module is the local feature map; S203: further extracting features based on the local feature map through a second dense residual module composed of multiple asymmetric residual attention modules stacked to obtain a global feature map; In step S203, the second dense residual module is composed of six asymmetric residual attention modules stacked in series and having a cavity rate of 4, 4, 8, 8, 16, and 16, respectively; the output of the last asymmetric residual attention module in the second dense residual module is the global feature map; S204: fusing the global feature map and the local feature map through a feature aggregation module to obtain an intermediate fusion feature map; S205: fusing the intermediate fusion feature map and the primary feature map through an enhanced feature fusion module to obtain a final fusion feature map; S206: performing classification prediction based on the final fusion feature map through a classifier to output a corresponding semantic segmentation image; S3: taking the semantic segmentation image output by the semantic segmentation model as the real-time semantic segmentation result of the target image to be segmented.

2. The real-time semantic segmentation method based on multi-feature reuse according to claim 1, characterized in that: In step S201, the initial feature information of the target image is extracted through three convolutional layers stacked in series to obtain a primary feature map.

3. The real-time semantic segmentation method based on multi-feature reuse according to claim 1, wherein, Each asymmetric residual attention module extracts feature information of an input feature map through the following steps: 1) perform channel dimension reduction on the input feature map through 3x3 convolution operation to obtain a reduced feature map; 2) perform 3x1 and 1x3 depth convolution, 3x1 and 1x3 depth dilation convolution on the reduced feature map in turn to obtain depth convolution feature map and depth dilation convolution feature map, respectively; 3) input the depth convolution feature map into a channel attention submodule for feature extraction to obtain a channel attention feature map; 4) channel connect the depth dilation convolution feature map and the channel attention feature map, and obtain a fusion feature map through 1x1 convolution operation; 5) use summation operation to connect the fusion feature map and the original input feature map in residual to obtain the final feature map.

4. The real-time semantic segmentation method based on multi-feature reuse of claim 3, wherein: In the channel attention submodule, first process the input feature map through the globally pooled layer, the fully connected layer, the ReLU function layer, the fully connected layer and the Sigmoid function layer connected in series to obtain a channel feature map; then multiply the channel feature map and the input feature map to obtain the corresponding channel attention feature map.

5. The real-time semantic segmentation method based on multi-feature reuse of claim 1, wherein, In step S204, the feature aggregation module comprises the following processing steps: S2041: use Add operation to add the global feature map X1 and the local feature map X2 to obtain an added feature map X; S2042: encode the spatial information of the added feature map X along the vertical and horizontal directions respectively by using the decomposed two one-dimensional average pooling (H, 1) and (1, W) to generate the aggregated feature maps of the horizontal and vertical two spatial directions and S2043: merge the aggregated feature maps using a concatenation operation and The intermediate aggregated feature maps F are then outputted through a 1x1 convolution, batch normalization, and a nonlinear activation function operation z ; The formula is described as: wherein: [] denotes a concatenation operation; {} denotes batch normalization; and δ denotes a nonlinear activation function; wherein F z ∈ C / r x 1 x (H + W), r denotes a reduction rate; and C denotes a number of channels of the feature map; H and W represent the height and width of the feature map, respectively; S2044: Aggregate the intermediate feature map F z Decomposed into two tensors F in the height and width directions h and F w F is recovered by using 1×1 convolution operation respectively. h and F w The number of channels is the same as the summed feature map X, and then two attention weights g are obtained by the Sigmoid function operation. h and g w ; The formula is described as: g h = σ(Conv h 1×1 (F h )); g w = σ(Conv w 1×1 (F w )); wherein σ denotes a sigmoid activation function; wherein F h ∈ C / r x H x 1, F w ∈ C / r x 1 x W; S2045: multiply the added feature map X and the attention weight g h and g w to obtain the intermediate fusion feature map Y; The formula is described as: Y = X x g h x g w .

6. The real-time semantic segmentation method based on multi-feature reuse of claim 1, wherein: In step S205, the enhanced feature fusion module includes the following processing steps: S2051: connect the intermediate fusion feature map Y and the primary feature map through a connection operation, and perform batch normalization and ReLU activation function operation to obtain the primary feature map F; S2052: The two branches are used to perform average pooling and maximum pooling operations on the initial feature map F, and then 1x1 convolution, ReLU activation function and Sigmoid function operations are sequentially performed to obtain the feature weights F obtained by the average pooling and maximum pooling corresponding branches avg and F max ; The formula is described as: F avg = f [AvgPool (F)]; F max = f [MaxPool (F)]; In the formula, f represents sequentially performing 1*1 convolution, ReLU activation function and Sigmoid function operation; AvgPool represents average pooling operation; MaxPool represents maximum pooling operation; S2053: add the feature weight F avg and F max to the initial feature map F to reweight the features, obtaining the final fused feature map F c with channel weighting The formula is described as: F c = (F avg + F max ) x F + F.

7. The real-time semantic segmentation method based on multi-feature reuse according to claim 1, wherein: In step S202, before extracting the feature information of the primary feature map, the primary feature map is subjected to a down-sampling operation; In step S203, before extracting the feature information of the local feature map, the local feature map is subjected to a down-sampling operation; In step S204, before fusing the global feature map and the local feature map, the global feature map is subjected to an up-sampling operation and a 3*3 convolution operation.

8. The real-time semantic segmentation method based on multi-feature reuse of claim 7, wherein: In step S206, the channel number of the final fusion feature map is first compressed into the class number through a convolution operation, then the final fusion feature map is up-sampled to be consistent with the size of the target image, and finally a predicted semantic segmentation image is output.

Images