Image processing algorithm based on multilevel focus enhanced backbone network and sparse space training head

Abstract

The invention discloses an image processing algorithm based on a multilevel focus enhanced backbone network and a sparse space training head. According to the algorithm, firstly, a multi-level focus enhanced backbone network structure is provided, on the basis that an original single-branch SCTNet lightweight architecture is reserved, a channel attention module and a channel space mixed attention module are embedded at different depths respectively, cooperative enhancement of features and effective channel space positions is achieved, and a multi-level focus enhanced backbone network structure is obtained. Therefore, background noise is effectively suppressed and key signal features are highlighted on the premise of not obviously increasing parameter quantity and calculation quantity, and small target omission ratio and boundary ambiguity are reduced. Besides, a sparse space sparse training head is designed, and a segmentation head performs spatial recombination of joint guidance through spatial adjacency relations in modeling paths respectively and by utilizing appearance high-level features, so that the recognition capability of small targets, slender structures and long tail categories is remarkably improved, and misclassification among appearance similar categories is reduced. Experimental results show that the mode of combining multi-level feature enhancement and sparse space modeling has better feature extraction capability compared with an original network structure, and the detection and segmentation precision of the algorithm is remarkably improved.

Description

Image processing algorithm based on multi-level focus enhancement backbone network and sparse space training head Technical Field

[0001] This invention belongs to the field of computer vision and image processing technology, specifically relating to an image semantic segmentation and processing algorithm for autonomous driving scenarios, based on a multi-level focus enhancement backbone network and a sparse space training head. Background Technology

[0002] Image semantic segmentation, an important branch of computer vision, is one of the core technologies in autonomous driving environmental perception systems. It aims to classify each pixel in an image to achieve accurate identification and localization of targets such as roads, vehicles, pedestrians, and traffic signs. With the rapid development of deep learning technology, convolutional neural networks (CNNs) and Transformer architectures have achieved significant results in image segmentation tasks and have been widely applied in fields such as autonomous driving, intelligent monitoring, and medical image analysis.

[0003] Early image segmentation algorithms relied primarily on hand-designed features (such as color, texture, and edges), which proved less robust in complex scenes. In recent years, deep learning methods, represented by Fully Convolutional Networks (FCNs), U-Net, and their variants, have significantly improved segmentation accuracy through end-to-end training. In particular, the introduction of lightweight networks (such as SCTNet) has made it possible to deploy segmentation models on resource-constrained edge devices.

[0004] However, despite the advantages of existing lightweight segmentation networks in inference speed, they still have significant limitations when dealing with complex autonomous driving scenarios:

[0005] 1. Insufficient feature extraction capability: Existing lightweight backbone networks often sacrifice in-depth feature mining in pursuit of speed, lacking targeted enhancement of key feature channels and spatial locations, resulting in an inability to effectively focus on foreground targets when there is a lot of background noise interference.

[0006] 2. Difficulty in identifying small targets and long-tail classes: Autonomous driving scenarios contain a large number of small targets (such as traffic lights and traffic cones in the distance) and slender structures (such as lane lines and curbs). Existing algorithms are prone to losing detailed information about these targets during downsampling, and for long-tail classes with fewer samples, the model is prone to misclassification.

[0007] 3. Blurred boundaries: Due to the lack of effective spatial context constraints, the pixels at the edges of objects are often difficult to classify accurately, resulting in blurred boundaries in the segmentation results, which affects subsequent path planning and decision-making.

[0008] A common approach to improvement is to introduce attention mechanisms or complex feature pyramids, but this often leads to huge computational overhead. Therefore, how to design a network structure that can simultaneously suppress background noise, enhance key features, and effectively model spatial and semantic relationships without significantly increasing computational costs is a pressing technical challenge in the field of image processing. Summary of the Invention

[0009] The technical problem this invention aims to solve is: addressing the issues of high false negative rates for small targets, blurred boundary segmentation, and weak anti-interference capabilities against background noise in complex autonomous driving scenarios using existing lightweight image segmentation algorithms. This invention proposes an image processing algorithm based on a multi-layered focus enhancement backbone network and a sparse spatial training head. This algorithm collaboratively enhances features by embedding hybrid attention modules at different depths and designs a dual-channel prototype adaptive training head to optimize spatial and semantic representations, thereby significantly improving detection and segmentation accuracy.

[0010] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0011] An image processing algorithm based on a multi-level focus enhancement backbone network and a sparse space training head includes the following steps:

[0012] Step 1: Data Input and Preprocessing

[0013] The original RGB image of the autonomous driving scene is acquired, and then subjected to resolution adjustment, random cropping, random flipping, and photometric distortion enhancement. The pixel values ​​are then normalized to obtain the input image tensor. , where B is the batch size, H and W are the height and width of the image, respectively; construct a training dataset containing image I and its corresponding pixel-level semantic label Y;

[0014] Step 2: Construct a multi-level focus-enhanced backbone network to extract features:

[0015] The hierarchical convolutional neural network SCTNet is used as the basic architecture. The network consists of four stages (Stage 1 to Stage 4), and the feature map resolutions of each stage are 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image, respectively.

[0016] (1) Shallow feature extraction and channel enhancement (Stage 1-2):

[0017] Image I is first processed by a Stem module consisting of two 3×3 convolutional layers and stacked residual blocks (BasicBlock);

[0018] The feature map is obtained at the output of Stage 2. Here, a channel attention module is embedded to suppress background noise; the calculation steps of this module are as follows:

[0019] First of all, Global average pooling (AvgPool) and global max pooling (MaxPool) are performed separately to compress the spatial dimension to 1×1, resulting in two channel descriptors. ;

[0020] Next, these two descriptors are fed into a shared multilayer perceptron (MLP) containing a dimension-reduced convolutional layer (the number of channels is reduced to...). r) and up-dimensional convolutional layers (channel number restored to r) );

[0021] Finally, the two feature vectors output by the MLP are added element-wise, and then the channel weight vectors are generated by passing them through the Sigmoid activation function σ. The original feature map is then recalibrated to obtain enhanced features. :

[0022]

[0023]

[0024] in This indicates a channel-by-channel multiplication operation;

[0025] (2) Deep Feature Extraction and Hybrid Attention Focusing (Stage 3):

[0026] Stage 3 uses CFBlock as its basic unit, which utilizes... and The strip convolution approximates the large kernel convolution to capture long-range context;

[0027] The feature map is obtained at the output of Stage 3. Embed the Channel-Spatial Hybrid Attention Module (CBAM) here.

[0028] First, calculate the channel attention map. And on Weighted average yields intermediate features ;

[0029] Subsequently, the spatial attention map is calculated. :right Max pooling and average pooling are performed along the channel dimension to obtain two two-dimensional feature maps. After concatenating the two along the channel dimension, pass them through a 7×7 convolutional layer. Spatial feature aggregation is performed, followed by Sigmoid activation to obtain... ;

[0030] Finally, deep key features are generated. :

[0031] Step 3: Construct a Sparse Space Training Head (SSA Head) for feature decoding:

[0032] Using the feature map F output in step two (corresponding to (or features after aggregation), construct a segmentation head that integrates Spatial Prototype Adaptation (SPPA) and Semantic Prototype Adaptation (SEPA);

[0033] (1) Generate coarse guide and position code

[0034] The feature map F is mapped to a coarse prediction map with the number of channels equal to the number of classes K using a 1×1 convolutional layer. Then, a softmax operation is performed on it to obtain a normalized attention matrix. Where N is the total number of pixels; and F is mapped to positional features through a 3×3 convolution. ;

[0035] (2) Space Prototype Adaptive (SPPA) Calculation

[0036] Define a learnable category center position embedding matrix Using attention matrix A

[0037] Location features Perform weighted aggregation, calculate the spatial geometric center of the feature map, and combine it with... By stitching and merging, the updated spatial center is obtained through linear mapping. :

[0038]

[0039] (3) Semantic Prototype Adaptive (SEPA) Computation

[0040] Obtain the weights of the classification convolutional layer as a fixed semantic prototype. The attention matrix A is used to perform a weighted average of the original features F to calculate the dynamic semantic prototype of the current image. The formula is:

[0041]

[0042] Will and The semantic center is obtained by splicing and fusion through linear layers. ;

[0043] (4) Final segmentation prediction

[0044] Compare the original feature F with the location feature Addition yields enhanced features Calculate its relationship with the central hub The matrix product is used to output the final segmentation result. :

[0045]

[0046] Step 4: Construct and train the distillation loss function:

[0047] Construct SegFormer-B2 as the teacher network, keeping its parameters frozen; use the networks constructed in steps two and three as the student network; define the total loss function. Cross-entropy loss of the main segmentation head Cross-entropy loss of the auxiliary head and feature alignment loss Weighted sum:

[0048]

[0049] in , The hyperparameters for balancing the weights, and These are the feature maps of the student network and the teacher network at the corresponding levels. Represents the mean square error; by minimizing End-to-end training was conducted on the student network.

[0050] Compared with the prior art, the present invention has the following technical effects:

[0051] 1. More accurate feature extraction: The multi-layered focus-enhancing backbone network proposed in this invention differs from traditional single-dimensional enhancement methods. In the shallow layers, it focuses on suppressing noise through channel attention, while in the deeper layers, it uses a spatial-channel hybrid attention approach to focus on key targets. This collaborative enhancement strategy effectively solves the problem of target feature obscuring caused by complex backgrounds without significantly increasing the number of parameters.

[0052] 2. Strong ability to recognize small targets and slender structures: By introducing a Sparse Spatial Training Head (SSA Head), the SPPA module explicitly models spatial adjacency relationships, and the SEPA module jointly guides semantic feature reorganization. This dual-channel prototype adaptive mechanism significantly improves the recognition ability of small targets (such as pedestrians in the distance), slender structures (such as lane lines), and long-tail categories, and reduces misclassification between similar-looking categories.

[0053] 3. Fast convergence speed and high accuracy: By combining the knowledge distillation strategy of the teacher network and using feature alignment loss to guide the student network to learn better feature representations, the algorithm converges faster during training and can still maintain high segmentation accuracy (mIoU) under low computing power conditions. Attached Figure Description

[0054] Figure 1 is a flowchart of the overall image processing algorithm of the present invention.

[0055] Figure 2 is a bar chart comparing the IoU performance of the algorithm of this invention. Detailed Implementation

[0056] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings:

[0057] This invention proposes an image processing algorithm based on a multi-level focus enhancement backbone network and a sparse space training head, aiming to solve problems such as insufficient feature extraction, missed detection of small targets, and blurred boundaries in semantic segmentation tasks in autonomous driving scenarios. Figure 1 is a flowchart of the overall image processing algorithm of this invention, which mainly includes four steps: data input and preprocessing, multi-level feature extraction, sparse space feature decoding, and knowledge distillation training.

[0058] Step 1: Data Input and Preprocessing

[0059] The original RGB image of the autonomous driving scene is acquired, and then subjected to resolution adjustment, random cropping, random flipping, and photometric distortion enhancement. The pixel values ​​are then normalized to obtain the input image tensor. , where B is the batch size, H and W are the height and width of the image, respectively; construct a training dataset containing image I and its corresponding pixel-level semantic label Y;

[0060] Step 2: Construct a multi-level focus-enhanced backbone network to extract features:

[0061] The hierarchical convolutional neural network SCTNet is used as the basic architecture. The network consists of four stages (Stage 1 to Stage 4), and the feature map resolutions of each stage are 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image, respectively.

[0062] (1) Shallow feature extraction and channel enhancement (Stage 1-2):

[0063] Image I is first processed by a Stem module consisting of two 3×3 convolutional layers and stacked residual blocks (BasicBlock);

[0064] The feature map is obtained at the output of Stage 2. Here, a channel attention module is embedded to suppress background noise; the calculation steps of this module are as follows:

[0065] First of all, Global average pooling (AvgPool) and global max pooling (MaxPool) are performed separately to compress the spatial dimension to 1×1, resulting in two channel descriptors. ;

[0066] Next, these two descriptors are fed into a shared multilayer perceptron (MLP) containing a dimension-reduced convolutional layer (the number of channels is reduced to...). r) and up-dimensional convolutional layers (channel number restored to r) );

[0067] Finally, the two feature vectors output by the MLP are added element-wise, and then the channel weight vectors are generated by passing them through the Sigmoid activation function σ. The original feature map is then recalibrated to obtain enhanced features. :

[0068]

[0069]

[0070] in This indicates a channel-by-channel multiplication operation;

[0071] (2) Deep Feature Extraction and Hybrid Attention Focusing (Stage 3):

[0072] Stage 3 uses CFBlock as its basic unit, which utilizes... and The strip convolution approximates the large kernel convolution to capture long-range context;

[0073] The feature map is obtained at the output of Stage 3. Embed the Channel-Spatial Hybrid Attention Module (CBAM) here.

[0074] First, calculate the channel attention map. And on Weighted average yields intermediate features ;

[0075] Subsequently, the spatial attention map is calculated. :right Max pooling and average pooling are performed along the channel dimension to obtain two two-dimensional feature maps. After concatenating the two along the channel dimension, pass them through a 7×7 convolutional layer. Spatial feature aggregation is performed, followed by Sigmoid activation to obtain... ;

[0076] Finally, deep key features are generated. :

[0077] Step 3: Construct a Sparse Space Training Head (SSA Head) for feature decoding:

[0078] Using the feature map F output in step two (corresponding to (or features after aggregation), construct a segmentation head that integrates Spatial Prototype Adaptation (SPPA) and Semantic Prototype Adaptation (SEPA);

[0079] (1) Generate coarse guide and position code

[0080] The feature map F is mapped to a coarse prediction map with the number of channels equal to the number of classes K using a 1×1 convolutional layer. Then, a softmax operation is performed on it to obtain a normalized attention matrix. Where N is the total number of pixels; and F is mapped to positional features through a 3×3 convolution. ;

[0081] (2) Space Prototype Adaptive (SPPA) Calculation

[0082] Define a learnable category center position embedding matrix Using attention matrix A

[0083] Location features Perform weighted aggregation, calculate the spatial geometric center of the feature map, and combine it with... By stitching and merging, the updated spatial center is obtained through linear mapping. :

[0084]

[0085] (3) Semantic Prototype Adaptive (SEPA) Computation

[0086] Obtain the weights of the classification convolutional layer as a fixed semantic prototype. The attention matrix A is used to perform a weighted average of the original features F to calculate the dynamic semantic prototype of the current image. The formula is:

[0087]

[0088] Will and The semantic center is obtained by splicing and fusion through linear layers. ;

[0089] (4) Final segmentation prediction

[0090] Compare the original feature F with the location feature Addition yields enhanced features Calculate its relationship with the central hub The matrix product is used to output the final segmentation result. :

[0091]

[0092] Step 4: Construct and train the distillation loss function:

[0093] Construct SegFormer-B2 as the teacher network, keeping its parameters frozen; use the networks constructed in steps two and three as the student network; define the total loss function. Cross-entropy loss of the main segmentation head Cross-entropy loss of the auxiliary head and feature alignment loss Weighted sum:

[0094]

[0095] in , The hyperparameters for balancing the weights, and These are the feature maps of the student network and the teacher network at the corresponding levels. Represents the mean square error; by minimizing End-to-end training was conducted on the student network.

[0096] To verify the effectiveness of the algorithm proposed in this invention, computer simulation experiments were conducted on the standard Cityscapes dataset.

[0097] 1. Experimental Environment and Parameter Settings

[0098] The experimental hardware platform uses an Intel Xeon processor and an NVIDIA GeForce RTX 3090 GPU (24GB of video memory). The software environment is based on the Linux operating system, the PyTorch deep learning framework, and the MMSegmentation toolkit.

[0099] The optimizer used was AdamW with an initial learning rate of 0.0004 and weight decay of 0.0125. To accelerate convergence, a multiplier of 10.0 was set for the learning rate of the decoder head. The learning rate scheduling strategy employed cosine annealing, with 1500 warm-up iterations and a minimum learning rate of 1×10⁻⁶. −6 The total number of iterations was set to 160,000, and the model was saved and validated every 2,000 iterations.

[0100] 2. Segmentation accuracy (mIoU) analysis

[0101] Experiment No. Bone-Based Enhancement (CBAM) Sparse Spatial Head (SSA Head) mIoU (%) Parameters 1 ×× 72.8 4.65 M 2 √× 73.3 1 4.66 M 3 ×√ 76.1 2 4.73 M 4 √√ 76.7 9 4.73 M surface

[0102] Table 1

[0103] The experimental results are shown in Table 1. On the Cityscapes validation set, the improved algorithm proposed in this invention achieved an average intersection-union ratio (mIoU) of 76.79%, which is about 3.99% higher than the original baseline model (72.8%), while the number of parameters only increased by 0.08M (from 4.65M to 4.73M), maintaining the lightweight advantage of the network.

[0104] The specific ablation experiment analysis is as follows:

[0105] (1) Effectiveness of the backbone enhancement module: Comparing Experiment 1 and Experiment 2, it can be seen that when only the backbone enhancement module (CBAM) is introduced, the mIoU increases from 72.8% to 73.31%, an increase of 0.51%. Moreover, the number of parameters only increases by 0.01M, which shows that the hybrid attention mechanism effectively enhances the feature extraction capability without increasing the computational cost.

[0106] (2) Effectiveness of the Sparse Space Training Head: Comparing Experiment 1 and Experiment 3, it can be seen that after replacing the original head with the Sparse Space Training Head (SSA Head), the mIoU is significantly improved by 3.32% (from 72.8% to 76.12%). This proves that the SSA module greatly improves the accuracy of pixel-level classification through the dual adaptive mechanism of spatial and semantic prototypes.

[0107] (3) Module synergy: Experiment 4 demonstrates the effect of simultaneously integrating backbone enhancement and sparse spatial head, with the model achieving an optimal performance of 76.79%. This indicates a good synergistic effect between the two modules, jointly promoting the maximization of segmentation accuracy.

[0108] 3. Segmented Scene and Visual Analysis

[0109] As shown in Figure 2, this embodiment performs a detailed quantitative IoU analysis on 19 semantic categories in the Cityscapes validation set. Experimental results show that the improved algorithm proposed in this invention (blue bar chart) significantly outperforms the baseline model (grey bar chart) in overall performance, with the average intersection-union ratio (mIoU) increasing from 72.94% to 76.79%, achieving a significant increase of 3.85%.

[0110] Specifically, the performance improvement of this invention is particularly outstanding for small targets, slender structures, and long-tail categories, verifying the effectiveness of the improved module:

[0111] (1) Significant improvement in slender structure targets:

[0112] In the pole category, the IoU improvement reached 11.3%. This is mainly attributed to the Spatial Prototype Adaptation (SPPA) module in the sparse spatial training head. Since pole-shaped objects occupy very few pixels and have a large span in the image, traditional convolutional downsampling is prone to losing spatial information. SPPA introduces a position embedding matrix to explicitly model the spatial geometric center of the object, effectively preserving the topological features of the slender structure, thereby avoiding "breaks" or missed detections.

[0113] (2) Enhanced ability to identify small targets:

[0114] For typical small targets such as Traffic Lights and Traffic Signs, the IoU was improved by 8.9% and 7.6%, respectively. This improvement is attributed to the synergistic effect of the CBAM hybrid attention mechanism embedded in the backbone network and the SSA head. The attention mechanism effectively suppresses background noise in deep features, focusing on the effective pixels of small targets; while the SSA head alleviates the class imbalance problem through a sparse point sampling strategy, significantly improving the recall rate of small targets.

[0115] (3) Distinguishing between complex and similar categories:

[0116] In the Motorcycle and Rider categories, improvements of 10.1% and 6.7% were achieved, respectively. These targets are typically structurally complex and prone to mutual occlusion. The Semantic Prototype Adaptation (SEPA) module of this invention enhances the discriminative power of features in the semantic space by dynamically updating the semantic center, effectively reducing confusion between riders and motorcycles, or motorcycles and the background.

[0117] In summary, the statistical data in Figure 2 strongly demonstrates that the multi-level focus enhancement and dual-channel prototype adaptive mechanism proposed in this invention can effectively solve the segmentation problem of "small, fine, and chaotic" targets in autonomous driving scenarios.

Claims

1. An image processing algorithm based on a multi-level focus enhancement backbone network and a sparse space training head, characterized in that... The process includes the following steps: Step 1, Data Input and Preprocessing: Obtain the original RGB image of the autonomous driving scene, perform resolution adjustment, random cropping, random flipping, and photometric distortion enhancement, and normalize the pixel values ​​to obtain the input image tensor. , where B is the batch size, H and W are the height and width of the image respectively; construct a training dataset containing image I and its corresponding pixel-level semantic label Y; step 2, construct a multi-level focus enhancement backbone network to extract features: use a hierarchical convolutional neural network SCTNet as the basic architecture, which contains four stages (Stage 1 to Stage 4), and the feature map resolution of each stage is 1 / 4, 1 / 8, 1 / 16, 1 / 32 of the original image respectively; (1) shallow feature extraction and channel enhancement (Stage 1-2): image I first goes through the Stem module composed of two 3×3 convolutional layers and stacked residual blocks (BasicBlock); at the output of Stage 2, the feature map is obtained. A channel attention module is embedded here to suppress background noise; The calculation steps for this module are as follows: First, for Global average pooling (AvgPool) and global max pooling (MaxPool) are performed separately to compress the spatial dimension to 1×1, resulting in two channel descriptors. Secondly, these two descriptors are fed into a shared multilayer perceptron (MLP), which contains a dimension-reduced convolutional layer (the number of channels is reduced to...). r) and up-dimensional convolutional layers (channel number restored to r) Finally, the two feature vectors output by the MLP are added element-wise, and then the channel weight vectors are generated by passing them through the Sigmoid activation function σ. The original feature map is then recalibrated to obtain enhanced features. : in (1) Indicates channel-wise multiplication operation; (2) Deep feature extraction and hybrid attention focusing (Stage 3): Stage 3 uses CFBlock as the basic unit, which utilizes and The strip convolution approximates a large kernel convolution to capture long-range context; at the Stage 3 output, the feature map is obtained. Here, a channel-spatial hybrid attention module (CBAM) is embedded; first, the channel attention map is computed. And on Weighted average yields intermediate features ； Subsequently, the spatial attention map is calculated. :right Max pooling and average pooling are performed along the channel dimension to obtain two two-dimensional feature maps. After concatenating the two along the channel dimension, pass them through a 7×7 convolutional layer. Spatial feature aggregation is performed, followed by Sigmoid activation to obtain... Finally, deep key features are generated. ： Step 3: Construct a Sparse Space Training Head (SSA Head) for feature decoding: Utilize the feature map F output from Step 2 (corresponding to...) (or features after aggregation), construct a segmentation head integrating spatial prototype adaptation (SPPA) and semantic prototype adaptation (SEPA); (1) Generate coarse guidance and position encoding by mapping the feature map F to a coarse prediction map with the number of channels equal to the number of categories K through a 1×1 convolutional layer. Then, a softmax operation is performed on it to obtain a normalized attention matrix. Where N is the total number of pixels; and F is mapped to positional features through a 3×3 convolution. (2) Spatial Prototype Adaptation (SPPA) computation defines a learnable class center location embedding matrix. ; Use attention matrix A to analyze location features Perform weighted aggregation, calculate the spatial geometric center of the feature map, and combine it with... By stitching and merging, the updated spatial center is obtained through linear mapping. ： (3) Semantic Prototype Adaptation (SEPA) Calculation: Obtain the weights of the classification convolutional layer as a fixed semantic prototype. The attention matrix A is used to perform a weighted average of the original features F to calculate the dynamic semantic prototype of the current image. The formula is: Will and The semantic center is obtained by splicing and fusion through linear layers. (4) The final segmentation prediction combines the original features F with the location features. Addition yields enhanced features Calculate its relationship with the central hub The matrix product is used to output the final segmentation result. : Step 4: Construct and train the distillation loss function: Construct SegFormer-B2 as the teacher network, keeping its parameters frozen; use the networks constructed in Steps 2 and 3 as the student networks; define the total loss function. Cross-entropy loss of the main segmentation head Cross-entropy loss of the auxiliary head and feature alignment loss Weighted sum: in , The hyperparameters for balancing the weights, and These are the feature maps of the student network and the teacher network at the corresponding levels. Represents the mean square error; by minimizing End-to-end training was conducted on the student network.

Images