A real-time segmentation modeling method based on a multi-scale constraint mechanism
By using multi-scale constraint mechanism for feature extraction and processing, combined with depthwise separable convolution and dynamic weight allocation, the problems of boundary accuracy and computational efficiency in autonomous vehicles are solved, achieving high-precision real-time segmentation and efficient computation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- 城市之光(深圳)无人驾驶有限公司
- Filing Date
- 2026-03-11
- Publication Date
- 2026-06-09
AI Technical Summary
Existing semantic segmentation technology for autonomous vehicles has low segmentation boundary accuracy in outdoor environments, affecting driving and operational accuracy, while computational efficiency needs to be improved.
A multi-scale constraint mechanism is adopted for feature extraction. Combining segmentation output head, compression processing and multi-scale feature upsampling, a hybrid loss function is designed. A dynamic weight allocation strategy is used to optimize global and local boundaries. Real-time processing is achieved through depthwise separable convolution.
It significantly improves boundary segmentation accuracy, alleviates boundary ambiguity issues, enhances the driving and operational accuracy of autonomous vehicles, reduces GPU memory usage, and achieves a real-time processing speed of 35 FPS.
Smart Images
Figure CN122176711A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence, and in particular to a real-time segmentation modeling method based on a multi-scale constraint mechanism. Background Technology
[0002] Segmentation technology in the field of artificial intelligence generally refers to image segmentation or video segmentation. It is one of the core tasks in computer vision. Its core goal is to classify each pixel in a mathematical image or video and divide the image into several regions with specific meanings (such as objects and backgrounds), so that the computer can understand the fine structure (specific contours and shapes) of the image.
[0003] Currently, based on different task objectives, segmentation techniques are generally classified into the following categories:
[0004] (1) Semantic segmentation: This involves assigning each pixel in an image to a category label. It does not distinguish between different individuals of the same category and is mainly used in autonomous driving and medical image analysis.
[0005] (2) Instance segmentation: classifying pixels and distinguishing different individuals of the same category. It is mainly used in robot grasping, crowd counting, video tracking, etc.
[0006] (3) Panoramic segmentation: It combines the features of semantic segmentation and instance segmentation, and can provide a unique label for each pixel. It can be widely used in different fields.
[0007] (4) Interactive segmentation: Allows users to guide the model to segment specific objects through simple interactions such as clicking and drawing frames. It has strong generalization ability and can segment everything without retraining for specific objects.
[0008] Clearly, semantic segmentation is currently the primary method used in the field of autonomous driving. However, since autonomous vehicles operate in outdoor environments with a high level of noise, directly employing existing semantic segmentation techniques can result in low accuracy of the segmented boundaries, affecting the driving and operational precision of autonomous vehicles. Furthermore, the computational efficiency of existing semantic segmentation techniques during the segmentation process needs further improvement.
[0009] Therefore, existing technologies still need to be improved. Summary of the Invention
[0010] In view of the shortcomings of the prior art, the purpose of this invention is to provide a real-time segmentation modeling method based on a multi-scale constraint mechanism, which aims to solve the problem that the segmentation boundary accuracy of the prior art is not high, which affects the driving and operation accuracy of autonomous vehicles, and at the same time solve the problem that the computational efficiency of the prior art needs to be improved.
[0011] The technical solution of the present invention is as follows: a real-time segmentation modeling method based on a multi-scale constraint mechanism, the method comprising:
[0012] Feature extraction is performed on the image, and a segmentation output header is added for compression.
[0013] The compressed features are upsampled by a factor of X to calculate loss1.
[0014] The compressed features are upsampled at full size, and their boundary features are extracted to calculate loss2.
[0015] Based on the loss1 and loss2, a hybrid loss function is designed, and the total loss L is calculated. The result is then connected to the segmentation output head to output the semantic segmentation result.
[0016] Obviously, the real-time segmentation modeling method based on multi-scale constraint mechanism in this invention extracts image features, adds a segmentation output head, and performs compression processing. Finally, it obtains loss1 and loss2 through different multi-scale feature processing, and combines loss1 and loss2 to make the final output semantic segmentation result capture multi-scale feature context information, which solves the limitations of traditional single-scale feature processing and ensures that the captured data is more comprehensive and has better integrity. The compressed features are upsampled by X times and full size respectively, and boundary feature extraction is performed on the basis of full-size upsampling. Finally, a hybrid loss function is designed based on the calculated loss1 and loss2, so that the final total loss is a combination of global and local boundary optimization results. The corresponding Cityscapes dataset achieves 82.3% mIoU, ensuring high accuracy of the segmentation results after using this invention and effectively improving the driving and operation accuracy of autonomous vehicles.
[0017] In one embodiment, the feature extraction of the image specifically includes:
[0018] The original classification header of the image is removed using a basic backbone network.
[0019] The image with the original classification header removed is retained in the conv5_x output.
[0020] In the above process, the size of the output feature map is 1 / 32 of the size of the input feature map.
[0021] In one embodiment, the segmentation output head includes: a first preset convolutional layer and a ReLU activation layer. The segmentation output head is connected to a 1*1 convolutional output class prediction.
[0022] In one embodiment, the feature compression process includes compressing the features through a convolution with a second preset size. This process compresses the features by a corresponding factor as needed.
[0023] In one embodiment, the step of performing full-size upsampling on the compressed features and extracting their boundary features specifically includes:
[0024] A cascaded deconvolution structure is used to perform full-size upsampling on the compressed features.
[0025] An edge detection kernel was constructed. The constructed edge detection kernel has a hollow spatial pyramid structure, thus it can identify large-scale objects and fine structures, improving mIoU by 1.8%.
[0026] Parallel computation of the horizontal / vertical gradients of the features.
[0027] Perform a fusion operation to output a boundary intensity map to extract boundary features.
[0028] After the above processing, differential edge detection can be achieved, avoiding information loss caused by traditional post-processing, and the boundary segmentation accuracy is improved by 15.6%, effectively improving the boundary blurring problem and ensuring enhanced boundary perception in the final result.
[0029] In one embodiment, the step of designing a hybrid loss function based on the loss1 and loss2 and calculating the total loss L specifically includes:
[0030] Based on loss1, the cross-entropy loss L1 with class weights is calculated.
[0031] The boundary weighted cross-entropy L2 is calculated based on loss2.
[0032] Design a dynamic weight allocation strategy: w1=σ(epoch / 10), w2=1-w1, where σ is the sigmoid function and epoch represents the current training round number.
[0033] Calculate the total loss: L = w1·L1 + w2·L2.
[0034] Clearly, the dynamic weight allocation strategy employed in the above process achieves automatic balancing across different training stages. The final calculated total loss is based on loss1 and loss2, comprehensively considering both global classification and local boundary optimization issues, achieving 82.3% mIoU on the Cityscapes dataset. Furthermore, combining loss1 and loss2 leverages the advantages of multi-scale feature fusion, significantly reducing GPU memory usage by up to 40%, thus overcoming the limitations of traditional single-scale feature extraction.
[0035] In one embodiment, the Sobel operator is used to construct the edge detection kernel.
[0036] In one embodiment, bilinear interpolation is used to perform an upsampling operation of X times on the compressed features to calculate loss1.
[0037] In summary, the real-time segmentation modeling method based on a multi-scale constraint mechanism proposed in this paper has the following beneficial effects:
[0038] 1. Effectively improves accuracy: It can realize differential edge detection, avoid information loss caused by traditional post-processing, improve the boundary segmentation accuracy by 15.6%, effectively improve the boundary blurring problem, and ensure the enhanced boundary perception of the final result;
[0039] 2. Improved and optimized computational efficiency: The depthwise separable convolutional design achieves a real-time processing speed of 35FPS. It leverages the advantages of multi-scale feature fusion, significantly reducing GPU memory usage by up to 40%, and overcoming the limitations of traditional single-scale feature processing.
[0040] 3. Automatic balancing: A dynamic weight allocation strategy is adopted to achieve automatic balancing at different training stages. The final calculated total loss is based on loss1 and loss2, which comprehensively considers global classification and local boundary optimization problems. It can achieve 82.3% mIoU on the Cityscapes dataset, which solves the limitations of traditional single-scale features and ensures that the captured data is more comprehensive and has better integrity. Attached Figure Description
[0041] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings:
[0042] Figure 1 This is a flowchart of the present invention. Detailed Implementation
[0043] To make the objectives, technical solutions, and effects of this invention clearer and more explicit, the invention is further described in detail below. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. The embodiments of the invention are described below in conjunction with the accompanying drawings.
[0044] Please refer to Figure 1This invention provides a real-time segmentation modeling method based on a multi-scale constraint mechanism. By extracting image features, adding a segmentation output header, and compressing the image, different multi-scale feature processing is performed to obtain loss1 and loss2. Finally, loss1 and loss2 are combined, enabling the final semantic segmentation output to capture multi-scale feature context information. This overcomes the limitations of traditional single-scale feature processing, ensuring more comprehensive and complete data capture. Furthermore, by performing 1 / 8 and full-size upsampling operations on the compressed features, and then extracting boundary features based on the full-size upsampling, a hybrid loss function is designed based on the calculated loss1 and loss2. This results in a total loss that integrates global and local boundary optimization, achieving 82.3% mIoU on the Cityscapes dataset. This ensures high accuracy of the segmentation results using this invention, effectively improving the driving and operational accuracy of autonomous vehicles.
[0045] Specifically, the real-time segmentation modeling method based on a multi-scale constraint mechanism in this invention includes the following steps:
[0046] S1: Extract features from the image, add a segmentation output header, and perform compression processing.
[0047] Specifically, in this embodiment, the feature extraction of the image includes:
[0048] S10: Remove the original classification header of the image using the basic backbone network.
[0049] The basic network backbone is a pre-trained ResNet50.
[0050] S11: And retain the image with the original classification header removed in the conv5_x output.
[0051] It should be noted that "image" refers to the original input image; after step S11, the final output feature map size is 1 / 32 of the input feature map.
[0052] Specifically, in this embodiment, the segmentation output head includes: a first preset convolution and a ReLU activation layer. The first preset convolution is a 3*3 convolution, and the segmentation output head composed of the first preset convolution and the ReLU activation layer is connected to a 1*1 convolution to output class prediction.
[0053] Specifically, in this embodiment, the feature compression process includes compressing the features using a convolution with a second preset size. In this embodiment, the second preset size is 1*1, and the 1*1 convolution in this invention is a depthwise separable convolution. Specifically, the 2048-dimensional features are compressed to 256 dimensions using a 1*1 convolution. This process involves compressing the features by a corresponding factor according to requirements. In this invention, the compressed features are processed in steps S2, S3, and S4 to achieve the advantages of multi-scale feature fusion, significantly reducing GPU memory usage by up to 40%, and overcoming the limitations of traditional single-scale feature extraction. Furthermore, the use of depthwise separable convolution enables a real-time processing speed of 35 FPS, improving overall computational efficiency.
[0054] S2: Perform an upsampling operation of X times on the compressed features and calculate loss1. In this embodiment, an upsampling operation of 1 / 8 times is performed, and bilinear interpolation is used for this upsampling operation.
[0055] S3: Perform full-size upsampling on the compressed features and extract their boundary features to calculate loss2.
[0056] Specifically, in this embodiment, step S3 includes:
[0057] S30: A cascaded deconvolution structure is used to perform full-size upsampling on the compressed features. This process involves an 8x upsampling operation; each stage includes: a deconvolution layer → BatchNorm → ReLU activation layer.
[0058] S31: Construct the edge detection kernel. The constructed edge detection kernel is a hollow spatial pyramid structure, thus it can identify large-scale objects and fine structures, improving mIoU by 1.8%. Specifically, in this embodiment, the Sobel operator is used to construct the edge detection kernel.
[0059] S32: Parallel computation of the horizontal / vertical gradients of the features.
[0060] S33: Perform a fusion operation and output a boundary intensity map to extract boundary features. Specifically, in this embodiment, the fusion formula used during the fusion operation is the vector magnitude calculation formula.
[0061] After the above processing, differential edge detection can be achieved, avoiding information loss caused by traditional post-processing, and the boundary segmentation accuracy is improved by 15.6%, effectively improving the boundary blurring problem and ensuring enhanced boundary perception in the final result.
[0062] S4: Design a hybrid loss function based on loss1 and loss2, calculate the total loss L, connect it to the segmentation output head, and output the semantic segmentation result.
[0063] Specifically, in this embodiment, step S4 includes:
[0064] S40: Calculate the cross-entropy loss L1 with class weights based on loss1.
[0065] In this embodiment, L1 = -Σ(y1*log(p1)+(1-y1)*log(1-p1)), where y1 represents the true label and p1 represents the predicted probability. When the true label y1=1, the formula simplifies to −log(p1), and it is desirable for the predicted probability p1 to be as close to 1 as possible (because log(1)=0, the loss is minimized; if p1 is close to 0, log(p1) tends to negative infinity, and the loss tends to positive infinity). When the true label y1=0, the formula simplifies to −log(1−p1), and it is desirable for the predicted probability p1 to be as close to 0 as possible (i.e., 1−p1 is close to 1), p1∈(0,1).
[0066] S41: Calculate the boundary weighted cross-entropy L2 based on loss2.
[0067] Specifically, in this embodiment:
[0068] The L2 = Σ(w_edge·y2*log(p)), w_edge=1+λ·G, where w_edge represents the edge weight coefficient, y2 represents the true label, p2 represents the predicted probability (the probability that the model predicts that the position belongs to the edge), and p2∈(0,1); when y2=1, it means that the position is the edge, and when y2=0, it means that the background.
[0069] S42: Design a dynamic weight allocation strategy: w1=σ(epoch / 10), w2=1-w1, where σ is the sigmoid function and epoch represents the current training round number.
[0070] S43: Calculate the total loss: L = w1·L1 + w2·L2.
[0071] Clearly, the above process employs a dynamic weight allocation strategy to achieve automatic balancing across different training stages. The final calculated total loss is based on loss1 and loss2, taking into account both global classification and local boundary optimization issues. On the Cityscapes dataset, it achieves 82.3% mIoU.
[0072] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.
Claims
1. A real-time segmentation modeling method based on a multi-scale constraint mechanism, characterized in that, The method includes: Perform feature extraction on the image, add it to the segmentation output header, and then perform feature compression processing; The compressed features are upsampled by a factor of X, and the loss1 is calculated. The compressed features are upsampled at full size, and their boundary features are extracted to calculate loss2. Based on the loss1 and loss2, a hybrid loss function is designed, and the total loss L is calculated. The result is then connected to the segmentation output head to output the semantic segmentation result.
2. The real-time segmentation modeling method based on a multi-scale constraint mechanism according to claim 1, characterized in that, The image feature extraction process specifically includes: The original classification header of the image is removed using a basic backbone network; The image with the original classification header removed is retained in the conv5_x output.
3. The real-time segmentation modeling method based on a multi-scale constraint mechanism according to claim 1, characterized in that, The segmentation output head includes: a first preset convolutional layer and a ReLU activation layer.
4. The real-time segmentation modeling method based on a multi-scale constraint mechanism according to claim 1, characterized in that, The feature compression process includes compressing the features through convolution of a second preset size.
5. The real-time segmentation modeling method based on a multi-scale constraint mechanism according to claim 1, characterized in that, The process of performing full-size upsampling on the compressed features and extracting their boundary features specifically includes: A cascaded deconvolution structure is used to perform full-size upsampling on the compressed features; Construct an edge detection kernel; Parallel computation of the horizontal / vertical gradients of features; Perform a fusion operation to output a boundary intensity map to extract boundary features.
6. The real-time segmentation modeling method based on a multi-scale constraint mechanism according to claim 1, characterized in that, The design of the hybrid loss function based on loss1 and loss2, and the calculation of the total loss L, specifically includes: Based on loss1, the cross-entropy loss L1 with class weights is calculated; Calculate the boundary-weighted cross-entropy L2 based on loss2; Design a dynamic weight allocation strategy: w1=σ(epoch / 10), w2=1-w1, where σ is the sigmoid function and epoch represents the current training round number; Calculate the total loss: L = w1·L1 + w2·L2.
7. The real-time segmentation modeling method based on a multi-scale constraint mechanism according to claim 5, characterized in that, An edge detection kernel is constructed using the Sobel operator.
8. The real-time segmentation modeling method based on a multi-scale constraint mechanism according to claim 1, characterized in that, The compressed features were upsampled by X times using bilinear interpolation to calculate loss1.