A lightweight semantic segmentation method fusing rectangular context calibration

Abstract

The application discloses a kind of fusion rectangle context calibration light weight semantic segmentation method, collects the image data under the scene of conventional vision, remote sensing and distortion, is constructed semantic segmentation dataset after pre-processing and cleaning, and training set and verification set are divided according to 9:1 proportion;With PSPNet as the architecture basis design rectangle context calibration attention RCCA module, design bidirectional fusion attention BFA module, select StarNet as backbone network, introduce star operation to realize low-dimensional to high-dimensional feature efficient conversion, combined with 7×7 deep convolution and 1×1 point-by-point convolution reduces parameter quantity, and improve forward propagation mechanism synchronous output multi-scale feature, finally form light weight semantic segmentation network TransitNet;The model and inference process of training are deployed, realize accurate semantic segmentation under multiple scenes.Compared with prior art, the segmentation performance of the present application is better than that of the conventional comparison network, the generalization ability is outstanding, and real-time environmental perception and remote sensing image road and building fine classification can be realized.

Description

Technical Field

[0001] This invention relates to the field of computer vision and semantic segmentation technology, specifically to a lightweight semantic segmentation method that integrates rectangular context calibration, applicable to scenarios such as real-time environmental perception of autonomous driving vehicle terminals and fine classification of roads and buildings in remote sensing images. Background Technology

[0002] Against the backdrop of rapid development in artificial intelligence and computer vision technologies, semantic segmentation, as one of the core tasks, has been deeply applied in key fields such as autonomous driving, remote sensing interpretation, and urban planning. Among these, semantic segmentation of traffic scenes has become a research focus for both academia and industry due to challenges such as complex dynamic environments, multi-scale target interactions, and spatiotemporal feature modeling. Accurate traffic scene segmentation can provide real-time environmental understanding for autonomous vehicles and provide fundamental support for traffic flow analysis and anomaly detection in intelligent transportation systems.

[0003] While existing semantic segmentation technologies have made significant progress—for example, FCN achieved end-to-end pixel-level segmentation using full convolution for the first time, PSPNet improved the parsing accuracy of complex scenes through pyramid pooling modules, and the DeepLab series enhanced long-distance dependency capture capabilities through dilated convolutions—two core problems still exist in practical applications. First, the use of deep networks in pursuit of high segmentation accuracy leads to a dramatic increase in the number of model parameters and computational complexity, making it difficult to achieve real-time deployment on resource-constrained devices such as autonomous vehicle terminals. Even with lightweight backbone networks, traditional feature fusion methods struggle to balance efficiency and accuracy. Second, traffic scenarios exhibit significant differences in target scale (from centimeter-level traffic signs to hundred-meter-level building clusters). Traditional multi-scale feature fusion mechanisms cannot simultaneously preserve details of small targets and represent the structure of large targets, and they lack sufficient modeling of contextual dependencies for rectangular structures such as roads and buildings, making it difficult to capture the spatiotemporal features of traffic flow and limiting the generalization ability across scenarios (such as vehicle images and remote sensing data).

[0004] For example, DeepLabv3+ achieved a mere 0.11 mIoU for Barren class (bare land) segmentation on the LoveDA remote sensing dataset. While CM-UNet improved the overall mIoU to 52.17%, it still exhibited significant shortcomings in small-scale land feature segmentation. Therefore, designing a semantic segmentation network that balances lightweight deployment with cross-scene segmentation accuracy has become a key requirement for overcoming the bottlenecks of existing technologies. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of existing semantic segmentation networks, such as large number of parameters, high computational complexity, and insufficient cross-scenario adaptability, and to provide a lightweight semantic segmentation method that integrates rectangular context calibration. Through innovative module design and architecture optimization, it achieves a balance between efficiency and accuracy, and meets the needs of resource-constrained device deployment and multi-scenario applications.

[0006] Technical solution: This invention discloses a lightweight semantic segmentation method that integrates rectangular context calibration, comprising the following steps:

[0007] Step 1: Multi-scene image data acquisition and preprocessing, constructing a semantic segmentation dataset, and dividing it into training and validation sets;

[0008] Step 2: Based on PSPNet, a lightweight semantic segmentation network, TransitNet, is constructed. The lightweight semantic segmentation network TransitNet integrates the rectangular context calibration attention (RCCA) module, the bidirectional fusion attention (BFA) module, and the StarNet backbone network, replacing the PSPNet backbone.

[0009] Step 3: Train the lightweight semantic segmentation network TransitNet, and use the trained lightweight semantic segmentation network TransitNet to perform semantic segmentation of multi-scene image data.

[0010] Furthermore, in step 1, the dataset collects image data for three core application scenarios: conventional vision, remote sensing, and distortion.

[0011] Typical visual scenarios: Collect images containing pedestrians, vehicles, buildings, and traffic signs, covering urban roads, residential areas, and commercial areas;

[0012] Remote sensing scene: High-resolution images of buildings, roads, and water bodies are acquired through satellite remote sensing, covering urban areas, suburbs, and rural areas;

[0013] Distortion scene: Collect roads, obstacles, and small-scale traffic signs with different degrees of distortion to simulate extreme perspectives.

[0014] Furthermore, when the Rectangular Context Calibration Attention (RCCA) module and the Bidirectional Fusion Attention (BFA) module are integrated with the StarNet backbone network, the RCCA module and the BFA module are connected in series after the StarNet backbone network. The features processed by the BFA module are then input into the basic modules of PSPNet for further processing.

[0015] Furthermore, the StarNet backbone network includes multiple Star Blocks structures connected in series, and the selection and optimization of the Star Blocks structures specifically includes:

[0016] 1) Introduce a star topology to the StarNet backbone network. The mathematical expression is:

[0017] ;

[0018] in, , where is the input feature and d is the feature dimension. ; , is the weight matrix. , from feature dimension Expand to The computational complexity is only ;

[0019] 2) Improved forward propagation path: In the forward propagation path, the high-level semantic features output by the 4th Star Blocks are upsampled to a 1 / 4 scale, and channel-joined with the detailed features of the shallow branches. After multiple rounds of skip fusion, a multi-level feature pyramid is constructed.

[0020] Furthermore, the rectangular context calibration attention (RCCA) module specifically comprises:

[0021] Contextual information aggregation: Horizontal pooling is used to aggregate road extension context along the row direction, and vertical pooling is used to capture the rectangular constraint features of building outlines along the column direction;

[0022] Adaptive convolution kernel application: 1×7 and 7×1 strip convolution kernels are used to adapt to linear structure targets, and 3×3 depthwise separable convolution is used to extract local details;

[0023] Dynamic feature fusion: A gated fusion mechanism is introduced to dynamically fuse the calibrated attention features with the original features, thereby enhancing the features of the rectangular target structure and suppressing irrelevant background interference.

[0024] Furthermore, the rectangular context calibration attention (RCCA) module enhances the rectangular structural features through the following process:

[0025] First, rectangular calibration attention RCA weights are generated. The RCA submodule analyzes the row and column dimensions of the feature map, calculates the feature response intensity of the rectangular structure target, generates a weight map using the Sigmoid activation function, initially filters rectangular structure regions, performs batch normalization (BN), and after processing by two fully connected layers, features are fused and output using Gamma. A learnable Gamma coefficient is introduced, initially set to 0.5, and the fusion formula is... , For output features, For attention features, The original features are used; the Gamma coefficient is adaptively adjusted with the segmentation loss to enhance the long-range dependence of the rectangular structure and suppress background interference.

[0026] Furthermore, the bidirectional fusion attention (BFA) module specifically comprises:

[0027] Preliminary feature encoding and channel compression: The number of channels is reduced to 1 / 4 of the original dimension through 1×1 shared convolution;

[0028] Multi-scale spatial attention branch construction: Set up 5×5, 7×7, and 9×9 convolutional kernel branches to adapt to small, medium, and large-scale target feature extraction, respectively;

[0029] Channel attention and feature fusion: Combining a channel attention module with L2 regularization, features are compressed into channel-level vectors through global average pooling (GAP), and channel weights are calculated by a multilayer perceptron (MLP). After feature concatenation, a 1×1 fusion convolution is used to achieve "space-channel" interaction, and then residual connections are used to weightedly fuse with the original features to preserve details of small targets.

[0030] Furthermore, the channel attention calculation process for the BFA module is as follows:

[0031] 1) Channel-level vector compression: concatenating 5×5, 7×7, and 9×9 branch features, size The average value is calculated along the H and W dimensions using GAP, and then compressed to... vector ;

[0032] 2) MLP calculates channel weights: MLP contains 2 fully connected layers: , Dimensional reduction ReLU activation; , Restore the dimension, obtain the weight w by Sigmoid activation; apply L2 regularization to the MLP weights;

[0033] 3) Channel feature optimization application: Channel weights are multiplied by multi-scale spatial features.

[0034] Furthermore, the preprocessing includes scaling all images to 473×473 pixels and cleaning the images by removing duplicate, blurred, and labeled abnormal images, ultimately forming a semantic segmentation dataset covering multiple scenarios.

[0035] Beneficial effects:

[0036] This invention proposes a lightweight scene segmentation method based on rectangular context calibration. The core improvements are as follows: A rectangular context calibration attention (RCCA) module is designed and integrated with feature modeling. Road and building-related contexts are aggregated through horizontal / vertical pooling, combined with 1×7 / 7×1 elongated convolutional kernels and 3×3 depthwise separable convolutions to accurately capture long-distance dependencies of rectangular structures, enhancing geometric feature modeling and suppressing background interference. The traditional unidirectional feature fusion mechanism is replaced with a bidirectional fusion attention (BFA) module. After initial encoding with 1×1 shared convolutions, 5×5, 7×7, and 9×9 convolutional kernel branches are used to adapt to multi-scale targets. Combined with channel attention containing L2 regularization, a "space-channel" interaction channel is constructed to improve the completeness and accuracy of feature fusion while preserving details of small targets. An optimized StarNet is selected to replace the original PSPNet backbone, efficiently transforming feature dimensions through star-shaped operations, combined with 7×7 depthwise convolutions and 1×1... Pointwise convolution reduces costs and increases efficiency, improves forward propagation to output multi-scale features and form a feature pyramid, adapting to targets in multiple scenarios; during the inference stage, TransitNet segmented images are loaded, and only high-confidence results are used to ensure application reliability.

[0037] Compared to other semantic segmentation algorithms, this method can stably adapt to scenarios such as vehicle-mounted images, remote sensing data, and distorted perspectives, accurately segmenting key semantic regions. Its lightweight design makes it suitable for resource-constrained platforms such as vehicle terminals and edge devices, and applicable to tasks such as environmental perception and remote sensing ground object classification, demonstrating effectiveness and feasibility in balancing segmentation accuracy and computational efficiency. Attached Figure Description

[0038] Figure 1 This is a flowchart illustrating the overall architecture of the TransitNet semantic segmentation network proposed in this invention.

[0039] Figure 2 This is a flowchart illustrating the workflow of the Rectangular Context Calibration Attention (RCCA) module of this invention.

[0040] Figure 3 This is an architectural diagram of the bidirectional fusion attention (BFA) module of the present invention;

[0041] Figure 4 This is a comparison chart of the segmentation performance of TransitNet, Deeplabv3+, ExFuse, and SpineNet on the VOC2012 dataset.

[0042] Figure 5 This image shows a comparison of the land cover segmentation performance of TransitNet and traditional semantic segmentation models on the LoveAD remote sensing dataset.

[0043] Figure 6The image shows the semantic segmentation performance of TransitNet on the OST300 distortion dataset.

[0044] Figure 7 This is an overview diagram of the technical solution of TransitNet of the present invention;

[0045] Figure 8 A visualization tool interface diagram of TransitNet on the VOC2012 dataset;

[0046] Figure 9 A diagram of the visualization tool interface for TransitNet on the LoveAD remote sensing dataset;

[0047] Figure 10 This is a screenshot of the visualization tool interface for TransitNet on the OST300 distortion dataset. Detailed Implementation

[0048] The following detailed description, with reference to the accompanying drawings, further illustrates a specific implementation of the lightweight semantic segmentation method incorporating rectangular context calibration according to the present invention. This implementation covers the entire process of data acquisition and preprocessing, model building and improvement, model training and validation, and inference deployment, ensuring the technical solution is reproducible and meets performance standards. The specific steps are as follows:

[0049] Step 1: Multi-scene image data acquisition and preprocessing

[0050] To construct a semantic segmentation dataset covering three types of scenarios: conventional vision, remote sensing, and distortion, image data was collected from multiple channels and standardized preprocessing was performed to ensure data quality and consistency. The specific operations are as follows:

[0051] 1. Data collection channels

[0052] Typical visual scenarios: The PASCAL VOC2012 public dataset was used, which includes 11,530 training images, 10,980 validation images, and 14,499 test images, covering 21 categories of everyday objects such as pedestrians, vehicles, buildings, and traffic signs. The image resolution ranges from 300×300 to 1000×1000 pixels, and the annotation accuracy reaches the pixel level, which meets the requirements for verifying the basic segmentation capabilities of the model under normal conditions.

[0053] Remote sensing scene: The LoveAD remote sensing public dataset is selected, which contains thousands of 1024×1024 pixel high-resolution satellite images, labeled with 7 major land cover categories: buildings, roads, water bodies, bare land, farmland, forest, and agricultural land. The data covers diverse landforms such as urban areas, suburbs, and rural areas, and is suitable for the task of fine classification of roads and buildings in remote sensing images.

[0054] Distortion scenarios: The OST300 public dataset was used, which integrates unconventional visual data such as drone aerial photography, vehicle-mounted fisheye lenses, and industrial monitoring. It contains more than 5,000 images with different degrees of distortion (fisheye distortion, perspective distortion), labeled with targets such as roads, obstacles, and small-scale traffic signs, to verify the generalization performance of the model under extreme viewpoints.

[0055] 2. Data Preprocessing Operations

[0056] Standardization: All images from the three datasets were scaled to 473×473 pixels using bilinear interpolation to fit the model input size; the RGB three channels were standardized with a mean of [0.485, 0.456, 0.406] and a standard deviation of [0.229, 0.224, 0.225] to eliminate differences in pixel value distribution between different datasets.

[0057] Training data augmentation: Data augmentation is performed on images only during the training phase, including random horizontal flipping (probability 0.5), random rotation (angle range ±15°), and brightness / contrast adjustment (amplitude ±0.2). The original annotation information is not changed, and the robustness of the model is improved only through image transformation.

[0058] Step 2: Dataset Partitioning

[0059] Stratified sampling is used to divide the training and validation sets to avoid class distribution bias. The specific process is as follows:

[0060] The training and validation sets are divided in a 9:1 ratio, and stratified sampling is used to ensure that the sample proportion of each target class in each scenario dataset is consistent with that in the original dataset.

[0061] After partitioning, a dataset index file is generated, recording the path, category label, and set (training / validation) of each image, which facilitates batch reading of data during model training.

[0062] Step 3: TransitNet Model Construction and Improvement

[0063] Based on PSPNet, the TransitNet model is constructed by "lightweight backbone replacement + core attention module embedding", which solves the problems of large number of parameters and insufficient multi-scale feature fusion in traditional models. The specific improvement measures are as follows:

[0064] 1. Lightweight backbone network (StarNet) optimization

[0065] Star operation integration: Star operations are introduced into the Star Blocks structure of StarNet. The mathematical expression is as follows:

[0066]

[0067] Among them, the input feature vector Weight matrix (Pick ), where d is the feature dimension. The feature dimension is reduced from element-wise multiplication. Expand to The computational complexity is only It reduces the number of parameters by 75% compared to traditional fully connected layers, while expanding the ability to express features and avoiding a surge in the number of parameters.

[0068] Forward propagation improvement: In stage 4, bilinear interpolation upsampling is used to generate a high-level semantic feature map with a resolution of 1 / 4 of the input, containing high-level information such as road category and building semantics, forming a dual-branch feature output to support subsequent multi-scale fusion. This dual-branch design... Figure 1 This is reflected in the text.

[0069] 2. Embedding of the Rectangular Context Calibration Attention (RCCA) module

[0070] Module structure design: The RCCA module consists of four parts: horizontal pooling, vertical pooling, strip convolutional kernels, and gating fusion (e.g., ...). Figure 2 (See the RCCA module flowchart). Horizontal pooling aggregates global contextual information along the road extension direction in the feature map row direction, while vertical pooling captures the rectangular constraint features of building outlines along the column direction. 1×7 and 7×1 strip convolution kernels are used to adapt to linear / rectangular structural targets such as roads and buildings. Combined with 3×3 depthwise separable convolution (first convolving each channel independently, then fusing channel information through 1×1 convolution) to extract local details and reduce parameter redundancy.

[0071] Feature calibration process: Input features are first processed by Rectangular Calibration Attention (RCA) to generate attention weights related to the rectangular structure. These weights are then adjusted sequentially through batch normalization (BN, momentum 0.9) and a two-layer MLP (hidden layer dimension is 1 / 4 of the input channel). Finally, they are weighted and fused with the original features under the influence of the Gamma fusion coefficient (a learnable parameter, initial value 0.5), outputting features that enhance the long-range dependence of the rectangular structure and effectively suppress interference from irrelevant backgrounds such as vegetation and water bodies. Experiments show that this module can improve road boundary segmentation accuracy by 12.3%.

[0072] 3. Replacement of the Bidirectional Fusion Attention (BFA) module

[0073] Module architecture design: The BFA module starts with a 1×1 shared convolution (e.g., Figure 3As shown in the BFA module architecture diagram, the input features undergo channel compression (reducing the number of channels to 1 / 4 of the original dimension) and initial encoding; spatial attention branches with three convolutional kernels (5×5, 7×7, and 9×9) are set to adapt to the extraction of target features at small scales (traffic signs, bare land patches), medium scales (vehicles, buildings), and large scales (roads, building complexes), respectively; combined with a channel attention module with L2 regularization (regularity coefficient 0.001), multi-scale spatial features are compressed into channel-level vectors through global average pooling (GAP). Then, the MLP calculates the channel weights, and the MLP output formula is:

[0074]

[0075] in, It is the Sigmoid activation function. , For linear transformation weights, , This is a bias term.

[0076] Feature fusion mechanism: Multi-branch spatial features and channel weights are fused by concatenating them along the channel dimension, followed by a 1×1 fusion convolution (compressing the number of channels to a minimum). After processing with BN (momentum 0.9) and ReLU activation, the original input features are adaptively weighted and fused with them through residual connections (formula: , (Attention weights generated for Sigmoid) preserve fine-grained details of small targets. Ablation experiments show that this module can improve the mIoU of small target segmentation by 8.7%.

[0077] Step 4: TransitNet model training

[0078] Under the specified hardware and software environment, set the training hyperparameters, train the TransitNet model using the training set partitioned in step 2, monitor key metrics in real time, and save the optimal model. The specific process is as follows:

[0079] 1. Experimental Environment Configuration

[0080] Hardware platform: It adopts an Alibaba Cloud server equipped with an NVIDIA A10 GPU (48GB HBM2 memory, 19.5 TFLOPS single-precision computing power), a 10-core CPU (3.2GHz), and 30GB of system memory to ensure that the memory usage does not exceed 40GB during training and to avoid memory overflow.

[0081] Software framework: Developed based on Python 3.9, configured with the PyTorch 1.12.1 deep learning framework, and accelerated by CUDA 11.8 and cuDNN 8.6; TensorBoard is used to monitor the loss function, mIoU (mean intersection-over-union ratio), and MPA (mean pixel accuracy) metrics in real time, and a visual log is generated every 10 iterations.

[0082] 2. Training Hyperparameter Settings

[0083] Optimizer and learning rate: The SGD optimizer is used with an initial learning rate of 0.01, momentum of 0.9, and weight decay of 0.0005 (to suppress overfitting). The learning rate is dynamically adjusted through a cosine annealing strategy, and the learning rate is decayed to 0.9 times the current value every 10 iterations. In the later stage of training (after 150 rounds), the minimum learning rate of 1e-5 is maintained.

[0084] Training batches and epochs: BatchSize is set to 16 (to adapt to GPU memory), and the total number of training epochs is 200. Every 10 iterations, model checkpoints are saved, including information such as the current epoch, loss value, mIoU, and model weights. After training, the model with the highest mIoU on the validation set is selected as the optimal model, and the remaining redundant checkpoints are deleted.

[0085] Loss function: A weighted combination of cross-entropy loss (CE Loss) and Dice loss (weight ratio 1:1) is used. Cross-entropy loss addresses the class imbalance problem, while Dice loss improves the segmentation accuracy of small objects. The formula is as follows:

[0086]

[0087] in For the model to predict features, For accurate labeling, .

[0088] 3. Monitoring the training process.

[0089] Every 10 iterations, the model performance is tested on the validation set, and metrics such as mIoU, MPA, and IoU for each category are recorded. If the mIoU on the validation set does not improve for 10 consecutive iterations, an early stopping mechanism is triggered to avoid overfitting. After training, a training log file is generated, containing the loss curve and the mIoU convergence curve. The convergence target must meet the mIoU requirements of the "Performance Test" section in Figure 7 (VOC2012>86.5%, LoveDA>60%, OST300≥75%), to facilitate subsequent performance analysis.

[0090] Step 5: Model Inference and Performance Verification

[0091] Load the optimal model weights saved in step 4, execute the inference process on the image to be segmented, and verify the model performance through quantitative indicators and qualitative visualization. The specific operations are as follows:

[0092] 1. Reasoning process design

[0093] The image to be segmented first undergoes the same preprocessing as the training data: scaling to 473×473 pixels and RGB three-channel normalization. After being input into the TransitNet model, multi-scale features are first extracted through the StarNet backbone, then the rectangular structure target response is enhanced through the RCCA module, and then the bidirectional fusion optimization of features is completed through the BFA module. The fused features are then input into the basic module of PSPNet for further processing, and finally the pixel-level semantic segmentation result is output. This inference process is the basic step of the "Deployment Application Layer" in Figure 7.

[0094] A threshold method (confidence level 0.85) was used to filter effective segmentation regions. Pixels below the threshold were labeled as "background", and pixels above the threshold were classified according to the highest probability. The segmentation results were overlaid with the original image using OpenCV to draw category labels (such as "road", "building", "traffic sign") and segmentation boundaries (red lines, 1 pixel wide) for easy and intuitive viewing.

[0095] 2. Performance Verification Methods

[0096] Quantitative evaluation metrics: Calculate metrics such as mIoU, MPA, Precision, and Accuracy on the three datasets. The results must meet the requirements of "Performance Testing" in Figure 7, as follows:

[0097] Table 1. Comparison of Model Parameter Quantities

[0098]

[0099] StarNet_s1, StarNet_s2, StarNet_s3, and StarNet_s4 are lightweight backbone network versions of StarNet with different complexities. They are gradient versions formed by adjusting core parameters such as network depth and number of channels on the basic StarNet architecture to adapt to deployment scenarios and accuracy requirements with different computing resources. They are also the core backbone of the TransitNet model in this invention. Each version increases in the number of parameters and computational cost (GFLOPS) in a gradient manner, and the feature extraction capability is gradually enhanced.

[0100] Embedding Dimensions and Network Depth:

[0101] StarNet-s1: Embedding width is set to 24, and depth is configured as [2,2,8,3];

[0102] StarNet-s2: Embedding width increased to 32, depth configured as [1,2,6,2];

[0103] StarNet-s3: The embedding width remains unchanged at 32, and the depth is adjusted to [2,2,8,4];

[0104] StarNet-s4: The embedding width remains unchanged at 32, and the depth is configured as [3,3,12,15].

[0105] Table 2. Comparison of segmentation class mIoU between the LoveDA dataset and other models

[0106]

[0107] Table 3. Ablation Experiment Results of PASCAL VOC2012 Dataset

[0108]

[0109] Table 4. Ablation Experiment Results of PASCAL VOC2012 Dataset

[0110]

[0111] Table 5. Ablation Experiment Results on OST300 Dataset

[0112]

[0113] Table 6. Ablation Experiment Results of LoveAD Dataset

[0114]

[0115] The segmentation results are visualized and compared (as shown in Figure 4, the segmentation effect comparison of the VOC2012 dataset, Figure 5, the segmentation effect of the LoveAD dataset, and Figure 6, the segmentation effect of the OST300 dataset).

[0116] Step 6: Model Deployment and Application

[0117] The TransitNet model, having passed performance verification, is deployed to target devices to adapt to scenarios such as autonomous driving vehicle terminals and edge computing devices, ensuring real-time performance and practicality. The specific deployment scheme is as follows, and the entire process is shown in Figure 7:

[0118] As required by the "Terminal Adaptation" step in Figure 7, the optimal model weights in PyTorch format are converted to ONNX format, and the inference process is optimized through ONNX Runtime. For embedded devices (such as Jetson Xavier), TensorRT is used for model quantization (INT8 quantization) and layer fusion, which improves the inference speed by 2-3 times and further reduces the number of parameters by 30%, ensuring the running efficiency on resource-constrained devices.

[0119] The model is pruned to remove redundant channels (keeping the top 80% of channels by absolute weight). Without losing more than 1% mIoU, GFLOPS is reduced by 15%. For example, after pruning StarNet_s4, GFLOPS is reduced from 9.873G to 8.392G, which meets the requirements for lightweight deployment.

[0120] 2. Visualization and Monitoring

[0121] Develop a web-based visualization interface and scenario-based visualization tools to accurately present the results of multi-dataset segmentation and enable real-time monitoring of system operation status. The specific functional design is as follows:

[0122] VOC2012 Regular Scene Visualization: Through the visualization tool interface shown in Figure 8, the segmentation results of regular visual scenes are loaded. The interface supports dual-window comparison display of the original image and the segmentation results. It automatically counts the pixel proportion and quantity of 21 types of targets such as pedestrians, vehicles, and buildings (e.g., in a certain frame image, the vehicle target accounts for 15% and the building target accounts for 22%), and marks the segmentation boundaries of small targets such as traffic signs, intuitively verifying the model's ability to preserve details in regular scenes.

[0123] LoveAD Remote Sensing Scene Visualization: Adapting to remote sensing data display needs in the tool interface shown in Figure 9, it uses differentiated color coding (e.g., red for roads, blue for buildings, and yellow for bare land) for 7 major land cover categories, clearly presenting the segmentation effect of rectangular structural targets in remote sensing images; it supports zooming to view the boundary details of the overlapping areas of bare land and buildings, and simultaneously displays the mIoU values ​​of each category (e.g., mIoU 39% for bare land and 50% for roads), quantitatively verifying the model's classification accuracy of remote sensing land cover.

[0124] OST300 Distortion Scene Visualization: Using the tool interface in Figure 10, the segmentation results under fisheye distortion and perspective distortion scenes are highlighted. The interface adds a "Distortion Correction Comparison" function, which overlays the model segmentation results with the distortion-corrected reference image to intuitively present the restoration degree of the road topology. The number of obstacles and small-scale signs identified in the distortion area is automatically counted, and the road category IoU is recorded (e.g., the road IoU is 78.2% in the fisheye distortion scene) to verify the robustness of the model under extreme viewpoints.

[0125] This invention, through the aforementioned steps, realizes the construction, training, and deployment of TransitNet, a lightweight semantic segmentation network that integrates rectangular context calibration. While maintaining segmentation accuracy, it reduces the number of model parameters and computational complexity, adapting to the needs of various application scenarios. Experimental and deployment results show that this method exhibits excellent performance in conventional vision, remote sensing, and distortion scenarios, providing an efficient solution for tasks such as autonomous driving environmental perception and remote sensing image interpretation.

Claims

1. A lightweight semantic segmentation method incorporating rectangular context calibration, characterized in that, Includes the following steps: Step 1: Multi-scene image data acquisition and preprocessing, constructing a semantic segmentation dataset, and dividing it into training and validation sets; Step 2: Based on PSPNet, a lightweight semantic segmentation network, TransitNet, is constructed. The lightweight semantic segmentation network TransitNet integrates the rectangular context calibration attention (RCCA) module, the bidirectional fusion attention (BFA) module, and the StarNet backbone network, replacing the PSPNet backbone. Step 3: Train the lightweight semantic segmentation network TransitNet, and use the trained lightweight semantic segmentation network TransitNet to perform semantic segmentation of multi-scene image data.

2. The lightweight semantic segmentation method with fused rectangular context calibration according to claim 1, characterized in that, In step 1, the dataset collects image data for three core application scenarios: conventional vision, remote sensing, and distortion. Typical visual scenarios: Collect images containing pedestrians, vehicles, buildings, and traffic signs, covering urban roads, residential areas, and commercial areas; Remote sensing scene: High-resolution images of buildings, roads, and water bodies are acquired through satellite remote sensing, covering urban areas, suburbs, and rural areas; Distortion scene: Collect roads, obstacles, and small-scale traffic signs with different degrees of distortion to simulate extreme perspectives.

3. The lightweight semantic segmentation method with fused rectangular context calibration according to claim 1, characterized in that, When the Rectangular Context Calibration Attention (RCCA) module and the Bidirectional Fusion Attention (BFA) module are integrated with the StarNet backbone network, the RCCA module and the BFA module are connected in series after the StarNet backbone network. The features processed by the BFA module are then fed into the basic modules of PSPNet for further processing.

4. The lightweight semantic segmentation method with fused rectangular context calibration according to claim 1, characterized in that, The StarNet backbone network comprises multiple Star Blocks structures connected in series. The selection and optimization of the Star Blocks structures specifically include: 1) Introduce a star topology to the StarNet backbone network. The mathematical expression is: ； in, , where is the input feature and d is the feature dimension. ; , is the weight matrix. , from feature dimension Expand to The computational complexity is only ; 2) Improved forward propagation path: In the forward propagation path, the high-level semantic features output by the 4th Star Blocks are upsampled to a 1 / 4 scale, and channel-joined with the detailed features of the shallow branches. After multiple rounds of skip fusion, a multi-level feature pyramid is constructed.

5. A lightweight semantic segmentation method incorporating rectangular context calibration according to claim 1, characterized in that, The rectangular context calibration attention (RCCA) module is specifically as follows: Contextual information aggregation: Horizontal pooling is used to aggregate road extension context along the row direction, and vertical pooling is used to capture the rectangular constraint features of building outlines along the column direction; Adaptive convolution kernel application: 1×7 and 7×1 strip convolution kernels are used to adapt to linear structure targets, and 3×3 depthwise separable convolution is used to extract local details; Dynamic feature fusion: A gated fusion mechanism is introduced to dynamically fuse the calibrated attention features with the original features, thereby enhancing the features of the rectangular target structure and suppressing irrelevant background interference.

6. A lightweight semantic segmentation method incorporating rectangular context calibration according to claim 5, characterized in that, The rectangular context calibration attention (RCCA) module enhances the rectangular structural features through the following process: First, rectangular calibration attention RCA weights are generated. The RCA submodule analyzes the row and column dimensions of the feature map, calculates the feature response intensity of the rectangular structure target, generates a weight map using the Sigmoid activation function, initially screens rectangular structure regions, performs batch normalization (BN), processes the data through two fully connected layers, and then fuses and outputs the features using Gamma. Introduction The learnable Gamma coefficient, initially set to 0.5, is used in the fusion formula. , For output features, For attention features, Original features; The Gamma coefficient adaptively adjusts with the segmentation loss, enhancing the long-range dependence of the rectangular structure and suppressing background interference.

7. The lightweight semantic segmentation method with fused rectangular context calibration according to claim 1, characterized in that, The bidirectional fusion attention (BFA) module is specifically as follows: Preliminary feature encoding and channel compression: The number of channels is reduced to 1 / 4 of the original dimension through 1×1 shared convolution; Multi-scale spatial attention branch construction: Set up 5×5, 7×7, and 9×9 convolutional kernel branches to adapt to small, medium, and large-scale target feature extraction, respectively; Channel attention and feature fusion: Combining a channel attention module with L2 regularization, features are compressed into channel-level vectors through global average pooling (GAP), and channel weights are calculated by a multilayer perceptron (MLP). After feature concatenation, "space-channel" interaction is achieved through 1×1 fusion convolution, and then the residual connection is used to weighted fuse with the original features to preserve the details of small targets.

8. A lightweight semantic segmentation method incorporating rectangular context calibration according to claim 7, characterized in that, The BFA module channel attention calculation process is as follows: 1) Channel-level vector compression: concatenating 5×5, 7×7, and 9×9 branch features, size After averaging along the H and W dimensions using GAP, it is compressed to vector ; 2) MLP calculates channel weights: MLP contains 2 fully connected layers: , Dimensional reduction ReLU activation; , Restore the dimension, obtain the weight w by Sigmoid activation; apply L2 regularization to the MLP weights; 3) Channel feature optimization application: Channel weights are multiplied by multi-scale spatial features.

9. A lightweight semantic segmentation method incorporating rectangular context calibration according to claim 1, characterized in that, The preprocessing includes scaling all images to 473×473 pixels and cleaning the images by removing duplicate, blurred, and labeled abnormal images, ultimately forming a semantic segmentation dataset covering multiple scenarios.

Images