Panoramic image semantic segmentation method combined with lightweight segmentation all-in-one model
By constructing a dual-branch architecture and a lightweight improved panoramic image segmentation method, the problem of insufficient segmentation capability for small targets in panoramic image segmentation, which makes it difficult to balance semantic information and boundary accuracy, is solved, and efficient panoramic image segmentation results are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- XIANGTAN UNIV
- Filing Date
- 2026-01-27
- Publication Date
- 2026-05-15
AI Technical Summary
Existing panoramic image segmentation technologies suffer from difficulties in balancing semantic information and boundary accuracy, insufficient segmentation capability for small targets, and high model deployment costs.
A dual-branch architecture of mask branch and semantic branch is constructed. A segmentation model is introduced and lightweighted. A decoupled distillation strategy and multi-scale attention mechanism are adopted. A composite loss function is designed by combining parallel convolution branch and cross-space learning strategy to improve the training stability and segmentation accuracy of the model. The results of the dual branches are then fused through a semantic voting module.
It achieves panoramic image segmentation with clear boundaries and semantic accuracy, reduces the computational complexity and resource consumption of the model, and improves the ability to recognize small targets and the segmentation accuracy.
Smart Images
Figure CN122049359A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer vision technology, specifically relating to a panoramic image semantic segmentation method, and more particularly to a panoramic image segmentation method that combines a lightweight segmentation model. It can be widely applied to technical scenarios that require high-precision panoramic image semantic segmentation, such as autonomous driving, intelligent monitoring, virtual reality, and medical image analysis. Background Technology
[0002] Panoramic images are images that cover all directions of a physical space, offering a wider field of view and capturing more complete information compared to general perspective images. Therefore, panoramic images play a crucial role in applications requiring rich information about the surrounding environment. Furthermore, panoramic images effectively compensate for the limitations of sensors with specific viewpoints in capturing overall environmental features.
[0003] Semantic segmentation is a fundamental task in computer vision, aiming to classify each pixel in an image and assign it to a predefined semantic category. The evolution of image segmentation techniques has transitioned from traditional methods to the deep learning paradigm. Early traditional methods relied on manually designed feature extraction and region partitioning strategies, which struggled to adapt to semantic variations in complex scenes. With the rise of deep learning, encoder-decoder architectures, exemplified by U-Net, achieved significant progress in medical image segmentation through skip connections to realize multi-scale feature fusion. Following the success of the Transformer architecture in natural language processing, its core self-attention mechanism was introduced into image segmentation tasks, further propelling the field's development. However, most of these methods are applied to planar perspective images. When used for panoramic images, the inherent distortions of panoramic images and complex scenes with large scale differences lead to a significant decrease in accuracy, while extremely high resolutions result in increased computational resource consumption. Summary of the Invention
[0004] To address the challenges of balancing semantic information and boundary accuracy, insufficient segmentation capability for small targets, and high model deployment costs in existing panoramic image segmentation techniques, this invention aims to provide a panoramic image semantic segmentation method that combines a lightweight all-encompassing segmentation model. This method constructs a dual-branch architecture with a mask branch and a semantic branch. The mask branch incorporates an all-encompassing segmentation model with lightweight improvements, employing a decoupling distillation strategy to efficiently inherit the semantic representation capabilities of the teacher model. By reconstructing the encoder structure, a multi-scale attention mechanism and a multi-level feature fusion module are introduced. Parallel convolutional branches and cross-space learning strategies enhance the model's ability to model contextual dependencies and local details. A composite loss function incorporating feature alignment, boundary awareness, and structure preservation improves training stability and segmentation accuracy. The semantic branch introduces a target detection method to enhance the recognition of small targets. Finally, a semantic voting module fuses the results from both branches, resulting in a panoramic image segmentation method with accurate semantics and clear boundaries.
[0005] To achieve the above objectives, this invention proposes a panoramic image semantic segmentation method that combines a lightweight segmentation model, comprising the following steps:
[0006] A panoramic image segmentation method combining a lightweight segmentation model is characterized by the following steps:
[0007] S1: Construct a panoramic image dataset, divide the panoramic image dataset into a training set and a test set; it contains panoramic image samples under different scenes and shooting angles, and each sample is labeled with a corresponding semantic label for model training and performance verification.
[0008] S2: Pre-divide all semantic labels into small target label sets and large target label sets, keeping the original panoramic image files unchanged, and only splitting the annotation files. For the original annotation files of a single image, generate annotation files containing only small target labels and annotation files containing only large target labels respectively. Then pair the original image with the two types of annotation files to obtain small target datasets and large target datasets.
[0009] S3: The training set of the large target is put into the semantic segmentation model for training, and the test set is put into the semantic segmentation model for segmentation task to obtain the semantic segmentation result of the large target;
[0010] S4: Input the small target training set into the YOLOv8 model for training, input the test set into the YOLOv8 model for target detection task, extract the detection box regions to form a new small target dataset, and then input the dataset into the semantic segmentation model for training to obtain the semantic segmentation results of the small targets.
[0011] S5: The semantic segmentation results of small and large targets are fused with the coordinate information of the previously extracted detection boxes when forming a new dataset to obtain an initial semantic segmentation result containing complete semantic information but with coarse boundaries.
[0012] S6: Introduce a decoupled distillation strategy to reduce the computational complexity and resource consumption of the model, and achieve lightweight deployment of the model; embed an efficient multi-scale attention mechanism and a multi-level feature fusion module into the lightweight encoder of the model student network to improve the encoder's ability to extract multi-scale features of panoramic images and the feature utilization rate; replace the original distillation loss function with a composite loss function to optimize the training effect of the model and improve the accuracy of model segmentation.
[0013] S7: Train the improved all-in-one segmentation model by feeding the training set into the improved all-in-one segmentation model, and perform segmentation tasks by feeding the test set into the improved all-in-one segmentation model to obtain segmentation results with clear boundaries but without semantic information, and evaluate the performance of the improved all-in-one segmentation model.
[0014] S8: Extract semantic label information from the semantic segmentation result obtained in S5. Based on the mask position information in the segmentation result obtained in S7, perform a semantic voting operation on the semantic labels within each mask range. Determine the semantic label with the highest proportion within the mask range as the final classification label of the mask region. Finally, output a panoramic image semantic segmentation result with clear boundaries and accurate semantics.
[0015] The semantic segmentation models used in S3 and S4 can be replaced according to the actual situation.
[0016] In step 4:
[0017] Small targets are identified by an object detection model. Then, a new dataset is created by cutting out the corresponding regions from the original image and the annotation file based on the detection boxes and extracting the annotation information of the corresponding regions. Then, the new dataset is used for semantic segmentation training and segmentation tasks to obtain the segmentation results of the new dataset.
[0018] In step 6:
[0019] (1) Lightweight Student Model Encoder: An efficient multi-scale attention mechanism is introduced, employing parallel convolutional branches to capture spatial context information at different scales. In the output fusion stage, a cross-spatial learning mechanism is used to fuse features from the parallel branches. Separable convolution and other techniques control complexity growth. A feature preprocessing module is built within the student model encoder architecture to handle multi-scale features: first, small convolutional layers compress and integrate the input features, followed by normalization, reducing computational requirements while preserving the spatial resolution of the feature maps. The EMA module is deployed after the preprocessing module to handle multi-scale features, thereby enhancing the model's ability to extract features from small targets and improving segmentation performance in complex environments. Finally, a feature fusion module is constructed, concatenating the multi-scale features processed by EMA along the channel dimension, and then fusing cross-channel information through convolutional layers.
[0020] (2) Composite Loss Function: The training loss function is replaced with the designed composite loss function, the expression of which is:
[0021]
[0022]
[0023]
[0024]
[0025] in For orientation alignment loss, For robust amplitude loss, For energy alignment loss, superscript This represents the j-th feature element of the i-th sample. , , The hyperparameter is used to adjust the relative weights of each loss term. N represents the batch size, and D = C×H×W, where C represents the number of channels, and H and W represent the spatial dimensions. Attached Figure Description
[0026] Figure 1 This is a flowchart of the panoramic image semantic segmentation method of the present invention, which combines a lightweight segmentation model.
[0027] Figure 2 This is a flowchart of the lightweight segmentation of all models in the mask branch of this invention;
[0028] Figure 3 This is a structural diagram of the efficient multi-scale attention mechanism used by the lightweight segmentation model;
[0029] Figure 4 This is a structural diagram of the improved student model encoder;
[0030] Figure 5 This is a comparison chart of the semantic segmentation results of the present invention and the PP-LiteSeg model (Densepass panoramic image dataset). Detailed Implementation
[0031] See below. Figure 1 , Figure 2 The genetically optimized deep adversarial network super-resolution reconstruction method described in this invention will be explained in detail.
[0032] like Figure 1 As shown, in order to obtain better panoramic image semantic segmentation results, this invention provides a panoramic image semantic segmentation method that combines a lightweight segmentation model, including the following steps:
[0033] S1: Construct a panoramic image dataset, divide the panoramic image dataset into a training set and a test set; it contains panoramic image samples under different scenes and shooting angles, and each sample is labeled with a corresponding semantic label for model training and performance verification.
[0034] S2: Pre-divide all semantic labels into small target label sets and large target label sets, keeping the original panoramic image files unchanged, and only splitting the annotation files. For the original annotation files of a single image, generate annotation files containing only small target labels and annotation files containing only large target labels respectively. Then pair the original image with the two types of annotation files to obtain small target datasets and large target datasets.
[0035] S3: The training set of the large target is put into the semantic segmentation model for training, and the test set is put into the semantic segmentation model for segmentation task to obtain the semantic segmentation result of the large target;
[0036] S4: Input the small target training set into the YOLOv8 model for training, input the test set into the YOLOv8 model for target detection task, extract the detection box regions to form a new small target dataset, and then input the dataset into the semantic segmentation model for training to obtain the semantic segmentation results of the small targets.
[0037] S5: The semantic segmentation results of small and large targets are fused with the coordinate information of the previously extracted detection boxes when forming a new dataset to obtain an initial semantic segmentation result containing complete semantic information but with coarse boundaries.
[0038] S6: Introduce a decoupled distillation strategy to reduce the computational complexity and resource consumption of the model, and achieve lightweight deployment of the model; embed an efficient multi-scale attention mechanism and a multi-level feature fusion module into the lightweight encoder of the model student network to improve the encoder's ability to extract multi-scale features of panoramic images and the feature utilization rate; replace the original distillation loss function with a composite loss function to optimize the training effect of the model and improve the accuracy of model segmentation.
[0039] S7: Train the improved all-in-one segmentation model by feeding the training set into the improved all-in-one segmentation model, and perform segmentation tasks by feeding the test set into the improved all-in-one segmentation model to obtain segmentation results with clear boundaries but without semantic information, and evaluate the performance of the improved all-in-one segmentation model.
[0040] S8: Extract semantic label information from the semantic segmentation result obtained in S5. Based on the mask position information in the segmentation result obtained in S7, perform a semantic voting operation on the semantic labels within each mask range. Determine the semantic label with the highest proportion within the mask range as the final classification label of the mask region. Finally, output a panoramic image semantic segmentation result with clear boundaries and accurate semantics.
[0041] The semantic segmentation models used in S3 and S4 can be replaced according to the actual situation.
[0042] In step 4:
[0043] Small targets are identified by an object detection model. Then, a new dataset is created by cutting out the corresponding regions from the original image and the annotation file based on the detection boxes and extracting the annotation information of the corresponding regions. Then, the new dataset is used for semantic segmentation training and segmentation tasks to obtain the segmentation results of the new dataset.
[0044] In step 6:
[0045] (1) Lightweight Student Model Encoder: An efficient multi-scale attention mechanism is introduced, employing parallel convolutional branches to capture spatial context information at different scales. In the output fusion stage, a cross-spatial learning mechanism is used to fuse features from the parallel branches. Separable convolution and other techniques control complexity growth. A feature preprocessing module is built within the student model encoder architecture to handle multi-scale features: first, small convolutional layers compress and integrate the input features, followed by normalization, reducing computational requirements while preserving the spatial resolution of the feature maps. The EMA module is deployed after the preprocessing module to handle multi-scale features, thereby enhancing the model's ability to extract features from small targets and improving segmentation performance in complex environments. Finally, a feature fusion module is constructed, concatenating the multi-scale features processed by EMA along the channel dimension, and then fusing cross-channel information through convolutional layers.
[0046] (2) Composite Loss Function: The training loss function is replaced with the designed composite loss function, the expression of which is:
[0047]
[0048]
[0049]
[0050]
[0051] in For orientation alignment loss, For robust amplitude loss, For energy alignment loss, superscript This represents the j-th feature element of the i-th sample. , , The hyperparameter is used to adjust the relative weights of each loss term. N represents the batch size, and D = C×H×W, where C represents the number of channels, and H and W represent the spatial dimensions.
[0052] The method of the present invention will be further illustrated below through implementation examples.
[0053] Dataset: DensePASS panoramic image dataset.
[0054] Evaluation metrics: The data uses the mean interaction ratio (mIou) and F1 score. Higher mIou and F1 scores indicate better segmentation quality.
[0055] Implementation steps:
[0056] The method of this invention is used to perform semantic segmentation on the DensePASS panoramic image dataset. The parameters of the method in the experiments are as follows: the semantic segmentation model used in the semantic branch is PP-LiteSeg, with initial learning rate 1e-2, weight decay coefficient 4e-5, batch size 8, and training epochs of 1000; the parameters for YOLOv8 are: initial learning rate 1e-3, weight decay coefficient 5e-4, batch size 4, and training epochs of 500; the improved lightweight segmentation model uses a grid sampling method, with the following parameters: number of sampling points per side set to 32, initial learning rate 1e-3, weight decay coefficient 5e-4, batch size 8, and training epochs of 100.
[0057] For the DensePASS panoramic image dataset, the image semantic segmentation results are as follows: Figure 5As shown, comparing the semantic segmentation results of the proposed method with those of the baseline model PP-LiteSeg, it can be found that the segmentation results of the proposed method are smoother at the edges than those of the baseline model, resulting in better visual segmentation and higher accuracy in super-resolution. To further demonstrate that the optimized framework obtained by the proposed method has good segmentation results, the mIou and F1 scores of the segmentation results were calculated. Figure 5 The comparison results show that the baseline model's semantic segmentation result has a mIou of 0.6756 and an F1 score of 0.7593; the semantic segmentation result of the method of this invention has mIou and F1 scores of 0.7634 and 0.8212, respectively. The baseline model's panoramic image semantic segmentation result has lower mIou and F1 scores than the panoramic image semantic segmentation result obtained using the method of this invention. Therefore, the frame optimized by the method of this invention does indeed have good semantic segmentation performance on panoramic images.
[0058] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A panoramic image segmentation method combining a lightweight segmentation model: characterized in that, Includes the following steps: S1: Construct a panoramic image dataset, divide the panoramic image dataset into a training set and a test set; it contains panoramic image samples under different scenes and shooting angles, and each sample is labeled with a corresponding semantic label for model training and performance verification. S2: Pre-divide all semantic labels into small target label sets and large target label sets, keep the original panoramic image files unchanged, and only split the annotation files; for the original annotation files of a single image, generate annotation files containing only small target labels and annotation files containing only large target labels respectively, and then pair the original image with the two types of annotation files to obtain small target datasets and large target datasets; S3: The training set of the large target is put into the semantic segmentation model for training, and the test set is put into the semantic segmentation model for segmentation task to obtain the semantic segmentation result of the large target; S4: Input the small target training set into the YOLOv8 model for training, input the test set into the YOLOv8 model for target detection task, extract the detection box regions to form a new small target dataset, and then input the dataset into the semantic segmentation model for training to obtain the semantic segmentation results of the small targets. S5: The semantic segmentation results of small and large targets are fused with the coordinate information of the previously extracted detection boxes when forming a new dataset to obtain an initial semantic segmentation result containing complete semantic information but with coarse boundaries. S6: Introduce a decoupled distillation strategy to reduce the computational complexity and resource consumption of the model, and achieve lightweight deployment of the model; embed an efficient multi-scale attention mechanism and a multi-level feature fusion module into the lightweight encoder of the model student network to improve the encoder's ability to extract multi-scale features of panoramic images and the feature utilization rate; replace the original distillation loss function with a composite loss function to optimize the training effect of the model and improve the accuracy of model segmentation. S7: Train the improved all-in-one segmentation model by feeding the training set into the improved all-in-one segmentation model, and perform segmentation tasks by feeding the test set into the improved all-in-one segmentation model to obtain segmentation results with clear boundaries but without semantic information, and evaluate the performance of the improved all-in-one segmentation model. S8: Extract semantic label information from the semantic segmentation result obtained in S5. Based on the mask position information in the segmentation result obtained in S7, perform a semantic voting operation on the semantic labels within each mask range. Determine the semantic label with the highest proportion within the mask range as the final classification label of the mask region. Finally, output a panoramic image semantic segmentation result with clear boundaries and accurate semantics. The semantic segmentation models used in S3 and S4 can be replaced according to the actual situation.
2. The panoramic image segmentation method combining a lightweight segmentation model as described in claim 1, characterized in that: In step 4: Small targets are identified by an object detection model. Then, a new dataset is created by cutting out the corresponding regions from the original image and the annotation file based on the detection boxes and extracting the annotation information of the corresponding regions. Then, semantic segmentation training and segmentation tasks are performed using the new dataset to obtain the segmentation results of the new dataset.
3. The panoramic image segmentation method combining a lightweight segmentation model as described in claim 1, characterized in that: In step 6: (1) Lightweight student model encoder: An efficient multi-scale attention mechanism is introduced, and parallel convolutional branches are used to capture spatial context information at different scales. In the output fusion stage, a cross-space learning mechanism is used to fuse the features of the parallel branches. Techniques such as separable convolution can control the growth of complexity. A feature preprocessing module is built in the student model encoder architecture to handle multi-scale features: First, the input features are compressed and integrated through small convolutional layers, and then normalization is performed to reduce the computational requirements while preserving the spatial resolution of the feature map. The EMA module is deployed after the preprocessing module to process multi-scale features, thereby enhancing the model's ability to extract features of small targets and improving the segmentation performance in complex environments. Finally, a feature fusion module is built to splice the multi-scale features processed by EMA along the channel dimension, and then fuse cross-channel information through convolutional layers. (2) Composite Loss Function: The training loss function is replaced with the designed composite loss function, the expression of which is: in For orientation alignment loss, For robust amplitude loss, For energy alignment loss, superscript This represents the j-th feature element of the i-th sample. , , The hyperparameter is used to adjust the relative weights of each loss term. N represents the batch size, and D = C×H×W, where C represents the number of channels, and H and W represent the spatial dimensions.