Lightweight double-branch panoramic image semantic segmentation method applied to shipping scene

Abstract

The invention discloses a lightweight double-branch panoramic image semantic segmentation method applied to a shipping scene. The method comprises the following steps: acquiring a to-be-segmented panoramic image; the panoramic image to be segmented is input to a double-branch fusion model, a segmentation result is obtained, the double-branch fusion model is obtained through training of a training set, the training set comprises a multi-view shipping image and a corresponding semantic annotation, and the multi-view shipping image and the corresponding semantic annotation are segmented. The double-branch fusion model is obtained by improving multi-scale feature perception and fusion of an encoder-decoder. According to the method, high-precision and high-frame-rate panoramic image semantic segmentation can be realized on the premise of keeping relatively low computing resource consumption, and the method is suitable for intelligent shipping application scenes needing real-time perception, such as unmanned ships and water robots.

Description

Technical Field

[0001] The present invention belongs to the technical field of image semantic segmentation, and in particular relates to a lightweight dual-branch panoramic image semantic segmentation method applied to shipping scenarios. Background Art

[0002] With the development of intelligent ship and unmanned vehicle (USV) technologies, their applications in maritime traffic control, surface monitoring, and marine resource exploration are becoming increasingly widespread. To ensure ships' autonomous navigation capabilities in complex waters and enhance their understanding and perception of their surroundings, semantic segmentation, as a fundamental visual task, is playing an increasingly important role in ship perception systems. Semantic segmentation, by semantically classifying each pixel in an image, enables accurate identification of multiple categories of areas, such as water areas, navigation channels, ice surfaces, buoys, and other vessels. It is a crucial component of building intelligent ship navigation and decision-making systems.

[0003] Early unmanned vessel visual perception systems mostly used monocular cameras for forward perception. Limited by the cost and computing power of sensors at the time, this approach had significant blind spots. With the advancement of panoramic vision systems and embedded hardware capabilities, 360-degree panoramic visual perception systems are now widely deployed on modern unmanned vessels, enabling them to achieve omnidirectional scene perception and effectively improving monitoring coverage of the maritime environment.

[0004] However, existing semantic segmentation technologies are mostly based on conventional images and are widely used in urban roads, autonomous driving and other fields. However, their application in the maritime field, especially in panoramic images, is still relatively limited. The main problems are as follows:

[0005] (1) The lack of publicly available panoramic image datasets of shipping scenes with semantic segmentation annotations limits the training and evaluation of algorithms. (2) Panoramic images have a wide field of view and large image distortion, which places higher accuracy and real-time requirements on semantic segmentation algorithms. (3) The computing power of embedded ship systems is limited, making it difficult to support complex deep models, resulting in the inability of existing methods to be efficiently deployed in practical environments. (4) Shipping scenes often contain a large number of small targets (such as buoys, distant ships, etc.), and the accuracy of existing models in small target segmentation is still insufficient.

[0006] Therefore, there is an urgent need for a semantic segmentation method that is suitable for panoramic shipping images and takes into account both segmentation accuracy and real-time performance under conditions of limited computing power, so as to meet the practical application needs of maritime intelligent devices such as unmanned ships. Summary of the Invention

[0007] To solve the above technical problems, the present invention proposes a lightweight dual-branch panoramic image semantic segmentation method for shipping scenarios, which can achieve high-precision and high-frame-rate panoramic image semantic segmentation while maintaining low computing resource consumption. It is suitable for intelligent shipping application scenarios such as unmanned ships and water robots that require real-time perception.

[0008] To achieve the above objectives, the present invention provides a lightweight dual-branch panoramic image semantic segmentation method for shipping scenarios, comprising:

[0009] Obtaining a panoramic image to be segmented;

[0010] The panoramic image to be segmented is input into a dual-branch fusion model to obtain a segmentation result, wherein the dual-branch fusion model is obtained by training a training set, the training set includes multi-view shipping images and corresponding semantic annotations, and the dual-branch fusion model is obtained by improving the multi-scale feature perception and fusion of the encoder-decoder.

[0011] Optionally, obtaining the training set includes:

[0012] Acquire multi-view shipping images;

[0013] Stitching the multi-view shipping images to obtain a panoramic image;

[0014] The training set is acquired based on the panoramic image.

[0015] Optionally, based on the panoramic image, obtaining the training set includes:

[0016] Annotating the panoramic image to obtain label information;

[0017] Combining the label information with the corresponding image to obtain a mask image;

[0018] Based on the mask image, the training set is obtained.

[0019] Optionally, inputting the panoramic image to be segmented into a dual-branch fusion model to obtain a segmentation result includes:

[0020] Performing preliminary feature extraction on the panoramic image to be segmented to obtain features of different scales of the image;

[0021] Performing spatial and contextual cross-fusion extraction on features of different scales of the image to obtain multi-scale features;

[0022] Processing the multi-scale features to obtain a feature map;

[0023] The feature map is converted into a semantic mask to obtain the segmentation result.

[0024] Optionally, performing preliminary feature extraction on the panoramic image to be segmented to obtain features of different scales of the image includes:

[0025] Extracting a first high-resolution feature and a first low-resolution feature of the panoramic image to be segmented;

[0026] The image feature is obtained based on the first high-resolution feature and the first low-resolution feature.

[0027] Optionally, performing spatial and contextual cross-fusion extraction on features of different scales of the image to obtain multi-scale features includes:

[0028] Downsampling the first high-resolution feature and performing interactive fusion and feature extraction on the first low-resolution feature to obtain a second low-resolution feature;

[0029] Upsampling the first low-resolution feature and performing interactive fusion and feature extraction on the first high-resolution feature to obtain a second high-resolution feature;

[0030] The multi-scale feature is obtained based on the second low-resolution feature and the second high-resolution feature.

[0031] Optionally, a method for obtaining the second low-resolution feature is:

[0032]

[0033] Among them, X h and X l are the high-resolution and low-resolution features input to the CNN module, respectively, S (·) and F C (·) are SFE module and CFE module respectively, F SD (·) refers to the downsampling operation, It is the low-resolution feature output by the CNN module that is fused with the high-resolution feature.

[0034] Optionally, the method for obtaining the second high-resolution feature is:

[0035]

[0036] Among them, F CB refers to the upsampling operation, It is the high-resolution feature output by the CNN module that is fused with the low-resolution features.

[0037] Optionally, processing the multi-scale features to obtain the feature map includes:

[0038] Performing unified alignment on the multi-scale features to obtain aligned multi-scale features;

[0039] The aligned multi-scale features are fused to obtain the feature map.

[0040] Optionally, the method for fusing the aligned multi-scale features is:

[0041]

[0042] Among them, F fusion is the fused feature, F i is the i-th feature, w i is the weight corresponding to the feature, Conv 3×3 It is a convolution with a kernel size of 3×3.

[0043] Compared with the prior art, the present invention has the following advantages and technical effects:

[0044] Based on real shipborne panoramic images, the present invention constructs the first panoramic semantic segmentation dataset MODSS for shipping scenes, filling the data gap in this field and making up for the data missing problem in the field of panoramic semantic segmentation of shipping. The dual-branch aggregation network proposed in the present invention combines spatial pyramid pooling with a hybrid attention mechanism to improve feature extraction and fusion capabilities. By designing a hybrid weighted loss function, the accuracy of small target segmentation is improved while ensuring the accuracy of global semantic parsing. Under the premise of ensuring segmentation accuracy, the real-time performance of the model is greatly improved, making it suitable for embedded system deployment; the present invention also effectively improves the detection and segmentation capabilities of small targets (such as buoys, distant ships, etc.). BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of this application. The exemplary embodiments and descriptions of this application are intended to explain this application and do not constitute an improper limitation on this application. In the accompanying drawings:

[0046] Figure 1 This is a flow chart of a lightweight dual-branch panoramic image semantic segmentation method applied to shipping scenarios according to an embodiment of the present invention;

[0047] Figure 2 This is a diagram of a panoramic image semantic segmentation network structure according to an embodiment of the present invention;

[0048] Figure 3 is a structural diagram of a convolution module in an encoder according to an embodiment of the present invention;

[0049] Figure 4 4 is a structural diagram of the ECA module according to an embodiment of the present invention. DETAILED DESCRIPTION

[0050] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0051] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0052] This embodiment proposes a lightweight dual-branch panoramic image semantic segmentation method for shipping scenarios, such as Figure 1 As shown, the specific steps include:

[0053] Obtaining a panoramic image to be segmented;

[0054] The panoramic image to be segmented is input into a dual-branch fusion model to obtain the segmentation result. The dual-branch fusion model is obtained by training with a training set, which includes multi-view shipping images and corresponding semantic annotations. The dual-branch fusion model is obtained by improving the multi-scale feature perception and fusion of the encoder-decoder.

[0055] Specifically, this method, based on the constructed MODSS maritime panoramic semantic segmentation dataset, achieves efficient extraction of spatial and contextual features within a dual-branch neural network framework. It also improves segmentation accuracy and real-time performance through an attention-guided weighted feature fusion module. By combining a weighted strategy with multiple loss functions, it effectively balances global semantic recognition and small object boundary segmentation performance. Experiments demonstrate that this method achieves a mean Intersection Over Union (MIoU) of 92.36% and a frame rate of 4.94 FPS on the MODSS dataset, demonstrating greater robustness and computational efficiency than existing methods. It is suitable for intelligent robotics and maritime applications requiring high precision and real-time performance.

[0056] Furthermore, obtaining a training set includes:

[0057] Acquire multi-view shipping images;

[0058] Stitching multi-view shipping images to obtain panoramic images;

[0059] Based on the panoramic image, a training set is obtained.

[0060] Furthermore, based on the panoramic image, obtaining a training set includes:

[0061] Annotate the panoramic image and obtain label information;

[0062] Combine the label information with the corresponding image to obtain the mask image;

[0063] Based on the mask image, obtain the training set.

[0064] Specifically, this embodiment stitches out a panoramic image of a 360-degree shipping scene based on open source multi-view shipping images, and marks semantic mask intervals of four categories including "Water", "Ship", "Buoy" and "Self". The open source tool LabelMe is used for labeling, and the label information is automatically stored in JSON format, with 1063 "Water", 3823 "Ship", 447 "Buoy" and 1693 "Self" labeled. The JSON label information is combined with the corresponding image to generate a mask image. In this way, the Ocean Panoramic Semantic Segmentation (MODSS) dataset is constructed and open sourced, filling the gap in the existing shipping panoramic semantic segmentation annotation dataset. At present, shipping semantic segmentation datasets are mostly constructed based on conventional cameras with limited field of view (FOV), while open source panoramic image datasets focus on land scenes. This dataset provides important support for reliable numerical evaluation of panoramic scene segmentation, effectively promotes the research and development of panoramic semantic segmentation in the shipping field, and promotes technological progress and application expansion in related fields.

[0065] Furthermore, the panoramic image to be segmented is input into the dual-branch fusion model to obtain the segmentation results including:

[0066] Perform preliminary feature extraction on the panoramic image to be segmented to obtain features of different scales of the image;

[0067] Perform spatial and contextual cross-fusion extraction on features of different scales of the image to obtain multi-scale features;

[0068] Process the multi-scale features to obtain feature maps;

[0069] The feature map is converted into a semantic mask to obtain the segmentation result.

[0070] Specifically, such as Figure 2As shown, inspired by the multi-branch network structures DDRNet and PIDNet, this embodiment designs a DBANet (dual-branch fusion network) embedded with weighted fusion based on the attention mechanism. This network is an encoding-decoding structure, where W and H are the width and height of the image, respectively, and C is the number of channels initialized. First, in the encoder, the image is subjected to preliminary feature extraction and N-times downsampling of the input image through a preliminary feature processing module (Feature stem module). Secondly, the features are cross-fused and extracted spatially and contextually through two CNN modules, thereby ensuring that the multi-scale features output by the encoder are cross-correlated. Subsequently, based on the above characteristics, a more efficient decoder is used to directly fuse features of four different scales instead of adopting step-by-step upsampling fusion, thereby improving the real-time performance of the network while ensuring a certain accuracy rate. Finally, the segmentation head is used to convert the feature map into a semantic mask result. According to the computing performance and scenario requirements of the robot system, four versions of the network are configured by adjusting the parameters. DBANet-R is the regular version with N set to 4 and C set to 128; DBANet-S is the small version with N set to 4 and C set to 64; DBANet-T is the tiny version with N set to 8 and C set to 128; and DBANet-N is the nano version with N set to 8 and C set to 64.

[0071] Furthermore, preliminary feature extraction is performed on the segmented panoramic image to obtain features of different scales of the image, including:

[0072] Extracting a first high-resolution feature and a first low-resolution feature of the panoramic image to be segmented;

[0073] Based on the first high-resolution feature and the first low-resolution feature, image features at different scales are obtained.

[0074] Furthermore, the spatial and contextual cross-fusion extraction of different scale features of the image is performed to obtain multi-scale features including:

[0075] Downsampling the first high-resolution feature and interactively fusing and extracting the first low-resolution feature to obtain a second low-resolution feature;

[0076] Upsampling the first low-resolution feature and interactively fusing and extracting the first high-resolution feature to obtain a second high-resolution feature;

[0077] A multi-scale feature is obtained based on the second low-resolution feature and the second high-resolution feature.

[0078] Specifically, such as Figure 3As shown in Figure 2, the Feature Stem module in the dual-branch cross-perception encoder is a modified ResNet stem, with its 7×7 convolutions replaced by multiple 3×3 convolutions. This module converts the 3×H×W feature map into C×H / N×W / N features for subsequent spatial and contextual feature extraction, where C is the number of channels; H and W are the height and width of the feature, respectively; and N is the downsampling factor. Two CNN modules in the dual-branch cross-perception encoder are used to cross-fuse the extracted spatial and contextual features.

[0079] The spatial feature extraction (SFE) module extracts high-resolution features using multiple convolutional BasicBlocks while maintaining the same number of channels. The contextual feature extraction (CFE) module downsamples the image using multiple convolutional BasicBlocks and extracts low-resolution features. These low-resolution features are then fed into the decoder. Subsequently, the high-resolution features extracted by the SFE module are enhanced in multi-scale representation by downsampling them based on the SPPENLAN module and fused with the low-resolution features to form new low-resolution features. The calculation process is as follows:

[0080]

[0081] Among them, X h and X l are the high-resolution and low-resolution features input to the CNN module, respectively, S (·) and F C (·) are SFE module and CFE module respectively, F SD (·) refers to SPPELAN and downsampling operations, It is the low-resolution feature output by the CNN module that is fused with the high-resolution feature. At the same time, the low-resolution feature channel extracted by the CFE module is compressed and its resolution is upsampled based on the Bilinear operation, the mathematical derivation of which is as follows:

[0082]

[0083] Among them, F CB Refers to feature compression and Bilinear operation, It is the high-resolution feature output by the CNN module that is fused with the low-resolution features.

[0084] Finally, the high-resolution features output by CNN module 2 are processed by the SFE module and then input into the decoder. The low-resolution features output by CNN module 2 are processed by the bottleneck module and then input into the decoder. The bottleneck module helps extract deep features and aggregate contextual information. In summary, the final output features at four different scales are cross-correlated.

[0085] Furthermore, the multi-scale features are processed to obtain feature maps including:

[0086] Perform unified alignment on multi-scale features to obtain aligned multi-scale features;

[0087] The aligned multi-scale features are fused to obtain the feature map.

[0088] Specifically, most multi-scale semantic segmentation decoders use a step-by-step decoding method to upsample images, which can better restore image information step by step. In addition, some guided upsampling methods applied to step-by-step processing have also been proposed for finer-grained upsampling. Although step-by-step decoding improves semantic segmentation performance, it significantly reduces the real-time performance of the network. Since the four scale features output by the decoder proposed in this embodiment have cross-correlation, the designed decoder uses direct feature cross-fusion to improve the real-time performance of the network. A convolution and resize module is used in the feature channel alignment module (Channel Spatial Aligner module) of the decoder to align all features to 2C×H / N×W / N. Subsequently, a feature fusion (WFAM) module based on the attention mechanism is designed to fuse features at different stages, and its calculation process is as follows:

[0089]

[0090] Among them, F fusion is the fused feature, F i is the i-th feature, w i is the weight corresponding to the feature, Conv 3×3 is a convolution with a kernel size of 3×3. i The calculation is as follows:

[0091] w i =ECA(F i ),i∈{1,2,3,4}

[0092] Among them, ECA(·) is the efficient channel attention (ECA) module. Figure 4As shown in Figure 3, in the ECA module, global average pooling (GAP) is adopted to obtain the global semantic information of each channel, and convolution & sigmoid activation function is used to generate the weight of each channel.

[0093] More specifically, in order to balance the global semantic classification accuracy and small target edge detail accuracy of the designed network model, a weighted combination of multiple loss functions is adopted to calculate the overall loss during training. Since the input image targeted by the designed network is a panoramic image, it has problems such as small targets, complex boundaries, and imbalanced categories. Therefore, Cross Entropy Loss is used as the main loss function to improve the overall semantic information classification accuracy, and its weight λ1 is set to 0.5. Dice Loss is adopted to focus on optimizing the small target segmentation performance, thereby compensating for the problem of low local attention, and its weight λ2 is set to 0.3. Tversky Loss is adopted to strengthen boundary optimization and category imbalance processing, and its weight λ3 is set to 0.2. In summary, the calculation of the overall loss function is:

[0094] L=λ1×L CE +λ2×L Dice +λ3×L Tversky

[0095] Among them, L CE , L Dice , L Tversky The calculations are as follows:

[0096]

[0097] Where N is the total number of pixels; y i and p i are the one-hot label and model prediction of pixel i, respectively; ε is a small constant used to prevent the denominator from being zero; α and β are hyperparameters that control the weights and are set to 0.5.

[0098] To test the performance of the proposed method, this example used the Sugon cloud computing platform's NVIDIA RTX3080 to train a network model on the MODSS dataset. To mitigate errors caused by server communication delays during network real-time calculations, FPS calculations were performed using a computer equipped with an Intel i7-9750H CPU and an NVIDIA GTX1660Ti.

[0099] The developed code is based on MMSegmentation, an open-source semantic segmentation library. During the experiments, all network models were trained with a batch size of 2 and an iterator of 170,000. In addition, a unified data preprocessing configuration was used based on the mean and standard deviation of the image dataset. The crop size was set to (512, 512).

[0100] During the verification process, the input images are uniformly resized to the real image size ratio and the meanIntersection overUnion (mIoU) metric is used to evaluate the semantic segmentation accuracy. The mIoU is calculated as follows:

[0101]

[0102] Among them, N c is the number of all categories of semantic segmentation results, IoU i is the IoU of the i-th class, which is calculated as follows:

[0103]

[0104] Among them, TP i is the number of pixels correctly predicted for the i-th class, FP i is the number of pixels incorrectly predicted as class i, FN i is the number of pixels that were incorrectly predicted as belonging to the i-th class. During the entire network model design process, each iteration includes both forward and backward passes for each dataset, with a standard loss applied throughout. Furthermore, frame rate (FPS) is also an important metric for evaluating the real-time performance of the network.

[0105] First, mainstream lightweight real-time semantic segmentation methods and their corresponding versions were tested, including DDRNet, FastSCNN, PIDNet, BiseNet, and ERFNet. Subsequently, mainstream high-performance semantic segmentation methods and their corresponding versions were tested, including PSPNet, DeeplabV3+, UNet, KNet, Mask2Former, and Segformer. The experimental results are shown in Table 1:

[0106] Table 1

[0107]

[0108]

[0109] Experimental results show that DDRNet achieves the best overall performance among lightweight network models, and Mask2Former achieves the highest accuracy among all network models. Compared to DDRNet and DDRNet-slim, the proposed Tiny and Nano versions achieve similar FPS while improving mIOU by 1.71 and 1.27, respectively. Compared to Mask2Former-SwinS, the proposed Regular version achieves a 0.63% decrease in mIOU but a 147.0% increase in FPS.

[0110] The above are merely preferred embodiments of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A lightweight dual-branch panoramic image semantic segmentation method for shipping scenarios, characterized by: include: Obtaining a panoramic image to be segmented; The panoramic image to be segmented is input into a dual-branch fusion model to obtain a segmentation result, wherein the dual-branch fusion model is obtained by training a training set, the training set includes multi-view shipping images and corresponding semantic annotations, and the dual-branch fusion model is obtained by improving the multi-scale feature perception and fusion of the encoder-decoder.

2. A lightweight dual-branch panoramic image semantic segmentation method for shipping scenarios according to claim 1, characterized in that: Obtaining the training set includes: Acquire multi-view shipping images; Stitching the multi-view shipping images to obtain a panoramic image; The training set is acquired based on the panoramic image.

3. The lightweight dual-branch panoramic image semantic segmentation method for shipping scenarios according to claim 2 is characterized in that: Based on the panoramic image, acquiring the training set includes: Annotating the panoramic image to obtain label information; Combining the label information with the corresponding image to obtain a mask image; Based on the mask image, the training set is obtained.

4. The lightweight dual-branch panoramic image semantic segmentation method for shipping scenarios according to claim 1 is characterized in that: Inputting the panoramic image to be segmented into the dual-branch fusion model to obtain the segmentation result includes: Performing preliminary feature extraction on the panoramic image to be segmented to obtain features of different scales of the image; Performing spatial and contextual cross-fusion extraction on features of different scales of the image to obtain multi-scale features; Processing the multi-scale features to obtain a feature map; The feature map is converted into a semantic mask to obtain the segmentation result.

5. The lightweight dual-branch panoramic image semantic segmentation method for shipping scenarios according to claim 4 is characterized in that: Performing preliminary feature extraction on the panoramic image to be segmented to obtain features of different scales of the image includes: Extracting a first high-resolution feature and a first low-resolution feature of the panoramic image to be segmented; Based on the first high-resolution feature and the first low-resolution feature, the image features at different scales are acquired.

6. The lightweight dual-branch panoramic image semantic segmentation method for shipping scenarios according to claim 5 is characterized in that: Performing spatial and contextual cross-fusion extraction on the different scale features of the image to obtain multi-scale features includes: Downsampling the first high-resolution feature and performing interactive fusion and feature extraction on the first low-resolution feature to obtain a second low-resolution feature; Upsampling the first low-resolution feature and performing interactive fusion and feature extraction on the first high-resolution feature to obtain a second high-resolution feature; The multi-scale feature is obtained based on the second low-resolution feature and the second high-resolution feature.

7. The lightweight dual-branch panoramic image semantic segmentation method for shipping scenarios according to claim 6 is characterized in that: The method for obtaining the second low-resolution feature is: Among them, X h and X l are the high-resolution and low-resolution features input to the CNN module, respectively, S (·) and F C (·) are SFE module and CFE module respectively, F SD (·) refers to the downsampling operation, It is the low-resolution feature output by the CNN module that is fused with the high-resolution feature.

8. The lightweight dual-branch panoramic image semantic segmentation method for shipping scenarios according to claim 7 is characterized in that: The method to obtain the second high-resolution features is: Among them, F CB refers to the upsampling operation, It is the high-resolution feature output by the CNN module that is fused with the low-resolution features.

9. The lightweight dual-branch panoramic image semantic segmentation method for shipping scenarios according to claim 4 is characterized in that: Processing the multi-scale features to obtain the feature map includes: Performing unified alignment on the multi-scale features to obtain aligned multi-scale features; The aligned multi-scale features are fused to obtain the feature map.

10. The lightweight dual-branch panoramic image semantic segmentation method for shipping scenarios according to claim 9, characterized in that: The method for fusing the aligned multi-scale features is: Among them, F fusion is the fused feature, F i is the i-th feature, w i is the weight corresponding to the feature, Conv 3×3 It is a convolution with a kernel size of 3×3.

Images