A method for positive and negative obstacle segmentation based on multi-task guidance

Abstract

The application discloses a kind of based on the method for jumping and fusing positive and negative obstacles segmentation of multi-task guidance, including the following steps: build the jumping and fusing positive and negative obstacles segmentation model based on multi-task guidance and training;Surface normal vector map is calculated by parallax map / depth map, after resolution adjustment to surface normal vector map and corresponding RGB image, input to the jumping and fusing positive and negative obstacles segmentation model trained, obtain positive and negative obstacles segmentation result;The jumping and fusing positive and negative obstacles segmentation model adopts normal vector branch to receive surface normal vector map, and output normal vector segmentation map is used for loss calculation in training process;RGB branch is received using RGB branch, and output RGB segmentation map is used as output positive and negative obstacles segmentation result, and RGB segmentation map participates in loss calculation in training process.The application solves the problem that multi-modal network is not good in the scene of inconsistent data between modes Fusion effect, improve the model recognition ability to obstacle.

Description

Technical Field

[0001] This invention relates to the field of semantic segmentation in computer vision, and in particular to a method for segmenting positive and negative obstacles based on multi-task guidance and jump-fusion. Background Technology

[0002] Semantic segmentation is a task in computer vision that aims to label different objects, regions, or features in an image and assign them unique labels, such as people, cars, and trees. Each pixel is assigned a specific category label, enabling fine-grained analysis of the image. In recent years, semantic segmentation has been widely applied in many scenarios, such as scene understanding in autonomous driving, lesion detection in medical images, and object segmentation in image editing and enhancement.

[0003] Road obstacle segmentation is a fundamental task in autonomous driving, belonging to the field of semantic segmentation. Road obstacles can generally be divided into two categories: positive obstacles and negative obstacles. Positive obstacles refer to obstacles above the ground, such as pedestrians, vehicles, and bicycles. These obstacles have attracted significant attention in the robotics research community because their detection or segmentation is crucial for downstream tasks such as tracking, obstacle avoidance, and navigation. Negative obstacles refer to obstacles below the ground, such as potholes and cracks, which can cause discomfort to passengers. Larger negative obstacles can even lead to accidents, such as vehicle rollovers. Therefore, accurate detection or segmentation of negative obstacles is also very important for the safety of autonomous vehicles. A common approach for segmenting positive and negative obstacles is to use deep learning techniques. Early researchers proposed many effective segmentation methods based on deep learning, but most used single-modal visual data (e.g., RGB images only). Since visual images can be affected by ambient lighting conditions, these methods are not entirely satisfactory. To address this issue, many multimodal (e.g., RGB depth / parallax, RGB thermal) fusion networks have been proposed. R. Fan et al. developed an effective training set enhancement technique based on adversarial domain adaptation to generate synthetic road RGB images and transformed road disparity (or inverse depth) images to enhance the training of semantic segmentation networks. C. Hazirbas et al. proposed FuseNet, a semantic segmentation network based on RGB and depth maps. It uses two CNN structures at the front end to extract features from the RGB and depth images respectively, and employs SparseFusion to fuse some features from the depth network into the RGB network. These multimodal segmentation networks have achieved results superior to single-modal networks, thus multimodal networks have gradually become a focus of research.

[0004] Previous multimodal fusion segmentation networks typically fused depth / disparity maps with RGB images. While depth / disparity maps provide depth features, fusing them with RGB images offers limited improvement to the segmentation network. Surface normal maps, on the other hand, provide pixel normal information. Derived from dense depth / disparity maps, they offer more texture information, making them crucial for obstacle segmentation. Furthermore, previous modal fusion methods did not fully utilize the diverse features across multiple modalities. This resulted in some scenarios that performed poorly in single-modal segmentation not showing significant improvement in multimodal segmentation, especially with datasets like NPOs (Network Objects) exhibiting inconsistent data across modalities. Summary of the Invention

[0005] The purpose of this invention is to solve the problem that in existing image obstacle segmentation techniques, multimodal fusion segmentation networks cannot improve segmentation performance due to feature fusion issues.

[0006] The technical solution adopted by this invention to solve its technical problem is: to provide a method for positive and negative obstacle segmentation based on multi-task guidance and fusion, comprising the following steps:

[0007] A jump-connection fusion positive and negative obstacle segmentation model based on multi-task guidance was built and trained.

[0008] The surface normal map is calculated using the disparity map / depth map. The resolution of the RGB images of the surface normal map and the corresponding disparity map / depth map is adjusted and then input into the trained jump-connection fusion positive and negative obstacle segmentation model to obtain the positive and negative obstacle segmentation results.

[0009] The jump-connection fusion positive and negative obstacle segmentation model adopts a dual-branch structure, including:

[0010] The normal vector branch receives the surface normal vector map, extracts the color features, texture features, and edge features of the surface normal vector map through the encoder to obtain the normal vector feature map, and performs feature mapping and image scale restoration on the normal vector feature map through the decoder to output the normal vector segmentation map for loss calculation in the training process.

[0011] The RGB branch receives an RGB image, extracts color, texture, and edge features from the RGB image through an encoder to obtain an RGB feature map, and fuses the normal vector feature map and the RGB feature map through a decoder and a multi-task guided jump fusion module to obtain a fused feature map. The decoder performs feature mapping and image scale restoration on the fused feature map and outputs an RGB segmentation map as the output positive and negative obstacle segmentation results. The RGB segmentation map participates in the loss calculation during the training process.

[0012] Preferably, both the normal vector branch and the RGB branch adopt a five-level encoder-five-level decoder structure, wherein the fifth-level encoder of the normal vector branch adopts a Transformer layer.

[0013] Preferably, in the normal vector branch, the input of each encoder level is fused with the output of the corresponding decoder level by pixel-level addition and used as the input of the next level decoder, and also as one of the inputs of the same level multi-task guidance jump fusion module.

[0014] Preferably, in the RGB branch, the input of each encoder level is fused with the output of the decoder at the same level by pixel-level addition, and then used as the input of the multi-task guidance jump-connection fusion module at the same level; the output of the multi-task guidance jump-connection fusion module is used as the input of the next level decoder, and the output of the last level multi-task guidance jump-connection fusion module is the output of the positive and negative obstacle segmentation results.

[0015] Preferably, the inputs of the first to fourth level multi-task guidance jumper fusion modules also include the output of the previous level normal vector encoder.

[0016] Preferably, in the RGB branch, the inputs of the fourth-level encoder and the fifth-level encoder are first processed by the channel attention layer to adjust the weight relationship between the channels, and then fused with the output of the same-level decoder via a skip connection, and output to the same-level multi-task guided skip connection fusion module.

[0017] Preferably, the feature fusion process performed by the multi-task guided jump-connection fusion module includes the following steps:

[0018] For the i-th level multi-task guided skip connection fusion module, the input is the feature map F, which is the output of the i-th level decoder from the normal vector branch after skip connection fusion. sne The feature map F obtained by fusing the output of the i-th level decoder of the RGB branch with the skip connections rgb At this point, the number of channels in the feature map is C;

[0019] For feature map F sne and feature map F rgb The residual feature map F is obtained by subtraction. res1 ;

[0020] Feature map F sne Feature map F rgb and residual feature map F res1 By adjusting the number of channels in the convolutional layer to 3, three segmentation images F are obtained. sne_seg F rgb_seg and F res_seg1 , where F res_seg1 Then, deep features are extracted through residual connections to form the final residual segmentation map F. res_seg ;

[0021] Residual partitioning diagram F res_seg By adjusting the number of channels C through a convolutional layer, a new F is obtained. res F res With F sne and F rgb Pixel-level multiplication is performed to obtain two weighted feature maps F. sne1 and F rgb1 , will F sne1 F rgb1 F sne F rgb and F res The feature maps are concatenated along the channel dimension, and the resulting feature maps are adjusted for the number of channels and then compared with the input feature map F. sne and F rgb Perform skip connections, then pass through a ReLU activation layer to obtain the feature map F. fusion ;

[0022] F fusion After residual connections through two convolution layers, a BN layer, and a ReLU layer, and then with the input feature map F sne and F rgb Perform a skip connection, and obtain F after passing through the ReLU activation layer. fusion1 ;

[0023] F fusion1 After four parallel dilated convolutions of different scales, four convolution outputs are obtained; the four convolution outputs are compared with F. fusion1 The layers are then stitched together, and then passed through a convolutional layer to adjust the number of channels to C, and then compared with F. sne and F rgb Perform a final jump connection to obtain the initial output;

[0024] For the last-stage multi-task guided jump-connection fusion module, the preliminary output is used as the final output; for other multi-task guided jump-connection fusion modules, the preliminary output is added pixel-wise to the output of the (i-1)th level encoder of the normal vector to obtain the final output.

[0025] Preferably, the loss calculation in the training process includes the following steps:

[0026] Downsampling the ground truth map gt makes its resolution equal to that of the i-th level segmentation map (F sne_seg F rgb_seg F res_seg With consistent resolution, the downsampled ground truth map (gt) is obtained. d ;

[0027] The normal vector segmentation map output by the last stage decoder in the normal vector branch and the corresponding ground truth map gt are used to calculate the loss L using the cross-entropy loss function.sne ;

[0028] The RGB segmentation image output by the last-level multi-task guidance and fusion module of the RGB branch and the corresponding ground truth image gt are used to calculate the loss L through the cross-entropy loss function. rgb ;

[0029] The gt is calculated using the cross-entropy loss function. d The normal vector segmentation map F generated by the i-th level multi-task guidance jump-connection fusion module sne_seg loss L sne_i , and gt d The RGB segmentation map F generated by the i-th level multi-task guidance jumper fusion module rgb_seg loss L rgb_i ;

[0030] gt d Subtract the RGB segmentation map F generated in the fusion module rgb_seg Obtain the true residual plot gt d_res gt d_res The residual segmentation map F generated by the fusion module res_seg The loss L is calculated using the cross-entropy loss function. res_i , where i represents the multi-task guided jump-fusion module that calculates the loss after the output of the i-th level decoder;

[0031] Total loss L total Represented as:

[0032]

[0033] Where i takes values ​​from 1 to 5.

[0034] The present invention has the following beneficial effects: It designs a multimodal fusion positive and negative obstacle segmentation network, proposing an efficient fusion module that can fuse feature maps at multiple scales, fully utilizing feature information between different modalities while preserving details before feature map fusion. This solves the problem of poor fusion performance in scenarios where data is inconsistent between modalities, a problem previously encountered with multimodal networks. Furthermore, it introduces surface normal vector maps with normal information, replacing the previous depth maps / disparity maps, increasing the model's focus on texture information and further improving the model's obstacle recognition ability. The present invention achieves the best results on the NPO dataset, and its real-time performance meets the requirements of road driving scenarios.

[0035] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments, but the present invention is not limited to the embodiments. Attached Figure Description

[0036] Figure 1 This is a diagram illustrating the method steps of an embodiment of the present invention;

[0037] Figure 2 This is a structural diagram of the jump-connection fusion positive and negative obstacle segmentation model according to an embodiment of the present invention;

[0038] Figure 3 This is a structural diagram of the multi-task guidance jumper fusion module according to an embodiment of the present invention;

[0039] Figure 4 This is a schematic diagram of the visualization segmentation results of an embodiment of the present invention. Detailed Implementation

[0040] See Figure 1 The diagram shows the method steps of an embodiment of the present invention, including the following steps:

[0041] S101, Build and train a jump-connection fusion positive and negative obstacle segmentation model based on multi-task guidance;

[0042] S102: Calculate the surface normal map using the disparity map / depth map, adjust the resolution of the RGB images of the surface normal map and the corresponding disparity map / depth map, and input them into the trained jump-connection fusion positive and negative obstacle segmentation model to obtain the positive and negative obstacle segmentation results.

[0043] In step S102, the surface normal map is calculated using the disparity map / depth map. In this embodiment, the disparity map in the NPO is used as input. The disparity map is first processed by the surface normal vector generator to calculate the optimal normal information corresponding to each pixel point, thereby obtaining the required surface normal map.

[0044] For details, see Figure 2 As shown, the jump-connection fusion positive and negative obstacle segmentation model adopts a dual-branch structure, including:

[0045] The normal vector branch receives the surface normal vector map, extracts the color features, texture features, and edge features of the surface normal vector map through the encoder to obtain the normal vector feature map, and performs feature mapping and image scale restoration on the normal vector feature map through the decoder to output the normal vector segmentation map for loss calculation in the training process.

[0046] The RGB branch receives an RGB image, extracts color, texture, and edge features from the RGB image through an encoder to obtain an RGB feature map, and fuses the normal vector feature map and the RGB feature map through a decoder and a multi-task guided jump fusion module to obtain a fused feature map. The decoder performs feature mapping and image scale restoration on the fused feature map and outputs an RGB segmentation map as the output positive and negative obstacle segmentation results. The RGB segmentation map participates in the loss calculation during the training process.

[0047] Specifically, the process of segmenting obstacles by merging positive and negative obstacle segmentation models includes the following steps:

[0048] (1) The surface normal vector map and RGB map are first encoded by the five-level encoder part of their respective branches to extract semantic information from the image from shallow to deep. The encoder backbone adopts ResNet-34. In order to capture the long-range dependency of pixels and improve the expressive power of the model, the fifth level encoder of the surface normal vector branch is replaced with a Transformer layer with a layer number of 3, including layer normalization (Layer Normal), multi-head self-attention (MSA) and multilayer perceptron (MLP). The number of attention heads is set to 8. Each level of the encoder reduces the resolution of the input image by half. The number of output channels of each level of the encoder is adjusted to 64, 64, 128, 256 and 512 respectively.

[0049] (2) After encoding, the decoding stage begins. A five-level decoder performs feature mapping and scale restoration on the image. Similarly, each decoder level doubles the resolution of the input image. The number of output channels for each encoder level is adjusted to 256, 128, 64, 32, and 3 respectively, finally generating a three-channel segmentation map. The segmentation map from the normal vector branch only participates in the loss function calculation during training, while the segmentation map from the RGB branch is used as the network output. Each decoder level operation includes the following steps:

[0050] a. First, the input feature map is fed into a two-branch residual structure. The residual structure reduces the number of channels in the input feature map by half. The upper branch has a Convolution layer with 1x1 convolution kernels, followed by a BN layer and a ReLU layer. The lower branch has three Convolution-BN-ReLU layers with 3x3 convolution kernels. The first layer in the lower branch reduces the number of channels in the input feature map by half. The other layers in the lower branch keep the number of channels unchanged, and then the outputs of the two branches are fused by pixel-level addition.

[0051] b. Next, the output of the residual structure is fed into a global average pooling layer to adjust the resolution to 1*1. Fully connected (FC)-BN-ReLU layers and FC layers are used to generate weights for different channels of the fused result. The FC-BN-ReLU layer reduces the number of channels by half, and the FC layer restores the number of channels. Then, a sigmoid layer is used to map the generated channel weights to [0, 1]. The output of the residual structure is then fused with the mapped weights through element-wise multiplication.

[0052] c. Use a transposed Convolution-BN-ReLU layer to double the resolution, while adjusting the number of channels to the set output channel number. The output of the transposed Convolution-BN-ReLU layer is the output of each decoder stage.

[0053] (3) In both branches, in order to preserve the original details during the decoding process, the input of each level encoder and the output of each level decoder are fused at the same level by pixel-level addition. In the RGB branch, the input of the fourth and fifth level encoders first passes through an attention layer, and then is fused with the output of the same level decoder by pixel-level addition as the input of the next level decoder. The purpose is to make the feature map with a high number of channels focus on the dependencies between channels.

[0054] (4) The output of each decoder stage is fused by a multi-task guided jumper fusion module. The structure of the multi-task guided jumper fusion module (MSFM) is as follows: Figure 3 As shown, its operation includes the following steps:

[0055] a. Output feature map F of a certain level decoder in two branches sne and F rgb Using all three as input, the feature map of the surface normal vector branch is first subtracted from the decoder output feature map of the RGB branch to obtain the residual feature map. The three feature maps are then adjusted by a 1*1 convolutional layer to become three-channel segmentation maps: the surface normal vector segmentation map, the RGB segmentation map, and the residual segmentation map. The residual segmentation map is then passed through a 3*3 convolutional layer for residual connection to obtain a new residual segmentation map.

[0056] b. Downsample the ground truth image to the same resolution as the RGB segmentation image in a, to obtain the ground truth image. d The gt is calculated using the cross-entropy loss function. d Loss L of the surface normal vector segmentation map sne_i and gt d Loss L compared to RGB segmentation map rgb_i Where i represents the fusion module that calculates the loss after the output of the i-th level decoder, and gt d The loss L is calculated using the cross-entropy loss function between the true residual image obtained by subtracting the RGB segmentation image and the residual segmentation image generated in 'a'. res_i , where i represents the fusion module that calculates the loss after the output of the i-th level decoder, and the value of i ranges from 1 to 5.

[0057] c. Preliminary feature map fusion: The residual segmentation map is adjusted to the number of image channels when the module input is processed by a 1*1 convolutional layer. At this point, the residual feature map is called F. res , respectively with the original input F of the surface normal vector branch in a sne The original input F of the RGB branch rgb After element-wise multiplication, the original input feature map of the normal vector branch becomes F. sne1 The original input feature map of the RGB branch becomes F rgb1 Then Fres F sne1 F rgb1 and F sne and F rgb The layers are concatenated along the channel dimension, then passed through a 1x1 convolutional layer to reduce the number of channels to the input channel count, and finally combined with F. sne and F rgb Perform a skip connection, followed by a ReLU activation layer, to obtain the fused feature map F. fusion .

[0058] dF fusion Deep features are extracted using residual connections. The residual connection operation is as follows: a 3x3 convolutional kernel layer – a batch normalization (BN) layer – a ReLU layer, then another 3x3 convolutional kernel layer – a batch normalization (BN) layer, and finally connected to the preceding F... fusion After pixel-level addition, the result passes through a ReLU activation layer. The residuals are then connected and combined with F... sne and F rgb The skip connections are added together, and finally passed through a ReLU activation layer to obtain F. fusion1 .

[0059] eF fusion1 Four parallel dilated convolutions with dilation rates of 1, 2, 3, and 4 are used. The outputs of these four convolutions are concatenated with the original input along the channel dimension. Finally, a convolution layer with a 1x1 kernel adjusts the number of channels to match the input channel count, and then the result is concatenated with F. sne and F rgb The jump connection is added, and in the first four fusion modules, it is finally added to the surface normal vector branch encoder of the same level once at the pixel level.

[0060] (5) Output: The segmentation map output by the last stage decoder of the normal vector branch and the ground truth map are compared using the cross-entropy loss function to calculate the loss L. sne The RGB branch outputs a fused segmentation map through the final fusion module as the model output, and calculates the loss L using the cross-entropy loss function together with the ground truth map. rgb .

[0061] Total model loss L total for:

[0062]

[0063] This invention uses the NPO dataset to train and test the model, which contains 2960 images from urban scenes and 2040 images from rural scenes. The dataset is divided into two categories based on road surface conditions: 3465 images of normal roads and 1535 images of abnormal roads (e.g., snow, puddles). The images in this dataset were collected under various weather conditions: 4231 on sunny days, 583 on snowy days, and 204 on cloudy days. In the NPO dataset, 4596 images contain negative obstacles, and 3105 images contain positive obstacles. The training set consists of 2500 images, the validation set 1250 images, and the test set 1250 images. During training and testing, the image resolution is reduced to 288×512. Training is performed using the publicly available PyTorch framework, implemented on an RTX 3080 GPU. The initial learning rate is 0.01, the learning rate decays to 0.95, the batch size is 2, the number of epochs is 100, and the ResNet encoder of the model is trained using pre-trained weights.

[0064] To verify the effectiveness of the invention, model comparison experiments and module ablation experiments were conducted:

[0065] To objectively evaluate the experimental results, mean Intersection over Union (mIoU), recall, and F1-score were used. IoU is the ratio of the intersection to the union of the true label and the predicted result for each class. mIoU is the average IoU across all classes and is the most commonly used metric for evaluating segmentation accuracy. Recall measures how many samples that are actually positive are correctly predicted by the model. F1-score is the harmonic mean of precision and recall, aiming to maximize both and minimize their difference. Recall is generally only used as a reference; the focus for evaluating model performance is on mIoU and F1-score. The formulas for calculating the evaluation metrics are as follows:

[0066]

[0067] Where i represents the true value, j represents the predicted value, p represents the predicted value, i is j, p, p and pij respectively, iiij ji represents TP, FP and FN; and k+1 is the class count, true example (TP), false negative example (FN), false positive example (FP) and true negative example (TN).

[0068]

[0069] Where R represents recall and F1 represents F1 score.

[0070] Several mainstream semantic segmentation models were selected for comparison, including unimodal and multimodal models. For unimodal models, training was performed using three unimodal datasets (RGB, depth, and disparity) and two multimodal datasets (RGB-depth and RGB-disparity). For multimodal models, unimodal training was performed by removing either the RGB or depth branch. The best training result from each model under different training strategies was compared with the model of this invention on the NPO dataset. For fairness, the backbone of the baseline model Inconseg in this invention was changed from ResNet-152 to ResNet-34. The experimental results are shown in Table 1.

[0071] Table 1

[0072]

[0073] As shown in Table 1, the present invention achieves optimal mIoU and mF1 in the NPO dataset. It can also be seen that some networks perform worse with multimodal input than with single-modal input.

[0074] In addition, to understand how each component of the method of the present invention affects segmentation, a series of ablation experiments were conducted, as shown in Tables 2 and 3:

[0075] Table 2

[0076]

[0077] Table 3

[0078]

[0079] In Table 2, the benchmark for the module ablation experiment is Inconseg, which is the baseline model of this invention. MSFM is the multi-task guided jump fusion module, TF is the introduction of the encoder Transformer layer, and SNE is the replacement of the disparity map mode with the surface normal vector mode. The symbol "√" indicates the adoption, and the symbol "×" indicates the method of maintaining the baseline model. The last column is the single-frame inference speed. It can be seen that each component of the method of this invention can improve the segmentation ability of the model without bringing too much computational burden. In Table 3, the experimental benchmark is still Inconseg. Three different fusion methods are used to fuse the feature maps. From top to bottom, they are pixel-level addition, RGF (i.e., the fusion method of the baseline model), and MSFM (the fusion method of this invention). It can be seen that the RGF module has a certain degree of improvement on the model segmentation ability compared with pixel-level addition, but the effect is not obvious. When fused by MSFM in the method of this invention, the model segmentation ability is significantly improved.

[0080] To make the comparison more intuitive, the results are visualized and compared with the baseline model Inconseg of this invention, such as... Figure 4 As shown, the first row is the RGB image, the second row is the ground truth image, the third row is the baseline model segmentation image, and the fourth row is the segmentation image of the present invention. It can be seen that the baseline model of the road section affected by shadows and puddles has omissions and incomplete segmentation, but the method of the present invention can still segment it relatively completely.

[0081] This invention proposes a multi-task-guided jump-fusion method for positive and negative obstacle segmentation. It uses a surface normal vector generator with disparity / depth maps as input to obtain a surface normal vector map and RGB values. Figure 1 The input to the model is used as the input, and each part is encoded and decoded through its own branch. The feature maps are fused during the decoding stage. The encoder adopts a CNN+Transformer structure, which enables the model to capture long-range pixel dependencies while extracting deep semantic features of the feature maps, thereby enhancing the model's expressive power. This invention uses a proposed multi-task guided jumper module for fusion. The module can guide the generation of normal vector maps, RGB maps, and residual maps in the decoding stage, and can make full use of the feature information of feature maps between different modalities to fuse feature maps at different levels. This solves the following problem: unreasonable fusion methods result in little improvement after fusion and poor segmentation effect, especially when facing datasets with inconsistent data between modalities such as NPO, the fusion effect may be even worse.

[0082] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for segmenting positive and negative obstacles based on multi-task guidance and jump-connection fusion, characterized in that, Includes the following steps: A jump-connection fusion positive and negative obstacle segmentation model based on multi-task guidance was built and trained. The surface normal map is calculated using the disparity map / depth map. The resolution of the RGB images of the surface normal map and the corresponding disparity map / depth map is adjusted and then input into the trained jump-connection fusion positive and negative obstacle segmentation model to obtain the positive and negative obstacle segmentation results. The jump-connection fusion positive and negative obstacle segmentation model adopts a dual-branch structure, including: The normal vector branch receives the surface normal vector map, extracts the color features, texture features, and edge features of the surface normal vector map through the encoder to obtain the normal vector feature map, and performs feature mapping and image scale restoration on the normal vector feature map through the decoder to output the normal vector segmentation map for loss calculation in the training process. The RGB branch receives an RGB image, extracts color, texture, and edge features from the RGB image through an encoder to obtain an RGB feature map, and fuses the normal vector feature map and the RGB feature map through a decoder and a multi-task guided jump fusion module to obtain a fused feature map. The decoder performs feature mapping and image scale restoration on the fused feature map and outputs an RGB segmentation map as the output positive and negative obstacle segmentation results. The RGB segmentation map is used in the loss calculation during the training process. In the normal vector branch, the input of each encoder level is fused with the output of the corresponding decoder level by pixel-level addition and then used as the input of the next level decoder, and also as one of the inputs of the same level multi-task guidance jump fusion module. In the RGB branch, the input of each encoder level is fused with the output of the decoder at the same level by pixel-level addition, and then used as the input of the multi-task guidance jump-connection fusion module at the same level; the output of the multi-task guidance jump-connection fusion module is used as the input of the next level decoder, and the output of the last level multi-task guidance jump-connection fusion module is the output of the positive and negative obstacle segmentation results.

2. The jump-connection fusion method for positive and negative obstacle segmentation based on multi-task guidance according to claim 1, characterized in that, Both the normal vector branch and the RGB branch adopt a five-level encoder-five-level decoder structure. The fifth level encoder of the normal vector branch adopts a Transformer layer.

3. The jump-connection fusion method for positive and negative obstacle segmentation based on multi-task guidance according to claim 2, characterized in that, in, The inputs to the first to fourth level multi-task guidance jumper fusion modules also include the output of the previous level normal vector encoder.

4. The jump-connection fusion method for positive and negative obstacle segmentation based on multi-task guidance according to claim 2, characterized in that, In the RGB branch, the inputs of the fourth-level encoder and the fifth-level encoder are first processed by the channel attention layer to adjust the weight relationship between the channels, and then fused with the output of the same-level decoder via a skip connection, and output to the same-level multi-task guided skip connection fusion module.

5. The jump-connection fusion method for positive and negative obstacle segmentation based on multi-task guidance according to claim 2, characterized in that, The feature fusion process performed by the multi-task guided jump-connection fusion module includes the following steps: For the i-th level multi-task guided skip connection fusion module, the input is the feature map fused by the skip connection from the i-th level decoder output of the normal vector branch. The feature map after skip connections is fused with the output of the i-th level decoder of the RGB branch. At this point, the number of channels in the feature map is C; For feature maps and feature map Subtraction is performed to obtain the residual feature map. ; feature map Feature map and residual feature map By adjusting the number of channels in the convolutional layer to 3, three segmentation images were obtained. , and ,in Then, deep features are extracted through residual connections to form the final residual segmentation map. ; Residual partitioning diagram By adjusting the number of channels C through a convolutional layer, a new... , and as well as Pixel-level multiplication is performed separately to obtain two weighted feature maps. and ,Will , , , as well as The feature maps are concatenated along the channel dimension, and the resulting feature maps are adjusted for the number of channels before being compared with the input feature maps. and Perform skip connections, then pass through a ReLU activation layer to obtain feature maps. ; The residual connection is then passed through two convolution layers, a BN layer, and a ReLU layer, and then connected to the input feature map. and Perform a skip connection, and obtain the result after passing through the ReLU activation layer. ; After four parallel dilated convolutions of different scales, four convolution outputs are obtained; the four convolution outputs are compared with... The layers are then stitched together, and the number of channels is adjusted to C using a convolutional layer. and Perform a final jump connection to obtain the initial output; For the last-stage multi-task guided jump-connection fusion module, the preliminary output is used as the final output; for other multi-task guided jump-connection fusion modules, the preliminary output is added pixel-level to the output of the (i-1)th level encoder of the normal vector to obtain the final output.

6. The jump-connection fusion method for positive and negative obstacle segmentation based on multi-task guidance according to claim 2, characterized in that, The loss calculation during the training process includes the following steps: Downsampling the ground truth map gt to make its resolution consistent with the resolution of the i-th level segmentation map yields the downsampled ground truth map. ; The loss is calculated using the cross-entropy loss function on the normal vector segmentation map output by the last stage decoder in the normal vector branch and the corresponding ground truth map gt. ; The RGB segmentation image output by the last-level multi-task guidance and fusion module of the RGB branch and the corresponding ground truth image (gt) are used to calculate the loss using the cross-entropy loss function. ; Calculated using the cross-entropy loss function The normal vector segmentation map generated by the i-th level multi-task guidance jump-connection fusion module loss ,as well as RGB segmentation map generated by the i-th level multi-task guidance jumper fusion module loss ; Subtract the RGB segmentation map generated in the fusion module Obtain the true residual plot , Residual segmentation map generated by the fusion module Calculate the loss using the cross-entropy loss function. , where i represents the multi-task guided jump-fusion module that calculates the loss after the output of the i-th level decoder; Total loss Represented as: ； Where i takes values ​​from 1 to 5.

Images