RGB-d semantic segmentation method and system based on adaptive context-aware network

Abstract

The application discloses an RGB-D semantic segmentation method and system based on an adaptive context perception network, and first constructs an adaptive context perception network; then uses an RGB branch and a depth branch of an encoder to respectively extract channel features of an RGB image and depth features of a depth map of an image to be processed, performs down-sampling, and obtains feature maps with compressed resolution; next, an adaptive pyramid context module is used to extract information of the encoder, a light-weight decoder is used to up-sample the information from the adaptive pyramid context module, and a light-weight residual unit is used to improve inference speed; finally, three jump connection layers are used to respectively fuse three layers of features from a attention fusion module into three layers of the light-weight decoder, and output a segmentation result after convolution. The application can realize RGB-D semantic segmentation with high segmentation accuracy, fast inference speed and small model parameters.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of image processing and semantic segmentation, and relates to an RGB-D semantic segmentation method and system, in particular to a spatial information guided adaptive context-aware network for realizing the RGB-D semantic segmentation method and system. BACKGROUND

[0002] Efficient RGB-D semantic segmentation technology is widely used in intelligent fields such as autonomous driving, medical image analysis, and robot technology, and it plays a crucial role in the analysis and identification of environmental information. According to previous research, depth information can provide corresponding geometric relationships for objects and scenes, and the spatial information in the depth map can avoid the lack of texture, insufficient lighting, and excessive exposure in image input, to some extent, more useful information can be extracted. However, in practice, due to insufficient camera conditions and external environmental interference, there are still some interference factors such as noise in the depth input, which will obviously have a great impact on the calculation accuracy. Therefore, it is necessary to build an attention network that can capture global context relationships and focus on effective information.

[0003] The most classic structure of semantic segmentation is the encoder-decoder structure, the encoder uses downsampling to extract features, and the decoder uses upsampling to restore the size of the feature map. It is known that most traditional visual methods use multi-layer networks for stacking, which requires a complex backbone to obtain rich semantic information. This not only causes gradient disappearance and network degradation problems, but also has high computational complexity and consumes a lot of computing resources, which will be detrimental to the deployment of the model to edge devices. SUMMARY

[0004] The purpose of the present application is to provide a spatial information guided adaptive context-aware RGB-D semantic segmentation method and system, which helps to solve the problems of noise, large information extraction error, gradient disappearance, network degradation, and large consumption of computing resources in semantic segmentation.

[0005] The technical solution adopted by the method of the present application is: an RGB-D semantic segmentation method based on an adaptive context-aware network, comprising the following steps:

[0006] Step 1: Construct an adaptive context-aware network;

[0007] The adaptive context-aware network includes an encoder of dual-modal input, an attention fusion module, an adaptive pyramid context module (APC), a lightweight decoder (LD), and a skip connection layer.

[0008] The encoder of the dual-mode input includes an RGB branch and a depth branch arranged in parallel, both of which are taken as a backbone of ResNet34-NBt1D and include five layers, the first layer is a 7x7 convolutional layer with a step of 2, followed by a normalization layer and an activation layer, the second layer is a maximum pooling layer with a step of 2, followed by three non-bottleneck blocks, the third to fifth layers are all non-bottleneck blocks, the number of which is 4, 6 and 3 respectively;

[0009] The attention fusion module is composed of a channel attention module and a spatial attention module and is arranged at the end of each layer of the encoder of the dual-mode input; the two feature maps of the RGB branch and the depth branch of the dual-mode are respectively subjected to channel and spatial attention feature enhancement, and then matrix element summation is performed to form a new RGB feature input, and the spatial feature of the depth branch will continue to be transmitted to the next stage; the channel attention module is located in the RGB branch, the information of each channel is compressed into a feature map irrelevant to the size through global average pooling, and then a weight coefficient is calculated through two 1x1 convolutional layers and different activation functions ReLU and Sigmoid, the weight coefficient is multiplied with the initial input matrix element to obtain the channel information enhanced feature; the spatial attention module is located in the depth branch, the stereo feature is obtained through a point-by-point convolutional layer with a convolution kernel size of 1 and a step of 2, then the feature is input into a 1x1, 4x4 and 7x7 spatial pyramid pooling of different scales to obtain three adaptive average pooling features and weights, and the weight value is obtained by cascading output, and the spatial information enhanced stereo feature is obtained by multiplying the initial input matrix element;

[0010] The adaptive pyramid context module first divides the input feature map into sub-regions of different sizes through 1x1 and 5x5 convolution, then performs pooling operation on each sub-region to obtain two fixed-size feature vectors; finally, the two scale feature vectors are concatenated together and input into a 1x1 convolution operation for feature fusion to generate the final feature map as the output of the adaptive pyramid module;

[0011] The lightweight decoder constructs three branches through a convolution kernel with a size of 1, the middle m-branch and the bottom d-branch are both subjected to symmetric convolution (3x1, 1x3), channel attention module, symmetric convolution (3x1, 1x3) processing in sequence, then the pixel of the two layers of features is summed, and the summed feature is subjected to 1x1 convolution again; wherein, the symmetric convolution in the middle adopts separable convolution for expanding the receptive field to obtain long-distance feature information, and the dilated rate dr is 1; the bottom convolution is a general convolution responsible for extracting local and short-distance feature information; the feature obtained by summing the middle and bottom branches is fused with the original feature of the upper branch, and the mixed feature is further integrated through channel shuffling, and then the feature is restored to the original image feature through bilinear upsampling;

[0012] The jump connection layer doubles the input channel number through a convolution with a kernel size of 1, and the size remains unchanged;

[0013] Step 2: using the RGB branch and the depth branch of the encoder, respectively extracting the channel features of the RGB image and the depth features of the depth map of the image to be processed, and performing down-sampling to obtain a feature map with a compressed resolution;

[0014] Step 3: extracting information from the encoder using the adaptive pyramid context module, using a lightweight decoder to bilinearly up-sample the information of the adaptive pyramid context module, outputting a multi-scale feature segmentation result through pyramid supervision, and additionally using a lightweight residual unit to transfer information to the next stage to improve inference speed;

[0015] Step 4: using three jump connection layers to make the three layers of features from the attention fusion module consistent with the channel number of the three lightweight decoding units through 1x1 convolution, and keeping the rest of the features unchanged, so as to facilitate the lightweight decoding unit to realize the fusion of low-level features and high-level features, solve the problems of gradient explosion and gradient disappearance in the training process, and output the segmentation result after convolution.

[0016] The technical scheme of the system of the present application is: an RGB-D semantic segmentation system based on an adaptive context perception network, comprising the following modules:

[0017] The first module is used for constructing an adaptive context perception network;

[0018] The adaptive context perception network comprises an encoder of dual-mode input, an attention fusion module, an adaptive pyramid context module (APC), a lightweight decoder (LD), and a jump connection layer;

[0019] The encoder of dual-mode input comprises an RGB branch and a depth branch arranged in parallel; both take ResNet34-NBt1D as the backbone, and comprise five layers; the first layer is a 7x7 convolution layer with a step of 2, followed by a normalization layer and an activation layer; the second layer is a maximum pooling layer with a step of 2, followed by three non-bottleneck blocks; the third to fifth layers are all non-bottleneck blocks, and the number of each layer is 4, 6 and 3, respectively;

[0020] The attention fusion module is composed of a channel attention module and a spatial attention module, and is arranged at the end of each layer of the encoder of the dual-modal input; two feature maps of the RGB branch and the depth branch of the dual-modal are respectively subjected to channel and spatial attention feature enhancement, and then matrix element summation is performed to form a new RGB feature input, and the spatial feature of the depth branch will continue to be transmitted to the next stage; the channel attention module is located in the RGB branch, the information of each channel is compressed into a feature map irrelevant to the size by global average pooling, and then a weight coefficient is calculated through two 1x1 convolution layers and different activation functions ReLU and Sigmoid, the weight coefficient is multiplied with the initial input matrix element to obtain the channel information enhanced feature; the spatial attention module is located in the depth branch, a stereo feature is obtained through a point-by-point convolution layer with a convolution kernel size of 1 and a step of 2, then the feature is input into a 1x1, 4x4 and 7x7 spatial pyramid pooling with different scales, three adaptive average pooling features are obtained and weighted, and the weight value is obtained through cascaded output, and the stereo feature with enhanced spatial information is obtained by multiplying the weight value with the initial input matrix element;

[0021] The adaptive pyramid context module first divides the input feature map into sub-regions of different sizes through 1x1 and 5x5 convolution, then performs pooling operation on each sub-region to obtain two fixed-size feature vectors; finally, the two scale feature vectors are concatenated together and input into a 1x1 convolution operation for feature fusion to generate the final feature map as the output of the adaptive pyramid module;

[0022] The light-weight decoder constructs three branches through a convolution kernel with a size of 1, the middle m-branch and the bottom d-branch are both subjected to symmetric convolution (3x1, 1x3), channel attention module, symmetric convolution (3x1, 1x3) processing in sequence, then the pixel of the two layers of features is summed, and the summed features are subjected to 1x1 convolution again; wherein, the symmetric convolution of the middle m-branch adopts separable convolution, which is used to expand the receptive field and obtain long-distance feature information, and the dilated rate dr is 1; the convolution of the bottom d-branch is general convolution, which is responsible for extracting local and short-distance feature information; the features summed by the middle and bottom branches are fused with the original features of the upper branch, and further integrated with different information through channel shuffling, and the shuffled features are restored to the original image features through bilinear upsampling;

[0023] The skip connection layer increases the input channel number by one through a convolution with a convolution kernel size of 1, and the size remains unchanged;

[0024] The second module is configured to use the RGB branch and the depth branch of the encoder to respectively extract the channel features of the RGB image and the depth features of the depth map of the image to be processed, and perform down-sampling to obtain feature maps with compressed resolution;

[0025] The third module is used for extracting information from the encoder by using the adaptive pyramid context module, bilinear up-sampling the information of the adaptive pyramid context module using a lightweight decoder, outputting a multi-scale feature segmentation result through pyramid supervision, and using a lightweight residual unit to transmit information to the next stage to improve inference speed.

[0026] The fourth module is used for connecting three layers of features from the attention fusion module through three skip connection layers, making the channel number of the three layers of features consistent with that of three lightweight decoding units through 1*1 convolution, keeping the rest of the features unchanged, facilitating the fusion of low-level features and high-level features by the lightweight decoding unit, solving the problems of gradient explosion and gradient disappearance in the training process, and outputting a segmentation result after convolution.

[0027] The innovations of the present application include:

[0028] (1) The encoder is composed of ResNet34-NBt1D and is divided into an image branch and a depth branch, and the two branches are connected by an attention fusion module.

[0029] (2) In the adaptive pyramid context module, a plurality of adaptive semantic modules in parallel are used to process the feature maps from the attention fusion module, and the output is obtained after convolution through a fully connected layer after integration.

[0030] (3) In the lightweight decoder, the features are constructed into three branches through a convolution kernel with a size of 1, the middle m-branch and the bottom d-branch are processed through symmetric convolution (3*1, 1*3), channel attention module, symmetric convolution (3*1, 1*3) in turn, and then the pixel sum of the two layers of features is calculated, and the sum is further convolved through a 1*1 convolution; wherein the symmetric convolution of the middle m-branch adopts separable convolution, which is used to expand the receptive field and obtain long-distance feature information, and the dilated rate dr is 1; the convolution of the bottom d-branch is a general convolution, which is responsible for extracting local and short-distance feature information; the features summed by the middle and bottom branches are fused with the original features of the upper branch, and further integrated through channel shuffling to integrate different information, and the features after shuffling are bilinearly up-sampled to restore the original image features.

[0031] (4) The skip connection layer connects the feature information of the encoder to the decoder.

[0032] The present application has the advantages of high segmentation accuracy, fast inference speed and small model parameters for RGB-D semantic segmentation. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 The spatial information guided adaptive context perception network structure diagram of the embodiment of the present application;

[0034] Figure 2A lightweight encoder structure diagram of an embodiment of the present application;

[0035] Figure 3 The visualization result diagram of the encoder fusion of an embodiment of the present application; the first column includes (a) RGB, (b) Layer4_rgb_feature map, (c) Layer4_rgb_attention map; the second column is a feature map (FM), including (d) depth map, (e) Layer4_depth_feature map, (f) Layer4_depth_attention map; the last column is an attention map (AM) including (g) semantic segmentation map, (h) Layer4_feature map, (i) Layer4_attention map. DETAILED DESCRIPTION

[0036] In order to facilitate those skilled in the art to understand and implement the present application, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application.

[0037] The present application provides an RGB-D semantic segmentation method based on an adaptive context perception network, comprising the following steps:

[0038] Step 1: Construct an adaptive context perception network;

[0039] See Figure 1 The adaptive context perception network of the present embodiment includes an encoder of dual-mode input, an attention fusion module, an adaptive pyramid context module (APC), a lightweight decoder (LD), and a skip connection layer;

[0040] The encoder of dual-mode input of the present embodiment includes an RGB branch and a depth branch arranged in parallel; both take ResNet34-NBt1D as the backbone, including five layers, the first layer is a 7x7 convolutional layer with a step of 2, followed by a normalization layer and an activation layer; the second layer is a maximum pooling layer with a step of 2, followed by three non-bottleneck blocks; the third to fifth layers are all non-bottleneck blocks, the number of which is 4, 6, and 3 respectively;

[0041] The attention fusion module of the embodiment is composed of a channel attention module and a spatial attention module, and is arranged at the end of each layer of an encoder of the dual-modal input; two feature maps of the RGB branch and the depth branch of the dual-modal are respectively subjected to channel and spatial attention feature enhancement, and then matrix element summation is performed to form a new RGB feature input, and the spatial feature of the depth branch will continue to be transmitted to the next stage; the channel attention module is located in the RGB branch, information of each channel is compressed into a feature map irrelevant to size by global average pooling, and then a weight coefficient is calculated by two 1x1 convolution layers and different activation functions ReLU and Sigmoid; for the purpose of simplifying the network, the expression of the activation function is omitted in the figure, but in actual application, they are necessary and are often used in combination to improve the model performance, so that the training is more stable and efficient; the weight coefficient is multiplied with the initial input matrix element to obtain channel information enhanced features; the spatial attention module is located in the depth branch, and stereo features are obtained by a point-by-point convolution layer with a convolution kernel size of 1 and a step of 2, and then 1x1, 4x4 and 7x7 three different scale spatial pyramid pooling is performed on the feature input to obtain three adaptive average pooling features and weighting, and the weight value is obtained by cascaded output, and the stereo features with enhanced spatial information are obtained by multiplying the weight value with the initial input matrix element;

[0042] The adaptive pyramid context module of the embodiment first divides the input feature map into sub-regions of different sizes by 1x1 and 5x5 convolution, and then performs pooling operation on each sub-region to obtain two fixed-size feature vectors; different scale pooling operations can capture different size context information. Finally, the two scale feature vectors are concatenated together and input into a 1x1 convolution operation for feature fusion to generate the final feature map as the output of the adaptive pyramid module;

[0043] See Figure 2 The lightweight decoder of the embodiment constructs three branches by a convolution kernel with a size of 1, the middle m-branch and the bottom d-branch are subjected to symmetric convolution (3x1, 1x3), channel attention module, symmetric convolution (3x1, 1x3) processing in sequence, and then the feature pixels of the two layers are summed, and the summed result is subjected to 1x1 convolution again; wherein the symmetric convolution of the middle m-branch adopts separable convolution for expanding the receptive field and obtaining long-distance feature information, and the dilation rate dr is 1; the convolution of the bottom d-branch is general convolution, which is responsible for extracting local and short-distance feature information; the features summed by the middle and bottom branches are fused with the original features of the upper branch, and the mixed features are further integrated by channel shuffling, and then the mixed features are bilinearly upsampled to restore the original image features;

[0044] The skip connection layer of the embodiment increases the input channel number by one time by convolution with a convolution kernel size of 1, and the size remains unchanged;

[0045] Step 2: using the RGB branch and depth branch of the encoder, respectively extracting the channel features of the RGB image and the depth features of the depth map of the image to be processed, and performing down-sampling to obtain feature maps with compressed resolution;

[0046] In this embodiment, the spatial pyramid attention module (SPA) uses 1x1, 4x4, and 7x7 average pooling to obtain strong structural regularity information, balance the relationship between structural information and structural regularization, and capture more feature representations and structural information. The combination of the three can not only maintain feature representation, but also inherit the advantages of global average pooling, avoiding model overfitting. The spatial attention pyramid module of the lth element is:

[0047] S(x l )=C(R7(P(x l )),R4(P(x l )),R1(P(x l ))) (1)

[0048] where X=[x1,x2,…,x l ] is the input feature, denotes the algebraic field, C, H, and W represent the number of channels, the height of the input image, and the width, respectively; C(·) represents the concatenation layer, P(·) and R(·) represent the pooling operator and the vector resizing operator, respectively; 1, 4, and 7 represent 1x1, 4x4, and 7x7 average pooling convolution;

[0049] To obtain the final attention result, element-wise multiplication M(·) is performed:

[0050] M(x l )=x l σ(F f (F f (U l ))) (2)

[0051]

[0052] where Q(x l ) represents the channel enhanced attention module (extraction-compression SE), F f (·) is a fully connected layer that realizes end-to-end information learning of the channel and spatial branch local classification, and the sigmoid function σ is used as the activation layer function;

[0053] Fusing the information processed by the spatial pyramid attention module and the channel enhanced attention module into the RGB input branch is beneficial to the features with rich semantic information participating in subsequent network processing, and the fused feature map is generated as:

[0054]

[0055] where the subscript C represents the channel branch where the RGB input is located, and the subscript S represents the depth branch where the depth input is located; M c ()、M s () respectively represent element summation of channel attention features and element summation of spatial attention features.

[0056] Step 3: Use the adaptive pyramid context module (APC) to extract the information of the encoder, use the lightweight decoder (LD) to up-sample the information from the adaptive pyramid context module, output the multi-scale feature segmentation result through pyramid supervision, and use the lightweight residual unit (LRU) to improve the inference speed;

[0057] The bilinear up-sampling of the embodiment increases the resolution of the depth features and the RGB features to the same size as the shortcut path features after passing through the lightweight residual unit;

[0058] The lightweight residual unit is composed of a normal convolution layer Z d，k and a dilated depth separable convolution layer Z m，k , which aims to reduce the calculation cost and obtain deeper network features; where the lower branch is denoted as d, the middle branch is denoted as m, and the value k represents the number of times the feature map passes through the unpaired convolution;

[0059] The formula of the lightweight residual unit is as follows:

[0060] z H，2 (y l )=A2(V ac (A1(W1(y l )))) (5)

[0061] where H represents the d-branch or the m-branch, y l is the decoder feature input, represents the algebraic field, C, H, and W respectively represent the number of channels, the height and width of the input image; W1 represents the weight of the 1x1 convolution layer, and the channel attention module (CAM) V ac (·) is used to enhance semantic expression; the asymmetric convolution A K (·) (1x3, 3x1) can approximate the existing convolution, ensuring the same calculation amount to speed up model inference and compress model size; K represents the number of times the feature map passes through the unpaired convolution;

[0062] The lightweight decoder (LD) integrates long-distance Z m，2 and short-distance Z d，2Features, as shown below:

[0063]

[0064] where Z(·) represents the output of the lower branch d or the middle branch m; after summing these branches, the channel shuffle f(·) realizes the feature communication between them.

[0065] Step 4: Use three jump connection layers to connect the three layers of features from the attention fusion module, pass through 1x1 convolution, make the channel number consistent with the three lightweight decoding units, and keep the rest of the features unchanged, so as to facilitate the lightweight decoding unit to realize the fusion of low-level features and high-level features, solve the problems of gradient explosion and gradient disappearance in the training process, and output the segmentation result after convolution.

[0066] The adaptive context-aware network of the embodiment is a trained adaptive context-aware network; during training, the publicly available Pytorch is used for training, the batch size is set to 8, and the epoch number is set to 500. The SGD optimizer with a momentum of 0.9 is used for the NYUv2 and SUN RGB-D data sets, and the Adam optimizer with learning rates of 0.00125, 0.0025, 0.005, 0.01, 0.02, 0.04, 0.0001 and 0.0004 is further used. A general data enhancement strategy is adopted, including random scaling, random horizontal flipping and random cropping.

[0067] During the training process, the spatial information guided adaptive context-aware network is trained using the data set, and the cross entropy (CE) function L p，q Calculate the loss;

[0068]

[0069] where n represents the number of categories, pi is the occurrence probability of each category of the identified real, and qi is the occurrence probability of each category of the predicted.

[0070] See Figure 3Figure 4 is a schematic diagram of the fusion visualization result of the fourth layer encoder Layer4 of the embodiment; the first column (a) RGB and (d) depth map are respectively the double-branch input of the neural network, and (g) is the semantic segmentation result of the scheme; the second column is the fusion feature map (FM) of the fourth layer encoder Layer4, including (b) Layer4_rgb_feature map, (e) Layer4_depth_feature map, and (h) Layer4_feature map, which respectively represent the feature representation of the double-branch input before fusion and the feature representation after final fusion, indicating that the attention fusion module combined with depth information can more comprehensively describe the object information and spatial information in the image; the last column is the fusion attention map (AM) of the fourth layer encoder Layer4, including (c) Layer4_rgb_attention map, (f) Layer4_depth_attention map, and (i) Layer4_attention map, which respectively represent the attention representation of the double-branch input before fusion and the attention representation after final fusion, indicating that the important points emphasized by the RGB and depth branches are different, and the outline and position information region concerned by both can be comprehensively obtained after the attention fusion module, thereby obtaining better segmentation performance. The attention weight comes from the feature map.

[0071] The present application has the following advantages:

[0072] (1) Two branches of the encoder are used to extract channel features from the RGB image and depth features from the depth map, respectively, and down-sampling is performed.

[0073] (2) The extracted depth features are connected to the RGB branch through an attention fusion module. The attention fusion module includes a channel enhancement attention module and a spatial pyramid attention module, which respectively focus on the channel information in the RGB image and the spatial local information in the depth map; in the spatial pyramid attention module, the depth input branch is adaptively averaged to gather a feature map, the output of the spatial pyramid attention module is added to the output of the channel enhancement attention module, and the position guiding effect of the depth information on the channel information is strengthened.

[0074] (3) The encoder information is extracted by using an adaptive pyramid context module, and the main structure is to use multiple adaptive semantic modules in parallel to process the feature map from the attention fusion module, and the output is obtained after integration and full connection layer convolution.

[0075] (4) The output information is up-sampled by a lightweight decoder, and a shortcut path is added to the bilinear up-sampling operation part of the lightweight decoder to add the feature map to the main path of the corresponding lightweight residual unit, and the lightweight residual unit is composed of an ordinary convolution layer for supplementing information continuity and obtaining close-range feature information and an expanded deep separable convolution layer to reduce the calculation cost and obtain deeper network features.

[0076] (5) Train the spatial information guided adaptive context-aware network using the dataset and calculate the loss according to the cross-entropy function.

[0077] It should be understood that the above description of the preferred embodiments is rather detailed and is not intended to be considered as limiting the scope of patent protection of the present application. Any person skilled in the art, under the guidance of the present application, can make substitutions or modifications without departing from the scope of protection of the present application, and all such substitutions or modifications shall fall within the scope of protection of the present application. The scope of protection of the present application shall be subject to the appended claims.

Claims

1. An RGB-D semantic segmentation method based on adaptive context-aware network, characterized in that, The method comprises the following steps: Step 1: constructing an adaptive context-aware network; The adaptive context-aware network comprises an encoder of dual-modal input, an attention fusion module, an adaptive pyramid context module, a lightweight decoder and a skip connection layer; The encoder of dual-modal input comprises an RGB branch and a depth branch arranged in parallel; both are taken as a backbone of ResNet34-NBt1D and comprise five layers; the first layer is a convolutional layer, followed by a normalization layer and an activation layer; The second layer is a maximum pooling layer, followed by three non-bottleneck blocks; the third to fifth layers are all non-bottleneck blocks; The attention fusion module is composed of a channel attention module and a spatial attention module and is arranged at the end of each layer of the encoder of dual-modal input; the two feature maps of the RGB branch and the depth branch of the dual-modal input are respectively subjected to channel and spatial attention feature enhancement, and then matrix element summation is performed to form a new RGB feature input; the spatial feature of the depth branch will continue to be transmitted to the next stage; the channel attention module is located in the RGB branch; the information of each channel is compressed into a feature map irrelevant to the size by global average pooling, and then a weight coefficient is calculated through two convolutional layers and different activation functions ReLU and Sigmoid; the weight coefficient is multiplied with the initial input matrix element to obtain the channel information enhanced feature; the spatial attention module is located in the depth branch; the stereo feature is obtained through a point-by-point convolutional layer, and then three spatial pyramid pooling of different scales are performed on the feature input to obtain three adaptive average pooling features and weights; the weight values are obtained by concatenation output; the weight values are multiplied with the initial input matrix element to obtain the spatial information enhanced stereo feature; The adaptive pyramid context module firstly divides the input feature map into sub-regions of different sizes through two convolutional operations, and then performs pooling operation on each sub-region to obtain two fixed-size feature vectors; finally, the two scale feature vectors are concatenated together and input into a convolutional operation for feature fusion to generate the final feature map as the output of the adaptive pyramid module; The lightweight decoder constructs three branches through a convolutional kernel; the middle m-branch and the bottom d-branch are subjected to symmetric convolution, channel attention module and symmetric convolution processing in sequence, and then the two layers of feature pixels are summed; the summed feature is subjected to convolution again; the symmetric convolution of the middle m-branch adopts separable convolution for expanding the receptive field and obtaining long-distance feature information; the convolution of the bottom d-branch is general convolution for extracting local and short-distance feature information; the summed feature of the middle and bottom branches is fused with the original feature of the upper branch, and the mixed feature is further integrated through channel shuffling; the mixed feature is subjected to bilinear upsampling to restore the original image feature; The skip connection layer increases the input channel number by one through convolution, and the size remains unchanged; Step 2: using the RGB branch and the depth branch of the encoder, the channel features of the RGB image and the depth features of the depth map of the image to be processed are extracted and down-sampled to obtain feature maps with compressed resolution; Step 3: Extract information from the encoder using the adaptive pyramid context module, bilinearly upsample the information of the adaptive pyramid context module using a lightweight decoder, output multi-scale feature segmentation results through pyramid supervision, and use a lightweight residual unit to transfer information to the next stage to improve inference speed; Step 4: Use three jump connection layers to connect the three layers of features from the attention fusion module, and use 1x1 convolution to make the channel number of the three lightweight decoding units consistent, while keeping the rest of the features unchanged, so that the lightweight decoding unit can fuse bottom features and high-level features, and output the segmentation result after convolution. 2.The method of claim 1, wherein the adaptive context-aware network-based RGB-D semantic segmentation method is characterized by: In step 1, the encoder of the dual-modal input, the first layer is a 7x7 convolutional layer with a stride of 2; The second layer is a max-pooling layer with a stride of 2; the third to fifth layers are all non-bottleneck blocks, with quantities of 4, 6, and 3 respectively; The channel attention module includes two 1x1 convolutional layers; the spatial attention module includes a pointwise convolutional layer with a kernel size of 1 and a stride of 2, and a spatial pyramid pooling with kernel sizes of 1x1, 4x4, and 7x7; The adaptive pyramid context module includes two convolutional layers with sizes of 1x1 and 5x5, which are divided into sub-regions of different sizes, then each sub-region is pooled to obtain two fixed-size feature vectors; finally, the two scale feature vectors are concatenated and fed into a 1x1 convolution operation for feature fusion; The lightweight decoder constructs three branches through a convolution kernel with a size of 1, the middle m-branch and the bottom d-branch are processed through symmetric convolution (3x1, 1x3), channel attention module, symmetric convolution (3x1, 1x3) in turn, then the pixel sum of the two layers of features is calculated, and the sum is passed through a 1x1 convolution again; The jump connection layer includes a convolution with a kernel size of 1.

3. The adaptive context-aware network based RGB-D semantic segmentation method according to claim 1, characterized in that: In step 1, the spatial attention module uses 1x1, 4x4, and 7x7 average pooling to obtain structural regularity information, balanced structure information, and the relationship between structure regularization, respectively; wherein the spatial attention pyramid module of the first element is: l ​ （1） wherein, as input features, , denotes an algebraic field, C, H, W denote the number of channels, height and width of the input picture respectively; C denotes a concatenation layer, P and R( ) denote a pooling operator and a vector resizing operator respectively; 1, 4, 7 denote 1x1, 4x4, 7x7 average pooling convolutions;​​ To obtain the final attention result, an element-wise multiplication is performed M ( ) （2） （3） wherein, denotes a channel attention module, The channel and spatial branch local classification end-to-end information learning is realized, and a sigmoid function σ is used as an activation layer function. The information from the spatial attention module and the channel attention module is fused into the RGB input branch, and the fused feature map is generated as: （4） wherein the lower index C represents the channel branch where the RGB input is located, and the lower index S represents the depth branch where the depth input is located; , respectively represent element summation of the channel attention feature and element summation of the spatial attention feature. 4.The method of claim 1, wherein: In step 3, the bilinear up-sampling, the depth feature and the RGB feature are up-sampled to the same size as the shortcut path feature after passing through the lightweight residual unit; The lightweight residual unit is composed of a normal convolution layer for supplementing information continuity and obtaining close-range feature information and an expanded depth separable convolution layer , wherein the lower branch is denoted as d , and the middle branch is denoted as m , k The value represents the number of times the feature map passes through the unpaired convolution. The lightweight residual unit formula is expressed as follows: （5） where H represents d - branch or m - branch, is the decoder feature input, , represents an algebraic field, C, H, W respectively represent the number of channels, the height and width of the input picture; represents the weight of the 1x1 convolution layer, using the channel attention module to enhance semantic expression; asymmetric convolution can approximate the existing convolution, ensure the same calculation amount, speed up the model inference and compress the model size; K represents the number of times the feature mapping passes through the unpaired convolution; The lightweight decoder integrates long distance and short distance features as follows: （6） wherein, represents the output of the lower branch d or middle branch m ; after summing these branches, the channel shuffle implements the feature communication between them.

5. The method of claim 1-4, wherein the method is based on an adaptive context-aware network. The adaptive context-aware network is a trained adaptive context-aware network; During the training process, the spatial information guided adaptive context-aware network is trained by using the data set, and the cross entropy function is used to calculate the loss ​ （7） wherein, represents the number of classes, is the occurrence probability of each class that identifies the real, is the occurrence probability of each class that is predicted.

6. An adaptive context-aware network based RGB-D semantic segmentation system, characterized in that, It includes the following modules: The first module is used to build an adaptive context-aware network; The adaptive context-aware network includes a dual-modal input encoder, an attention fusion module, an adaptive pyramid context module, a lightweight decoder, and a jump connection layer; The dual-modal input encoder includes an RGB branch and a depth branch arranged in parallel; both use ResNet34-NBt1D as the backbone, including five layers, the first layer is a convolutional layer followed by a normalization layer and an activation layer; The second layer is a max-pooling layer followed by three non-bottleneck blocks; the third to fifth layers are all non-bottleneck blocks; The attention fusion module is composed of a channel attention module and a spatial attention module, and is arranged at the end of each layer of the encoder of the dual-modal input; two feature maps of the RGB branch and the depth branch of the dual-modal are respectively subjected to channel and spatial attention feature enhancement, and then matrix element summation is performed to form a new RGB feature input, and the spatial feature of the depth branch will continue to be transmitted to the next stage; the channel attention module is located in the RGB branch, compresses the information of each channel into a feature map irrelevant to the size through global average pooling, calculates a weight coefficient through two convolutional layers and different activation functions ReLU and Sigmoid, and obtains channel information enhanced features by multiplying the weight coefficient with the initial input matrix element; the spatial attention module is located in the depth branch, obtains stereo features through a point-by-point convolutional layer, obtains three adaptive average pooling features by inputting the features into three different scale spatial pyramid pooling, weights the three adaptive average pooling features, concatenates the output to obtain a weight value, and obtains spatial information enhanced stereo features by multiplying the weight value with the initial input matrix element; The adaptive pyramid context module firstly divides the input feature map into sub-regions of different sizes through two convolutional operations, then performs a pooling operation on each sub-region to obtain two fixed-size feature vectors, and finally, the two scale feature vectors are concatenated together and input into a convolutional operation for feature fusion to generate a final feature map as the output of the adaptive pyramid context module; The lightweight decoder is configured to construct three branches through a convolutional kernel, the middle m-branch and the bottom d-branch are both subjected to symmetric convolution, a channel attention module and symmetric convolution processing in sequence, then the features of the two layers are summed, and the summed features are subjected to convolution again; wherein the symmetric convolution of the middle m-branch adopts separable convolution for expanding the receptive field and obtaining long-distance feature information; the convolution of the bottom d-branch is general convolution, which is responsible for extracting local and short-distance feature information; the features of the middle and bottom branches are summed and fused with the original features of the upper branch, and the mixed features are further integrated through channel shuffling, and then the shuffled features are subjected to bilinear upsampling to restore the original image features; The skip connection layer is configured to increase the number of input channels by one through convolution, and the size remains unchanged; The second module is configured to use the RGB branch and the depth branch of the encoder to respectively extract channel features of an RGB image and depth features of a depth map of the image to be processed, and perform down-sampling to obtain feature maps with compressed resolution; The third module is configured to use the adaptive pyramid context module to extract information from the encoder, use the lightweight decoder to perform bilinear upsampling on the information of the adaptive pyramid context module, output a multi-scale feature segmentation result through pyramid supervision, and additionally use the lightweight residual unit to transmit information to the next stage to improve the inference speed. The fourth module is configured to utilize three skip connection layers to connect three layers of features from the attention fusion module, pass the three layers of features through 1*1 convolution, make the number of channels of the three layers of features consistent with the number of channels of the three lightweight decoding units, keep the rest of the features unchanged, facilitate the lightweight decoding units to realize fusion of bottom layer features and high layer features, and output a segmentation result after the convolution. 7.The adaptive context-aware network based RGB-D semantic segmentation system according to claim 6, characterized in that: In the first module, the encoder of the dual-modal input is a 7*7 convolutional layer with a step of 2; The second layer is a maximum pooling layer with a step of 2; the third to fifth layers are all non-bottleneck blocks, and the number of the non-bottleneck blocks is 4, 6 and 3 respectively; The channel attention module includes two 1*1 convolutional layers; the spatial attention module includes a point-by-point convolutional layer with a convolution kernel size of 1 and a step of 2, and a spatial pyramid pooling with three different scales of 1*1, 4*4 and 7*7; The adaptive pyramid context module includes two convolutional layers with sizes of 1*1 and 5*5, which are divided into sub-regions with different sizes, and then a pooling operation is performed on each sub-region to obtain two fixed-size feature vectors; finally, the two scale feature vectors are concatenated together and input into a 1*1 convolution operation for feature fusion; The lightweight decoder includes three branches constructed by a convolution kernel with a size of 1, the middle m-branch and the bottom d-branch are both sequentially subjected to symmetric convolution (3*1, 1*3), channel attention module, symmetric convolution (3*1, 1*3) processing, and then the pixel values of the two layers of features are summed, and the summed features are subjected to 1*1 convolution again; The skip connection layer includes a convolution with a convolution kernel size of 1.

8. The adaptive context-aware network based RGB-D semantic segmentation system of claim 6, wherein: In the first module, the spatial attention module uses 1x1, 4x4, and 7x7 average pooling to obtain structural regularity information, balanced structure information, and the relationship between structure regularization, respectively; wherein the spatial attention pyramid module of the first element is: l ​ （1） wherein, as input features, , denotes an algebraic field, C, H, W denote the number of channels, height and width of the input picture respectively; C ( ) denotes a concatenation layer, P ( ) and R( ) denote a pooling operator and a vector resizing operator respectively; 1, 4, 7 denote 1x1, 4x4, 7x7 average pooling convolutions; To obtain the final attention result, an element-wise multiplication is performed M ( ) （2） （3） wherein, denotes a channel attention module, The channel and spatial branch local classification end-to-end information learning is realized, and a sigmoid function σ is used as an activation layer function. The information passing through the spatial attention module and the channel attention module is fused into the RGB input branch, and the fused feature map is generated as: （4） where the subscript C represents the channel branch where the RGB input is located, and the subscript S represents the depth branch where the depth input is located. , respectively represent element summation of the channel attention feature and element summation of the spatial attention feature. 9.The adaptive context-aware network based RGB-D semantic segmentation method of claim 6, wherein: In the third module, the bilinear upsampling is configured to make the resolution of the depth feature and the RGB feature rise to the same size as the feature of the shortcut path after the lightweight residual unit; The lightweight residual unit is composed of a normal convolution layer for supplementing information continuity and obtaining close-range feature information and an expanded depth separable convolution layer , wherein the lower branch is denoted as d , and the middle branch is denoted as m , k The value represents the number of times the feature map passes through the unpaired convolution. The lightweight residual unit is expressed by the following formula: （5） where H represents d a branch or m a branch, is input to the decoder feature, , represents an algebraic field, C, H, W respectively represent the number of channels, the height and the width of the input picture; represents the weight of the 1x1 convolution layer, using the channel attention module to enhance semantic expression; the asymmetric convolution can approximate the existing convolution, ensuring the same calculation amount, accelerating model inference and compressing model size; K represents the number of times of feature mapping through the unpaired convolution; The lightweight decoder integrates long distance and short distance features as follows: （6） wherein, represents the output of the lower branch d or middle branch m ; after summing these branches, the channel shuffle implements the feature communication between them.

10. The adaptive context-aware network based RGB-D semantic segmentation method according to any one of claims 6-9, characterized in that: The adaptive context-aware network is a trained adaptive context-aware network. During the training process, the spatial information guided adaptive context-aware network is trained by using the data set, and the cross entropy function is used to calculate the loss loss; （7） wherein, denotes the number of classes, is the probability of occurrence of each class that is identified as real, is the probability of occurrence of each class that is predicted.

Images