Indoor scene semantic segmentation method based on improved fully convolutional neural network

Abstract

The present invention discloses a method for semantic segmentation of indoor scenes based on an improved full convolutional neural network. First, a convolutional neural network is constructed, wherein the hidden layer includes 5 neural network blocks, 5 feature re-extraction convolutional layer blocks, 5 block attention convolutional blocks, 12 fusion layers, and 4 upsampling layers; the original indoor scene image is input into the convolutional neural network for training to obtain a corresponding semantic segmentation prediction map; then, the loss function value between the set consisting of the semantic segmentation prediction map corresponding to the original indoor scene image and the set consisting of the one-hot encoded image processed from the corresponding real semantic segmentation image is calculated to obtain the optimal weight vector and bias term of the convolutional neural network classification training model; the indoor scene image to be semantically segmented is input into the trained convolutional neural network classification training model to obtain a predicted semantic segmentation image. The advantage of the present invention is that the semantic segmentation efficiency and accuracy of indoor scene images are improved.

Description

Technical Field

[0001] The present invention relates to a deep learning semantic segmentation method, and in particular to an indoor scene semantic segmentation method based on an improved fully convolutional neural network. Background Art

[0002] Image semantic segmentation is one of the most challenging tasks in computer vision and plays a key role in applications such as autonomous driving, medical image analysis, virtual reality, and human-computer interaction. The core purpose of semantic segmentation is to give a category label to each pixel in an image and determine which category the pixel belongs to. Since semantic segmentation datasets generally involve indoor or outdoor scenes and there are many segmented objects, it is essentially a multi-classification problem.

[0003] From the perspective of supervised learning, image semantic segmentation can be divided into three types: fully supervised, semi-supervised, and unsupervised. However, from the perspective of operability and theoretical application, the current mainstream models mostly use the fully supervised type, and a few use the semi-supervised type because it is easier to implement and the model is easier to train.

[0004] In terms of model application, based on the emergence and development of fully convolutional neural networks, the application of fully convolutional neural networks has achieved superior performance and segmentation effects in image semantic segmentation tasks, but there are still many deficiencies and defects, such as large number of parameters, a large amount of redundant information, insufficient feature expression extraction, etc. Therefore, there is still a lot of room for improvement in the image semantic segmentation model based on fully convolutional neural networks. Proposing and training better models based on the characteristics of the image itself, the structure of the model, and the operating principles of the biological human visual system has become the current and future development goal in the field of image semantic segmentation. Summary of the invention

[0005] In order to solve the problems in the background technology, the present invention provides a semantic segmentation method for indoor scenes based on an improved fully convolutional neural network, which obtains the model design direction and ideas from aspects such as semantic feature expression and the operation principle of the human visual system, improves the traditional fully convolutional neural network, and effectively improves the image segmentation performance.

[0006] The technical solution adopted by the present invention comprises the following steps:

[0007] Step 1: Select Q pairs of original indoor scene images and their corresponding real semantic segmentation images, and form a training set with all the original indoor scene images and their corresponding real semantic segmentation images; each pair of original indoor scene images includes an original indoor scene color image and an original indoor scene depth image, and the real semantic segmentation images in the training set are processed into 41 one-hot encoded images using one-hot encoding technology;

[0008] Step 2: Construct a convolutional neural network classification training model: The convolutional neural network classification training model includes an input layer, a hidden layer, and an output layer; the input layer includes a color image input layer and a depth image input layer; the hidden layer includes a color image processing module and a depth image processing module; the color image processing module and the depth image processing module have symmetrical structures, each including five neural network blocks, five feature re-extraction convolution blocks, and ten fusion layers; the hidden layer also includes five block attention convolution blocks, four upsampling layers, and two fusion layers;

[0009] Step 3: Input the training set into the convolutional neural network classification training model in step 2 for training. During the training process, each iterative training process obtains 41 semantic segmentation prediction images corresponding to each pair of original indoor scene images, and calculates the loss function value between the set of 41 semantic segmentation prediction images and the set of 41 one-hot encoded images corresponding to the true semantic segmentation images;

[0010] The loss function value is obtained using categorical cross entropy;

[0011] Step 4: Repeat step 3 for a total of V times, and obtain a total of Q×V loss function values; then find the minimum loss function value from the Q×V loss function values, and use the weight vector and bias item corresponding to the minimum loss function value as the optimal weight vector and optimal bias item of the convolutional neural network classification training model, thereby completing the training of the convolutional neural network classification training model;

[0012] Step 5: Use the convolutional neural network classification training model obtained after training to perform prediction processing on the indoor scene image to be predicted, and output the corresponding predicted semantic segmentation image to achieve semantic segmentation of the indoor scene image.

[0013] The step 2) is specifically as follows:

[0014] The color image input layer and the depth image input layer are respectively input into the first neural network block in the color image processing module and the depth image processing module;

[0015] The structure of the color image processing module and the depth image processing module is the same, specifically:

[0016] One of the outputs of the first neural network block is input into the first fusion layer via the first feature re-extraction convolution block, and the other output of the first neural network block is input into the first fusion layer; one of the outputs of the second neural network block is input into the third fusion layer via the second feature re-extraction convolution block, and the other output of the second neural network block is input into the third fusion layer; one of the outputs of the third neural network block is input into the fifth fusion layer via the third feature re-extraction convolution block, and the other output of the third neural network block is input into the fifth fusion layer; one of the outputs of the fourth neural network block is input into the seventh fusion layer via the fourth feature re-extraction convolution block, and the other output of the fourth neural network block is input into the seventh fusion layer; one of the outputs of the fifth neural network block is input into the ninth fusion layer via the fifth feature re-extraction convolution block, and the other output of the fifth neural network block is input into the ninth fusion layer; the two inputs of each fusion layer are fused by element-by-element addition;

[0017] The output of the first fusion layer is respectively input into the first block attention convolution block and the corresponding second fusion layer, the output of the third fusion layer is respectively input into the second block attention convolution block and the corresponding fourth fusion layer, the output of the fifth fusion layer is respectively input into the third block attention convolution block and the corresponding sixth fusion layer, the output of the seventh fusion layer is respectively input into the fourth block attention convolution block and the corresponding eighth fusion layer, and the output of the ninth fusion layer is respectively input into the fifth block attention convolution block and the corresponding tenth fusion layer;

[0018] The two outputs of the first block attention convolution block are respectively input into the second fusion layer of the color image processing module and the depth image processing module, the two outputs of the second block attention convolution block are respectively input into the fourth fusion layer of the color image processing module and the depth image processing module, the two outputs of the third block attention convolution block are respectively input into the sixth fusion layer of the color image processing module and the depth image processing module, the two outputs of the fourth block attention convolution block are respectively input into the eighth fusion layer of the color image processing module and the depth image processing module, and the two outputs of the fifth block attention convolution block are respectively input into the tenth fusion layer of the color image processing module and the depth image processing module;

[0019] The two inputs of the second fusion layer are fused by element-by-element addition and are respectively input into the eleventh fusion layer and the corresponding second neural network block. The two inputs of the fourth fusion layer are fused by element-by-element addition and are respectively input into the first upsampling layer and the corresponding third neural network block. The two inputs of the sixth fusion layer are fused by element-by-element addition and are respectively input into the second upsampling layer and the corresponding fourth neural network block. The two inputs of the eighth fusion layer are fused by element-by-element addition and are respectively input into the third upsampling layer and the corresponding fifth neural network block. The output of the tenth fusion layer is input into the fourth upsampling layer.

[0020] The two inputs of the eleventh fusion layer, the first upsampling layer, the second upsampling layer, the third upsampling layer, and the fourth upsampling layer are fused by element-by-element addition and then input into the twelfth fusion layer;

[0021] The twelfth fusion layer concatenates all inputs and outputs them through the output layer. The output layer is mainly composed of the convolutional layer and the fifth upsampling layer connected in sequence.

[0022] The correspondence indicates that the previous input and the next output are located in the same color image processing module or the same depth image processing module.

[0023] The five neural network blocks adopt the MobileNet V2 network structure. The first neural network block adopts 1 to 4 layers of MobileNetV2 (the number of repetitions n is 1, 1, and 2, totaling four layers), the second neural network block adopts 5 to 7 layers of MobileNetV2 (the number of repetitions n is 3, totaling three layers), the third neural network block adopts 8 to 11 layers of MobileNetV2 (the number of repetitions n is 4, totaling four layers), the fourth neural network block adopts 12 to 14 layers of MobileNetV2 (the number of repetitions n is 3, totaling three layers), and the fifth neural network block adopts 15 to 17 layers of MobileNetV2 (the number of repetitions n is 3, totaling three layers).

[0024] Each of the feature re-extraction convolution blocks is composed of four re-extraction modules connected in sequence, each of which includes a convolution layer, a normalization layer, and an activation layer connected in sequence; the activation method of all activation layers is ReLU6; the step size of all convolution layers is 1; the number of convolution kernels of the convolution layer in each re-extraction module is the same as the normalization parameter of the normalization layer; the convolution kernel sizes of the convolution layers in the first re-extraction module, the third re-extraction module, and the fourth re-extraction module are the same, and the convolution kernel size of the convolution layer in the second re-extraction module is the same as the convolution kernel size of the convolution layer in the first re-extraction module. The convolution kernel size of the convolution layer in the first re-extraction module is half of the convolution kernel size; the convolution kernel size of the convolution layer in the first and third re-extraction modules is 3*3, and the convolution kernel size of the convolution layer in the second and fourth re-extraction modules is 1*1; the expansion factor of the convolution layer in the first re-extraction module is 1, and the expansion factor of the convolution layer in the third re-extraction module is 2; the zero-padding parameter of the convolution layer in the first re-extraction module is 1, the zero-padding parameter of the convolution layer in the third re-extraction module is 2, and the zero-padding parameter of the convolution layer in the second and fourth re-extraction modules is 0.

[0025] The convolution kernel size of the convolution layer of the first re-extraction module in the first feature re-extraction convolution block is 24, the convolution kernel size of the convolution layer of the first re-extraction module in the second feature re-extraction convolution block is 32, the convolution kernel size of the convolution layer of the first re-extraction module in the third feature re-extraction convolution block is 64, the convolution kernel size of the convolution layer of the first re-extraction module in the fourth feature re-extraction convolution block is 96, and the convolution kernel size of the convolution layer of the first re-extraction module in the fifth feature re-extraction convolution block is 160;

[0026] The grouping parameters of the four convolutional layers set sequentially in the first feature re-extraction convolution block are 24, 12, 24, and 24 respectively; the grouping parameters of the four convolutional layers set sequentially in the second feature re-extraction convolution block are 32, 16, 16, and 32 respectively; the grouping parameters of the four convolutional layers set sequentially in the third feature re-extraction convolution block are 64, 32, 32, and 64 respectively; the grouping parameters of the four convolutional layers set sequentially in the fourth feature re-extraction convolution block are 64, 48, 48, and 99 respectively; and the grouping parameters of the four convolutional layers set sequentially in the fifth feature re-extraction convolution block are 96, 80, 80, and 160 respectively.

[0027] Each of the block attention convolution blocks includes a block layer, a channel attention layer and a spatial size attention layer. The input of the block attention convolution block is respectively input into the channel attention layer and the spatial size attention layer after passing through the block layer. The input of the block attention convolution block is multiplied with the output of the channel attention layer and the output of the spatial size attention layer respectively, and then the multiplication results are added as the output of the block attention convolution block.

[0028] The block layer uses the split function provided by pytorch, and the parameter is half of the number of channels corresponding to the input feature map of the block attention convolution block;

[0029] The channel attention layer includes an adaptive maximum pooling layer, a channel attention first convolution layer, a channel attention second convolution layer and a channel attention activation layer connected in sequence; the maximum pooling parameter of the adaptive maximum pooling layer is 1; the convolution kernel size of the channel attention first convolution layer and the channel attention second convolution layer are both 1x1, the step size is 1, the bias item is False, the grouping parameters are the same, and the number of convolution kernels in the channel attention first convolution layer is half of the number of convolution kernels in the channel attention second convolution layer;

[0030] The number of convolution kernels of the first channel attention convolution layer of the channel attention layer in the first block attention convolution block is 12, and the grouping parameter is 12; the number of convolution kernels of the first channel attention convolution layer of the channel attention layer in the second block attention convolution block is 16, and the grouping parameter is 16; the number of convolution kernels of the first channel attention convolution layer of the channel attention layer in the third block attention convolution block is 32, and the grouping parameter is 32; the number of convolution kernels of the first channel attention convolution layer of the channel attention layer in the fourth block attention convolution block is 48, and the grouping parameter is 48; the number of convolution kernels of the first channel attention convolution layer of the channel attention layer in the fifth block attention convolution block is 80, and the grouping parameter is 80;

[0031] The spatial-size attention layer includes a channel-wise maximization layer, a spatial-size attention convolution layer, and a spatial-size attention activation layer connected in sequence; the channel-wise maximization layer uses the max function that comes with pytorch; the convolution kernel size of the spatial-size attention convolution layer is 3×3, the number of convolution kernels is 1, the expansion factor is 2, the zero-padding parameter is 2, the grouping parameter is 1, and the bias term is False;

[0032] The activation method used by the channel attention activation layer and the spatial size attention activation layer is the Sigmoid function.

[0033] The upsampling layer adopts the UpsamlingBilinear2d function in pytorch. The function parameter of the first upsampling layer is 2, and the function parameter of the second upsampling layer, the third upsampling layer, the fourth upsampling layer and the fifth upsampling layer is 4; the convolution kernel size of the convolution layer in the output layer is 1×1, the number of convolution kernels is 41, the step size is 1, and the bias item is False.

[0034] The input end of the color image input layer receives the color image of the indoor scene, the input end of the depth image input layer receives the depth image of the indoor scene, and the output of the output layer is 41 semantic segmentation prediction images corresponding to the indoor scene images input by the input layer.

[0035] Beneficial effects of the present invention:

[0036] 1) The method of the present invention starts from effectively extracting image channels and spatial semantic information, and reduces information loss as much as possible during gradient propagation, while not significantly increasing the number of model parameters. A module called feature re-extraction module is designed. The module structure is roughly uniformly columnar, which contains a 1*1 convolution and two 3*3 convolutional neural networks, and a 3*3 convolutional neural network uses an expanded convolution setting. To promote the effective propagation of gradients, the two 3*3 convolutional neural networks are located at both ends of the module, and the 1*1 convolution is located inside the module.

[0037] 2) The method of the present invention starts from the operating principle of the human visual system, combines the attention mechanism of human vision and block convolution, and designs a module called block attention convolution block. This module includes a channel-based and space-based attention mechanism module. Using the principle of block convolution, the input convolution feature map is first divided into two parts according to the channel. One part is used for feature screening based on the channel attention mechanism, and the other part is used for feature screening based on the spatial attention mechanism. Finally, the features after the two parts are screened are multiplied with the input convolution feature map, which effectively reduces redundant features.

[0038] 3) The method of the present invention combines the design of the above two modules, takes MobileNetV2 as the main framework, and designs the network model MSCNet of the present invention. Experiments show that this model has fewer parameters and faster speed. Each training round takes only about 43 seconds, while maintaining high accuracy. It is a lightweight network suitable for mobile terminals. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a general implementation block diagram of the method of the present invention;

[0040] Figure 2a is the first original indoor scene color image of the same scene;

[0041] Figure 2b is the first original indoor scene depth image of the same scene;

[0042] Figure 2c In order to utilize the method of the present invention Figure 2a and 2b The original indoor scene image shown is predicted to obtain a predicted semantic segmentation image;

[0043] Figure 3a is the second original indoor scene color image of the same scene;

[0044] Figure 3b is the second original indoor scene depth image of the same scene;

[0045] Figure 3c In order to utilize the method of the present invention Figure 3a and 3b The original indoor scene image shown is predicted to obtain a predicted semantic segmentation image;

[0046] Figure 4a is the third original indoor scene color image of the same scene;

[0047] Figure 4b is the third original indoor scene depth image of the same scene;

[0048] Figure 4c In order to utilize the method of the present invention Figure 4a and 4b The original indoor scene image shown is predicted to obtain a predicted semantic segmentation image;

[0049] Figure 5a is the fourth original indoor scene color image of the same scene;

[0050] Figure 5b is the fourth original indoor scene depth image of the same scene;

[0051] Figure 5c In order to utilize the method of the present invention Figure 5a and 5b The original indoor scene image shown is predicted to obtain a predicted semantic segmentation image.

[0052] Figure 6 This is a structural block diagram of the block attention convolution block of the present invention. DETAILED DESCRIPTION

[0053] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0054] The present invention proposes an indoor scene semantic segmentation method based on an improved fully convolutional neural network, and its overall implementation block diagram is as follows: Figure 1 As shown, it includes two processes: training phase and testing phase;

[0055] The specific steps of the training phase process are:

[0056] Step 1_1: Select Q pairs of original indoor scene RGB color images and Depth depth map images and the real semantic segmentation images corresponding to each pair of original indoor scene images, and form a training set. The qth pair of original indoor scene images in the training set is recorded as {RGB q (i,j),Depth q (i,j)}, the training set {RGB q (i,j),Depth q (i,j)} and the corresponding true semantic segmentation image are recorded as Then, the existing one-hot encoding technology is used to process the real semantic segmentation images corresponding to each pair of original indoor scene images in the training set into 41 one-hot encoded images. The set of 41 processed one-hot encoded images is recorded as Among them, the indoor scene image includes RGB color image and Depth depth map, Q is a positive integer, Q ≥ 200, such as Q = 794, q is a positive integer, 1 ≤ q ≤ Q, 1 ≤ i ≤ W, 1 ≤ j ≤ H, W represents {RGBq (i,j), Depth q (i, j)}, H represents the width of the color image RGB and the depth image Dept. q (i,j),Depth q The height of the color image RGB and the depth image Depth in (ij,)}, such as taking W = 480, H = 640, RGB q (i,j), Depth q (i,j) respectively represent {RGB q (i,j),Depth q The pixel value of the pixel point with coordinate position (i, j)}, express Here, the original indoor scene images are directly selected from the 1448 images in the indoor scene image database NYUv2 training set. In order to facilitate training, the size of each image is reduced to a width of 480 and a height of 640. In addition, in order to effectively alleviate the problem of model overfitting, three data enhancement methods, random cropping, random horizontal flipping, and random scaling, are used to expand the data in the training set.

[0057] Step 1_2: Construct a convolutional neural network classification training model: the convolutional neural network includes an input layer, a hidden layer and an output layer; the hidden layer includes the first neural network block, the second neural network block, the third neural network block, the fourth neural network block, the fifth neural network block, the first feature re-extraction convolution block, the second feature re-extraction convolution block, the third feature re-extraction convolution block, the fourth feature re-extraction convolution block, the fifth feature re-extraction convolution block, the first block attention convolution block, The 2nd block attention convolution block, the 3rd block attention convolution block, the 4th block attention convolution block, the 5th block attention convolution block, the 1st fusion layer, the 2nd fusion layer, the 3rd fusion layer, the 4th fusion layer, the 5th fusion layer, the 6th fusion layer, the 7th fusion layer, the 8th fusion layer, the 9th fusion layer, the 10th fusion layer, the 11th fusion layer, the 12th fusion layer, the 1st upsampling layer, the 2nd upsampling layer, the 3rd upsampling layer, the 4th upsampling layer.

[0058] For the input layer, the present invention has two inputs, namely a color image RGB input layer and a depth map Depth input layer. The color image RGB input end of the input layer receives the R channel component, the G channel component, and the B channel component of an original RGB input image, and the depth map Depth input end of the input layer receives a single channel component of an original Depth input image. The color image RGB output end of the input layer outputs the R channel component, the G channel component, and the B channel component of the original input image to the hidden layer, and the depth map Depth output end of the input layer outputs the single channel component of the original input image to the hidden layer; wherein, the width of the original input image received by the input end of the input layer is required to be 480 and the height is 640; in addition, the convolutional neural network structure whose input is the color image RGB is symmetrical with the convolutional neural network structure whose input is the depth map Depth.

[0059] For the five neural network blocks, the five neural network blocks adopt the MobileNet V2 network structure. The first neural network block adopts 1 to 4 layers of MobileNetV2 (the number of repetitions n is 1, 1, and 2, with a total of four layers), the second neural network block adopts 5 to 7 layers of MobileNetV2 (the number of repetitions n is 3, with a total of three layers), the third neural network block adopts 8 to 11 layers of MobileNetV2 (the number of repetitions n is 4, with a total of four layers), the fourth neural network block adopts 12 to 14 layers of MobileNetV2 (the number of repetitions n is 3, with a total of three layers), and the fifth neural network block adopts 15 to 17 layers of MobileNetV2 (the number of repetitions n is 3, with a total of three layers).

[0060] The color image RGB input end of the first neural network block receives the R channel component, G channel component and B channel component of the original input image output by the output end of the input layer, and the depth map Depth input end receives the single channel component of the original input image output by the output end of the input layer. The color image RGB output end of the first neural network block outputs 24 feature maps, and the set they constitute is recorded as R1. The depth map Depth output end of the first neural network block outputs 24 feature maps, and the set they constitute is recorded as D1; ​​the width of each feature map in R1 and D1 is Height is The color map RGB output terminal of the second neural network block outputs 32 feature maps, and the set they constitute is denoted as R2; the depth map Depth input terminal of the second neural network block receives all feature maps in DF1, and the depth map Depth output terminal of the second neural network block outputs 32 feature maps, and the set they constitute is denoted as D2; the width of each feature map in R2 and D2 is The height is The color map RGB input terminal of the third neural network block receives all feature maps in RF2, and the color map RGB output terminal of the third neural network block outputs 64 feature maps, and the set of 64 feature maps is recorded as R3; the depth map Depth input terminal of the third neural network block receives all feature maps in DF2, and the depth map Depth output terminal of the third neural network block outputs 64 feature maps, and the set of 64 feature maps is recorded as D3; the width of each feature map in R3 and D3 is The height is The color map RGB input end of the fourth neural network block receives all feature maps in RF3, and the color map RGB output end of the fourth neural network block outputs 96 feature maps, and the set composed of 96 feature maps is recorded as R4; the depth map Depth input end of the fourth neural network block receives all feature maps in DF3, and the depth map Depth output end of the fourth neural network block outputs 96 feature maps, and the set composed of them is recorded as D4; the width of each feature map in R4 and D4 is The height is The color map RGB input end of the fifth neural network block receives all feature maps in RF4, and the color map RGB output end of the fifth neural network block outputs 160 feature maps, and the set of 160 feature maps is recorded as R5; the depth map Depth input end of the fifth neural network block receives all feature maps in DF4, and the depth map Depth output end of the fifth neural network block outputs 160 feature maps, and the set of 160 feature maps is recorded as D5; the width of each feature map in R5 and D5 is Height is

[0061] For the first feature re-extraction convolution block, it is composed of the fifty-first convolution layer, the fifty-first standardization layer, the fifty-first activation layer, the fifty-second convolution layer, the fifty-second standardization layer, the fifty-second activation layer, the fifty-third convolution layer, the fifty-third standardization layer, the fifty-third activation layer, the fifty-fourth convolution layer, the fifty-fourth standardization layer, and the fifty-fourth activation layer, which are arranged in sequence; the input end of the first feature re-extraction convolution block receives all feature maps in R1 and D1, the color map RGB output end of the first feature extraction block outputs 24 feature maps, and the set composed of them is recorded as RS1, and the depth map Depth output end outputs 24 feature maps, and the set composed of them is recorded as DS1; wherein, the convolution kernel size of the fifty-first convolution layer is 3×3, the number of convolution kernels is 24, the stride is 1, the dilation factor is 1, and the padding parameter is 1, The groups parameter is 24, and the 51st layer standardization parameter is 24; the 52nd convolution layer has a kernel size of 1x1, a kernel number of 12, a stride of 1, a padding parameter of 0, a groups parameter of 12, and a 52nd layer standardization parameter of 12; the 53rd convolution layer has a kernel size of 3×3, a kernel number of 24, a stride of 1, a dilation factor of 2, a padding parameter of 2, a groups parameter of 24, and a 53rd layer standardization parameter of 24; the 54th convolution layer has a kernel size of 1x1, a kernel number of 24, a stride of 1, a padding parameter of 0, a groups parameter of 24, and a 54th layer standardization parameter of 24. The activation mode of all activation layers is "ReLU6", and the width of each feature map in RS1 and DS1 is Height is

[0062] For the first fusion layer, the color map RGB input end of the first fusion layer receives all feature maps in R1 and all feature maps in RS1, and fuses R1 and RS1 through the existing add (element-by-element addition) method to obtain the set RA1, and the color map RGB output end of the first fusion layer outputs RA1; the depth map Depth input end of the first fusion layer receives all feature maps in D1 and all feature maps in DS1, and the first connection layer fuses D1 and DS1 through the existing add (element-by-element addition) method to obtain the set DA1, and the depth map Depth output end of the first fusion layer outputs DA1; wherein, the total number of feature maps contained in RA1 and DA1 is 24, and the width of each feature map in RA1 and DA1 is The height is

[0063] For the first block-wise attention convolution block, the structure is as follows Figure 6 As shown in the figure, it consists of the first block layer, the first channel attention layer, and the first spatial size attention layer set in sequence; the color image RGB input end of the first block attention convolution block receives all feature maps in RA1, and the depth map Depth input end of the first block attention convolution block receives all feature maps in DA1. All feature maps in RA1 and DA1 are respectively input into the first block attention convolution block. First, all the input feature maps are divided into two parts according to the channel through the first block layer, one part is used as the input of the channel attention layer, and the other part is used as the input of the spatial size attention layer. Finally, the two parts are The results of the edge attention processing are multiplied with the original input feature map and then the multiplication results are added. The color map RGB output end of the first block attention convolution block outputs 24 feature maps, and its set is recorded as RT1. The depth map Depth output end of the first block attention convolution block outputs 24 feature maps, and its set is recorded as DT1. Among them, the first block layer uses the split function that comes with pytorch, and the parameters are half of the original input feature map and the corresponding number of channels; the first channel attention layer includes the first adaptive maximum pooling, the first channel attention first convolution layer, and the first channel attention The second convolution layer and activation layer of the first channel attention are used, where the first adaptive maximum pooling parameter is 1, the convolution kernel size of the first channel attention first convolution layer is 1x1, the number of convolution kernels is 12, the stride is 1, the grouping parameter is 12, and the bias is False; the convolution kernel size of the second channel attention convolution layer is 1x1, the number of convolution kernels is 24, the stride is 1, the grouping parameter is 12, and the bias is False; the activation mode of the first channel attention activation layer is "S The first spatial dimension attention layer includes the first channel maximization layer, the first spatial dimension attention convolution layer and the activation function, where the first channel maximization layer uses the max function provided by pytorch; the convolution kernel size of the first spatial dimension attention convolution layer is 3×3, the number of convolution kernels is 1, the dilation factor is 2, the padding parameter is 2, the grouping parameter is 1, the bias is False, and the activation mode of the first spatial dimension attention activation layer is "Sigmoid". The width of each feature map in RT1 and DT1 is The height is

[0064] For the second fusion layer, the color map RGB input end of the second fusion layer receives all feature maps in DT1, and RA1 and DT1 are fused by the existing add (element-by-element addition) method to obtain the set RF1, and the color map RGB output end of the second fusion layer outputs RF1; the depth map Depth input end of the second fusion layer receives all feature maps in RT1 and all feature maps in DA1, and the second fusion is fused by the existing add (element-by-element addition) method to obtain the set DF1, and the depth map Depth output end of the second fusion layer outputs DF1; wherein, the total number of feature maps contained in RF1 and DF1 is 24, and the width of each feature map in RF1 and DF1 is The height is

[0065] For the second feature re-extraction convolution block, it is composed of the fifty-fifth convolution layer, the fifty-fifth normalization layer, the fifty-fifth activation layer, the fifty-sixth convolution layer, the fifty-sixth normalization layer, the fifty-sixth activation layer, the fifty-seventh convolution layer, the fifty-seventh normalization layer, the fifty-seventh activation layer, the fifty-eighth convolution layer, the fifty-eighth normalization layer, and the fifty-eighth activation layer arranged in sequence; the color image RGB input end of the second feature re-extraction convolution block receives all feature maps in R2, and the color image RGB output end of the second feature re-extraction convolution block outputs 32 feature maps, and the set formed by them is recorded as RS2; the depth map Depth input end of the second feature re-extraction convolution block receives all feature maps in D2, and the depth map Depth output end of the second feature re-extraction convolution block outputs 32 feature maps, and the set formed by them is recorded as DS2; wherein, the convolution kernel size of the fifty-fifth convolution layer is 3×3, the number of convolution kernels is 32, the stride is 1, and the dilation factor (di The convolution kernel size of the fifty-sixth convolution layer is 1x1, the number of convolution kernels is 16, the stride is 1, the padding parameter is 0, the groups parameter is 16, and the standardization parameter of the fifty-sixth layer is 16; the convolution kernel size of the fifty-seventh convolution layer is 3×3, the number of convolution kernels is 32, the stride is 1, the dilation factor is 2, the padding parameter is 2, the groups parameter is 16, and the standardization parameter of the fifty-seventh layer is 32; the convolution kernel size of the fifty-eighth convolution layer is 1x1, the number of convolution kernels is 32, the stride is 1, the padding parameter is 0, the groups parameter is 32, and the standardization parameter of the fifty-eighth layer is 32. The activation mode of all activation layers is "ReLU6", and the width of each feature map in RS2 and DS2 is Height is

[0066] For the third fusion layer, the color map RGB input end of the third fusion layer receives all feature maps in R2 and all feature maps in RS2, and the third fusion layer fuses R2 and RS2 through the existing add (element-by-element addition) method to obtain the set RA2, and the color map RGB output end of the third fusion layer outputs RA2; the depth map Depth input end of the third fusion layer receives all feature maps in D2 and all feature maps in DS2, and the second connection layer fuses D2 and DS2 through the existing add (element-by-element addition) method to obtain the set DA2, and the depth map Depth output end of the third fusion layer outputs DA2; wherein, the total number of feature maps contained in RA2 and DA2 is 32, and the width of each feature map in RA2 and DA2 is The height is

[0067] For the second block attention convolution block, it consists of a second block layer, a second channel attention layer, and a second spatial size attention layer set in sequence; the color image RGB input end of the second block attention convolution block receives all feature maps in RA2, and all feature maps in RA2 are input into the second block attention convolution block. First, all the input feature maps are divided into two parts according to the channel through the second block layer, one part is used as the input of the channel attention layer, and the other part is used as the input of the spatial size attention layer. Finally, the results of the attention processing on both sides are multiplied with the original input feature maps and the multiplication results are added. The color image RGB output end of the second block attention convolution block outputs 32 feature maps. The set formed is recorded as RT2; the depth map Depth input end of the second block attention convolution block receives all feature maps in DA2, and all feature maps in DA2 are input into the second block attention convolution block. First, all the input feature maps are divided into two parts by channel through the second block layer, one part is used as the input of the channel attention layer, and the other part is used as the input of the spatial size attention layer. Finally, the results of the attention processing on both sides are multiplied with the original input feature map and then the multiplication results are added. The depth map Depth output end of the second block attention convolution block outputs 32 feature maps, and the set formed by them is recorded as DT2; Among them, the second block layer uses pytorch's own sp lit function, with the parameters being half of the original input feature map and the corresponding number of channels; the second channel attention layer includes the second adaptive maximum pooling, the second channel attention first convolutional layer, the second channel attention second convolutional layer and the activation layer, where the second adaptive maximum pooling parameter is 1, the second channel attention first convolutional layer has a convolution kernel size of 1x1, the number of convolution kernels is 16, the stride is 1, the groups parameter is 16, and the bias is False; the second channel attention second convolutional layer has a convolution kernel size of 1x1, the number of convolution kernels is 32, the stride is 1, the groups parameter is 16, and the bias is False. The bias is set to False; the activation mode of the second channel attention activation layer is "Sigmoid"; the second spatial dimension attention layer includes the second channel maximization layer, the second spatial dimension attention convolution layer and the activation function, where the second channel maximization layer uses the max function provided by pytorch; the convolution kernel size of the second spatial dimension attention convolution layer is 3×3, the number of convolution kernels is 1, the dilation factor is 2, the padding parameter is 2, the grouping parameter is 1, the bias is False, and the activation mode of the second spatial dimension attention activation layer is "Sigmoid". The width of each feature map in RT2 and DT2 is The height is

[0068] For the fourth fusion layer, the color map RGB input end of the fourth fusion layer receives all feature maps in RA2 and DT2, and the fourth fusion layer fuses RA2 and DT2 through the existing add (element-by-element addition) method to obtain the set RF2, and the color map RGB output end of the fourth fusion layer outputs RF2; the depth map Depth input end of the fourth fusion layer receives all feature maps in RT2 and all feature maps in DA2, and the fourth fusion fuses RT2 and DA2 through the existing add (element-by-element addition) method to obtain the set DF2, and the depth map Depth output end of the fourth fusion layer outputs DF2; wherein, the total number of feature maps contained in RF2 and DF2 is 32, and the width of each feature map in RF2 and DF2 is The height is

[0069] For the third feature re-extraction convolution block, it is composed of the fifty-ninth convolution layer, the fifty-ninth standardization layer, the fifty-ninth activation layer, the sixtieth convolution layer, the sixtieth standardization layer, the sixtieth activation layer, the sixty-first convolution layer, the sixty-first standardization layer, the sixty-first activation layer, the sixty-second convolution layer, the sixty-second standardization layer, and the sixty-second activation layer arranged in sequence; the color image RGB input end of the third feature re-extraction convolution block receives all feature maps in R3, and the color image RGB output end of the third feature re-extraction convolution block outputs 64 feature maps, and the set constituted by them is recorded as RS3; the depth map Depth input end of the third feature re-extraction convolution block receives all feature maps in D3, and the depth map Depth output end of the third feature re-extraction convolution block outputs 64 feature maps, and the set constituted by them is recorded as DS3; wherein, the convolution kernel size of the fifty-ninth convolution layer is 3×3, the number of convolution kernels is 64, the stride is 1, and the dilation factor (dil ation) is 1, the padding parameter is 1, the grouping parameter is 64, and the standardized parameter of the fifty-ninth layer is 64; the convolution kernel size of the sixtieth convolution layer is 1x1, the number of convolution kernels is 32, the stride is 1, the padding parameter is 0, the grouping parameter is 32, and the standardized parameter of the sixtieth layer is 32; the convolution kernel size of the sixty-first convolution layer is 3×3, the number of convolution kernels is 64, the stride is 1, the dilation factor is 2, the padding parameter is 2, the grouping parameter is 32, and the standardized parameter of the sixty-first layer is 64; the convolution kernel size of the sixty-second convolution layer is 1x1, the number of convolution kernels is 64, the stride is 1, the padding parameter is 0, the grouping parameter is 64, and the standardized parameter of the sixty-second layer is 64. The activation mode of all activation layers is "ReLU6", and the width of each feature map in RS3 and DS3 is Height is

[0070] For the fifth fusion layer, the color map RGB input end of the fifth fusion layer receives all feature maps in R3 and all feature maps in RS3, and the fifth fusion layer fuses R3 and RS3 through the existing add (element-by-element addition) method to obtain the set RA3, and the color map RGB output end of the fifth fusion layer outputs RA3; the depth map Depth input end of the fifth fusion layer receives all feature maps in D3 and all feature maps in DS3, and the third fusion layer fuses D3 and DS3 through the existing add (element-by-element addition) method to obtain the set DA3, and the depth map Depth output end of the fifth fusion layer outputs DA3; wherein, the total number of feature maps contained in RA3 and DA3 is 64, and the width of each feature map in RA3 and DA3 is The height is

[0071] For the third block attention convolution block, it consists of a third block layer, a third channel attention layer, and a third spatial size attention layer which are arranged in sequence; the color image RGB input end of the third block attention convolution block receives all feature maps of RA3, and all feature maps in RA3 are input into the third block attention convolution block. First, all the input feature maps are divided into two parts according to the channel through the third block layer, one part is used as the input of the channel attention layer, and the other part is used as the input of the spatial size attention layer. Finally, the results of the attention processing on both sides are multiplied with the original input feature maps respectively and then the multiplication results are added. The color image RGB output end of the third block attention convolution block outputs 64 feature maps, and the 64 feature maps are added. The set of feature maps is recorded as RT3; the depth map Depth input end of the third block attention convolution block receives all the feature maps in DA3, and all the feature maps in DA3 are input into the third block attention convolution block. First, all the input feature maps are divided into two parts by channel through the third block layer, one part is used as the input of the channel attention layer, and the other part is used as the input of the spatial size attention layer. Finally, the results of the attention processing on both sides are multiplied with the original input feature map and then the multiplication results are added. The depth map Depth output end of the third block attention convolution block outputs 64 feature maps, and the set of 64 feature maps is recorded as DT3; Among them, the third block layer adopts pytorch The built-in split function has parameters of the original input feature map and half of the corresponding number of channels; the third channel attention layer includes the third adaptive maximum pooling, the third channel attention first convolution layer, the third channel attention second convolution layer and the activation layer, where the third adaptive maximum pooling parameter is 1, the third channel attention first convolution layer has a convolution kernel size of 1x1, the number of convolution kernels is 32, the stride is 1, the grouping parameter is 32, and the bias is False; the third channel attention second convolution layer has a convolution kernel size of 1x1, the number of convolution kernels is 64, the stride is 1, and the grouping parameter is 32 , the bias is False; the activation mode of the third channel attention activation layer is "Sigmoid"; the third spatial dimension attention layer includes the third channel maximization layer, the third spatial dimension attention convolution layer and the activation function, among which the third channel maximization layer uses the max function provided by pytorch; the convolution kernel size of the third spatial dimension attention convolution layer is 3×3, the number of convolution kernels is 1, the dilation factor is 2, the padding parameter is 2, the grouping parameter is 1, the bias is False, and the activation mode of the third spatial dimension attention activation layer is "Sigmoid". The width of each feature map in RT3 and DT3 is The height is

[0072] For the sixth fusion layer, the color map RGB input end of the sixth fusion layer receives all feature maps in RA3 and DT3, and the sixth fusion layer fuses RA3 and DT3 through the existing add (element-by-element addition) method to obtain the set RF3, and the color map RGB output end of the sixth fusion layer outputs RF3; the depth map Depth input end of the sixth fusion layer receives all feature maps in RT3 and all feature maps in DA3, and the sixth fusion fuses RT3 and DA3 through the existing add (element-by-element addition) method to obtain the set DF3, and the depth map Depth output end of the sixth fusion layer outputs DF3; wherein, the total number of feature maps contained in RF3 and DF3 is 64, and the width of each feature map in RF3 and DF3 is The height is

[0073] For the fourth feature re-extraction convolution block, it is composed of the sixty-third convolution layer, the sixty-third normalization layer, the sixty-third activation layer, the sixty-fourth convolution layer, the sixty-fourth normalization layer, the sixty-fourth activation layer, the sixty-fifth convolution layer, the sixty-fifth normalization layer, the sixty-fifth activation layer, the sixty-sixth convolution layer, the sixty-sixth normalization layer, the sixty-sixth activation layer, the sixty-sixth convolution layer, the sixty-sixth normalization layer, and the sixty-sixth activation layer arranged in sequence; the color image RGB input end of the fourth feature re-extraction convolution block receives all feature maps in R4, and the output end of the fourth feature re-extraction convolution block outputs 96 feature maps, and the set composed of 96 feature maps is recorded as RS4; the depth map Depth input end of the fourth feature re-extraction convolution block receives all feature maps in D4, and the output end of the fourth feature re-extraction convolution block outputs 96 feature maps, and the set composed of 96 feature maps is recorded as DS4; wherein the convolution kernel size of the sixty-third convolution layer is 3×3, the number of convolution kernels is 96, the stride is 1, and the dilation factor (dilati The convolution kernel size of the sixty-fourth convolution layer is 1x1, the number of convolution kernels is 48, the stride is 1, the padding parameter is 0, the groups parameter is 48, and the standardization parameter of the sixty-fourth layer is 48; the convolution kernel size of the sixty-fifth convolution layer is 3×3, the number of convolution kernels is 96, the stride is 1, the dilation factor is 2, the padding parameter is 2, the groups parameter is 48, and the standardization parameter of the sixty-fifth layer is 96; the convolution kernel size of the sixty-sixth convolution layer is 1x1, the number of convolution kernels is 96, the stride is 1, the padding parameter is 0, the groups parameter is 96, and the standardization parameter of the sixty-sixth layer is 96. The activation mode of all activation layers is "ReLU6", and the width of each feature map in RS4 and DS4 is Height is

[0074] For the seventh fusion layer, the color map RGB input end of the seventh fusion layer receives all feature maps in R4 and all feature maps in RS4, and the seventh fusion layer fuses R4 and RS4 through the existing add (element-by-element addition) method to obtain the set RA4, and the color map RGB output end of the seventh fusion layer outputs RA4; the depth map Depth input end of the seventh fusion layer receives all feature maps in D4 and all feature maps in DS4, and the seventh fusion layer fuses D4 and DS4 through the existing add (element-by-element addition) method to obtain the set DA4, and the depth map Depth output end of the seventh fusion layer outputs DA4; wherein, the total number of feature maps contained in RA4 and DA4 is 96, and the width of each feature map in RA4 and DA4 is The height is

[0075] For the 4th block attention convolution block, it consists of the 4th block layer, the 4th channel attention layer, and the 4th spatial size attention layer which are arranged in sequence; the color image RGB input end of the 4th block attention convolution block receives all the feature maps in RA4, and all the feature maps in RA4 are input into the 4th block attention convolution block. First, all the input feature maps are divided into two parts according to the channel through the 4th block layer, one part is used as the input of the channel attention layer, and the other part is used as the input of the spatial size attention layer. Finally, the results of the attention processing on both sides are multiplied with the original input feature maps respectively and then the multiplication results are added. The output end of the 4th block attention convolution block outputs 96 feature maps, and the 96 feature maps constitute The collection is recorded as RT4; the depth map Depth input end of the 4th block attention convolution block receives all the feature maps in DA4, and all the feature maps in DA4 are input into the 4th block attention convolution block. First, all the input feature maps are divided into two parts by channel through the 4th block layer, one part is used as the input of the channel attention layer, and the other part is used as the input of the spatial size attention layer. Finally, the results of the attention processing on both sides are multiplied with the original input feature map and then the multiplication results are added. The depth map Depth output end of the 4th block attention convolution block outputs 96 feature maps, and the collection composed of 96 feature maps is recorded as DT4; Among them, the 4th block layer uses pytorch's built-in split function, the parameters are half of the original input feature map and the corresponding number of channels; the fourth channel attention layer includes the fourth adaptive maximum pooling, the fourth channel attention first convolutional layer, the fourth channel attention second convolutional layer and the activation layer, where the fourth adaptive maximum pooling parameter is 1, the fourth channel attention first convolutional layer has a convolution kernel size of 1x1, the number of convolution kernels is 48, the stride is 1, the grouping parameter is 48, and the bias is False; the fourth channel attention second convolutional layer has a convolution kernel size of 1x1, the number of convolution kernels is 96, the stride is 1, and the grouping parameter is 48. The bias is False; the activation mode of the fourth channel attention activation layer is "Sigmoid"; the fourth spatial dimension attention layer includes the fourth channel maximization layer, the fourth spatial dimension attention convolution layer and the activation function, among which the fourth channel maximization layer uses the max function provided by pytorch; the convolution kernel size of the fourth spatial dimension attention convolution layer is 3×3, the number of convolution kernels is 1, the dilation factor is 2, the padding parameter is 2, the grouping parameter is 1, the bias is False, and the activation mode of the fourth spatial dimension attention activation layer is "Sigmoid". The width of each feature map in RT4 and DT4 is Height is

[0076] For the eighth fusion layer, the color map RGB input end of the eighth fusion layer receives all feature maps in RA4 and DT4, and the eighth fusion layer fuses RA4 and DT4 through the existing add (element-by-element addition) method to obtain the set RF4, and the color map RGB output end of the eighth fusion layer outputs RF4; the depth map Depth input end of the eighth fusion layer receives all feature maps in RT4 and all feature maps in DA4, and the eighth fusion layer fuses RT4 and DA4 through the existing add (element-by-element addition) method to obtain the set DF4, and the depth map Depth output end of the eighth fusion layer outputs DF4; wherein, the total number of feature maps contained in RF4 and DF4 is 96, and the width of each feature map in RF4 and DF4 is The height is

[0077] For the fifth feature re-extraction convolution block, it is composed of the sixty-seventh convolution layer, the sixty-seventh normalization layer, the sixty-seventh activation layer, the sixty-eighth convolution layer, the sixty-eighth normalization layer, the sixty-eighth activation layer, the sixty-ninth convolution layer, the sixty-ninth normalization layer, the sixty-ninth activation layer, the seventy-ninth convolution layer, the seventy-ninth normalization layer, and the seventy-ninth activation layer arranged in sequence; the color image RGB input end of the fifth feature re-extraction convolution block receives all feature maps in R5, and the color image RGB output end of the fifth feature re-extraction convolution block outputs 160 feature maps, and the set composed of 160 feature maps is recorded as RS5; the depth map Depth input end of the fifth feature re-extraction convolution block receives all feature maps in D5, and the depth map Depth output end of the fifth feature re-extraction convolution block outputs 160 feature maps, and the set composed of 160 feature maps is recorded as DS5; wherein, the convolution kernel size of the sixty-seventh convolution layer is 3×3, the number of convolution kernels is 160, the stride is 1, and the expansion factor The dilation factor is 1, the padding parameter is 1, the groups parameter is 96, and the standardization parameter of the sixty-third layer is 160; the convolution kernel size of the sixty-eighth convolution layer is 1x1, the number of convolution kernels is 80, the stride is 1, the padding parameter is 0, the groups parameter is 80, and the standardization parameter of the sixty-eighth layer is 80; the convolution kernel size of the sixty-ninth convolution layer is 3×3, the number of convolution kernels is 160, the stride is 1, the dilation factor is 2, the padding parameter is 2, the groups parameter is 80, and the standardization parameter of the sixty-ninth layer is 160; the convolution kernel size of the seventieth convolution layer is 1x1, the number of convolution kernels is 160, the stride is 1, the padding parameter is 0, the groups parameter is 160, and the standardization parameter of the seventieth layer is 160. The activation mode of all activation layers is "ReLU6", and the width of each feature map in RS5 and DS5 is Height is

[0078] For the ninth fusion layer, the color map RGB input end of the ninth fusion layer receives all feature maps in R5 and all feature maps in RS5, and the ninth fusion layer fuses R5 and RS5 through the existing add (element-by-element addition) method to obtain the set RA5, and the color map RGB output end of the ninth fusion layer outputs RA5; the depth map Depth input end of the ninth fusion layer receives all feature maps in D5 and all feature maps in DS5, and the ninth fusion layer fuses D5 and DS5 through the existing add (element-by-element addition) method to obtain the set DA5, and the depth map Depth output end of the ninth fusion layer outputs DA5; wherein, the total number of feature maps contained in RA5 and DA5 is 160, and the width of each feature map in RA5 and DA5 is The height is

[0079] For the 5th block attention convolution block, it consists of the fifth block layer, the fifth channel attention layer, and the fifth spatial size attention layer which are arranged in sequence; the color image RGB input end of the 5th block attention convolution block receives all feature maps in RA5, and all feature maps in RA5 are input into the fifth block attention convolution block. First, all the input feature maps are divided into two parts according to the channel through the fifth block layer, one part is used as the input of the channel attention layer, and the other part is used as the input of the spatial size attention layer. Finally, the results of the attention processing on both sides are multiplied with the original input feature maps respectively and then the multiplication results are added. The color image RGB output end of the 5th block attention convolution block outputs 160 feature maps, which The set formed is recorded as RT5; the depth map Depth input end of the 5th block attention convolution block receives all the feature maps in DA5, and all the feature maps in DA5 are input into the fifth block attention convolution block. First, all the input feature maps are divided into two parts by channel through the fifth block layer, one part is used as the input of the channel attention layer, and the other part is used as the input of the spatial size attention layer. Finally, the results of the attention processing on both sides are multiplied with the original input feature map and then the multiplication results are added. The depth map Depth output end of the 5th block attention convolution block outputs 160 feature maps, and the set formed by them is recorded as DT5. Among them, the fifth block layer uses pytorch's built-in sp lit function, with parameters of the original input feature map and half of the corresponding number of channels; the fifth channel attention layer includes the fifth adaptive maximum pooling, the fifth channel attention first convolutional layer, the fifth channel attention second convolutional layer and the activation layer, where the fifth adaptive maximum pooling parameter is 1, the fifth channel attention first convolutional layer has a convolution kernel size of 1x1, the number of convolution kernels is 80, the stride is 1, the grouping parameter is 80, and the bias is False; the fifth channel attention second convolutional layer has a convolution kernel size of 1x1, the number of convolution kernels is 160, the stride is 1, the grouping parameter is 80, and the bias is False. The bias is False; the activation mode of the fifth channel attention activation layer is "Sigmoid"; the fifth spatial dimension attention layer includes the fifth channel maximization layer, the fifth spatial dimension attention convolution layer and the activation function, among which the fifth channel maximization layer uses the max function provided by pytorch; the convolution kernel size of the fifth spatial dimension attention convolution layer is 3×3, the number of convolution kernels is 1, the dilation factor is 2, the padding parameter is 2, the grouping parameter is 1, the bias is False, and the activation mode of the fifth spatial dimension attention activation layer is "Sigmoid". The width of each feature map in RT5 and DT5 is Height is

[0080] For the tenth fusion layer, the color map RGB input end of the tenth fusion layer receives all feature maps in RA5 and DT5, and the tenth fusion layer fuses RA5 and DT5 through the existing add (element-by-element addition) method to obtain the set RF5, and the color map RGB output end of the tenth fusion layer outputs RF5; the depth map Depth input end of the tenth fusion layer receives all feature maps in RT5 and all feature maps in DA5, and the tenth fusion layer fuses RT5 and DA5 through the existing add (element-by-element addition) method to obtain the set DF5, and the depth map Depth output end of the tenth fusion layer outputs DF5; wherein, the total number of feature maps contained in RF5 and DF5 is 160, and the width of each feature map in RF5 and DF5 is The height is

[0081] For the first upsampling layer, the first upsampling layer input receives all feature maps in RF2 and all feature maps in DF2. RF2 and DF2 are first fused by the existing add (element-by-element addition) method to obtain the set RD2. The first upsampling layer obtains the RD2 feature map after (height and width) is doubled by the UpsamlingBilinear2d function in pytorch. The function parameter is 2. The set it constitutes is recorded as RD2x2. The output of the first upsampling layer outputs RD2x2; the total number of feature maps (number of channels) contained in RD2x2 is 32, and the width of each feature map in RD2x2 is Height is

[0082] For the second upsampling layer, the second upsampling layer input receives all feature maps in RF3 and all feature maps in DF3. First, RF3 and DF3 are fused by the existing add (element-by-element addition) method to obtain the set RD3. The second upsampling layer obtains the RD3 feature map after four times magnification through the UpsamlingBilinear2d function in pytorch. The function parameter is 4. The set it constitutes is recorded as RD3x4. The output of the second upsampling layer outputs RD3x4; the total number of feature maps contained in RD3x4 is 64, and the width of each feature map in RD3x4 is Height is

[0083] For the third upsampling layer, the input end of the third upsampling layer receives all feature maps in RF4 and all feature maps in DF4. First, RF4 and DF4 are fused by the existing add (element-by-element addition) method to obtain the set RD4. The third upsampling layer obtains the RD4 feature map after four times magnification through the UpsamlingBilinear2d function in pytorch. The function parameter is 4. The set formed by it is recorded as RD4x4. The output end of the third upsampling layer outputs RD4x4; the total number of feature maps contained in RD4x4 is 96, and the width of each feature map in RD4x4 is Height is

[0084] For the third upsampling layer, the input end of the third upsampling layer receives all feature maps in RF4 and all feature maps in DF4. First, RF4 and DF4 are fused by the existing add (element-by-element addition) method to obtain the set RD4. The third upsampling layer obtains the RD4 feature map after four times magnification through the UpsamlingBilinear2d function in pytorch. The function parameter is 4. The set formed by it is recorded as RD4x4. The output end of the third upsampling layer outputs RD4x4; the total number of feature maps contained in RD4x4 is 96, and the width of each feature map in RD4x4 is Height is

[0085] For the fourth upsampling layer, the input end of the fourth upsampling layer receives all feature maps in RF5 and all feature maps in DF5. First, RF5 and DF5 are fused by the existing add (element-by-element addition) method to obtain the set RD5. The fourth upsampling layer obtains the RD5 feature map after four times magnification through the UpsamlingBilinear2d function in pytorch. The function parameter is 4. The set it constitutes is recorded as RD5x4. The output end of the fourth upsampling layer outputs RD5x4; the total number of feature maps contained in RD5x4 is 160, and the width of each feature map in RD5x4 is Height is

[0086] For the eleventh fusion layer, the input end of the eleventh fusion layer receives all feature maps in RF1 and all feature maps in DF1, and fuses RF1 and DF1 through the existing add (element-by-element addition) method to obtain a set RD1, where the total number of feature maps contained in RD1 is 24, and the width of each feature map in RD1 is and the height is Height is

[0087] For the twelfth fusion layer, the input end of the twelfth fusion layer receives all feature maps in RD1, RD2x2, RD3x4, RD4x4 and RD5x4, and the twelfth fusion layer connects RD1, RD2x2, RD3x4, RD4x4 and RD5x4 in the existing concatenate method to obtain the set RDM, and the output end of the twelfth fusion layer outputs RDM; wherein the total number of feature maps contained in RDM is 376 (24+32+64+96+160), and the width of each feature map in RDM is

[0088] For the output layer, it is composed of the seventy-first convolutional layer and the fifth upsampling layer connected in sequence, where the convolution kernel size of the seventy-first convolutional layer is 1×1, the number of convolution kernels is 41, the stride is 1, and the bias is False; the fifth upsampling layer obtains the feature map after four times magnification through the UpsamlingBilinear2d function in pytorch, and the function parameter is 4; the input end of the output layer receives all feature maps in RDM, and the output end of the output layer outputs 41 semantic segmentation prediction maps corresponding to the original input image.

[0089] Step 1_3: Take each pair of original indoor scene RGB images and Depth images in the training set as the original input images and input them into the convolutional neural network for training. Then, 41 semantic segmentation prediction images corresponding to each pair of original indoor scene images in the training set are obtained. q (i,j),Depth q The set of 41 semantic segmentation prediction images corresponding to (i, j)} is recorded as

[0090] Step 1_4: Calculate the loss function value between the set of 41 semantic segmentation prediction images corresponding to each pair of original indoor scene images in the training set and the set of 41 one-hot encoded images processed from the corresponding real semantic segmentation images. and The loss function value between It is obtained using categorical crossentropy.

[0091] Step 1_5: Repeat steps 1_3 and 1_4 for V times to obtain a convolutional neural network classification training model, and obtain Q×V loss function values ​​in total; then find the loss function value with the smallest value from the Q×V loss function values; then use the weight vector and bias term corresponding to the loss function value with the smallest value as the optimal weight vector and optimal bias term of the convolutional neural network classification training model, which are recorded as W best and bbest ; Wherein, V>1, and in this embodiment, V=500.

[0092] The specific steps of the testing phase process are:

[0093] Step 2_1: Separate P pairs of color images and depth images from the original data set as the test set, where P is a positive integer. In this invention, P is 699. Let {RGB p (i',j'),Depth p (i',j')} represents the indoor scene image to be semantically segmented in the test set, 1≤p≤P; where 1≤i'≤W', 1≤j'≤H', W' represents {RGB p (i',j'),Depth p (i',j')} width of the color image and depth image, H' represents {RGB p (i',j'),Depth p (i',j)'} The height of the color image and depth image, RGB p (i',j'), Depth p (i',j') respectively represent {RGB p (i',j'),Depth p The pixel value of the pixel with coordinate position (i, j) in the color image RGB and the depth image Depth in (i', j')}.

[0094] Step 2_2: Set {RGB p (i',j'),Depth p The color image RGB and depth image Depth in (i',j')} are input into the trained improved full convolutional neural network semantic segmentation model, and the optimal weight W of the convolution kernel obtained in training is used. best and the optimal bias term b best Make a prediction and get {RGB p (i',j'),Depth p (i',j')} corresponding to the predicted semantic segmentation image, denoted as in, express The pixel value of the pixel with coordinate position (i', j').

[0095] To further verify the effect of the present invention, specific experiments are performed below to verify it.

[0096] The present invention uses the architecture of an improved fully convolutional neural network semantic segmentation model built on the deep learning library Pytorch1.1.0 based on python3.6. The indoor scene image database NYUv2 test set is used to analyze the segmentation effect of the indoor scene images (699 pairs of indoor scene images) predicted by the method of the present invention. Here, three commonly used objective parameters for evaluating semantic segmentation methods are used as evaluation indicators, namely class accuracy (Class Acurracy), mean pixel accuracy (MeanPixel Accuracy, MPA), and the ratio of the intersection and union of the segmented image and the label image (Mean Intersection overUnion, MIoU) to evaluate the segmentation performance of the predicted semantic segmentation image. The higher the values ​​of the three indicators, the better the model performance in the present invention.

[0097] The method of the present invention is used to predict each pair of indoor scene images in the test set of the indoor scene image database NYUv2, and the predicted semantic segmentation images corresponding to each pair of indoor images are obtained. The class accuracy CA, average pixel accuracy MPA, and the ratio of the intersection and union of the segmented image and the label image MIoU reflecting the semantic segmentation effect of the method of the present invention are listed in Table 1. It can be seen from the data in Table 1 that the method of the present invention has achieved good results in the three indicators of class accuracy CA, average pixel accuracy MPA, and the ratio of the intersection and union of the segmented image and the label image MIoU, which shows the effectiveness of the method of the present invention.

[0098] Table 1 Evaluation results of the method of the present invention on the test set

[0099] CA 61.01％ MPA 74.34％ MIoU 47.92％

[0100] Figure 2a The first original indoor scene color RGB image of the same scene is given. Figure 2b The first original indoor scene depth image of the same scene is given. Figure 2c The method of the present invention is given Figure 2a and 2b The original indoor scene image shown is predicted to obtain a predicted semantic segmentation image; Figure 3a The second original indoor scene color RGB image of the same scene is given. Figure 3b The second original indoor scene depth image of the same scene is given. Figure 3c The method of the present invention is given Figure 3a and 3b The original indoor scene image shown is predicted to obtain a predicted semantic segmentation image; Figure 4a The third original indoor scene color RGB image of the same scene is given. Figure 4bThe third original indoor scene depth image of the same scene is given. Figure 4c The method of the present invention is given Figure 4a and 4b The original indoor scene image shown is predicted to obtain a predicted semantic segmentation image; Figure 5a The fourth original indoor scene color RGB image of the same scene is given. Figure 5b The fourth original indoor scene depth image of the same scene is given. Figure 5c The method of the present invention is given Figure 5a and 5b The original indoor scene image shown in Figure 1 is used for prediction to obtain the predicted semantic segmentation image. Figure 2a , 2b and Figure 2c ,contrast Figure 3a , 3b and Figure 3c ,contrast Figure 4a , 4b and Figure 4c ,contrast Figure 5a , 5b and Figure 5c It can be seen that the segmentation accuracy of the predicted semantic segmentation image obtained by the method of the present invention is high.

Claims

1. Indoor scene semantic segmentation method based on improved fully convolutional neural network, characterized by: The following steps are involved: Step 1: Select Q pairs of original indoor scene images and their corresponding real semantic segmentation images, and form a training set with all the original indoor scene images and their corresponding real semantic segmentation images; each pair of original indoor scene images includes an original indoor scene color image and an original indoor scene depth image, and the real semantic segmentation images in the training set are processed into 41 one-hot encoded images using the one-hot encoding technology; Step 2: Construct a convolutional neural network classification training model: The convolutional neural network classification training model includes an input layer, a hidden layer, and an output layer; the input layer includes a color image input layer and a depth image input layer; the hidden layer includes a color image processing module and a depth image processing module; the color image processing module and the depth image processing module have symmetrical structures, each including five neural network blocks, five feature re-extraction convolution blocks, and ten fusion layers; the hidden layer also includes five block attention convolution blocks, four upsampling layers, and two fusion layers; Step 3: Input the training set into the convolutional neural network classification training model in step 2 for training. During the training process, each iterative training process obtains 41 semantic segmentation prediction images corresponding to each pair of original indoor scene images, and calculates the loss function value between the set of 41 semantic segmentation prediction images and the set of 41 one-hot encoded images corresponding to the true semantic segmentation images; Step 4: Repeat step 3 for V times and obtain Q×V loss function values ​​in total; Then, the minimum loss function value is found from the Q×V loss function values, and the weight vector and bias item corresponding to the minimum loss function value are used as the optimal weight vector and optimal bias item of the convolutional neural network classification training model, thereby completing the training of the convolutional neural network classification training model; Step 5: Use the convolutional neural network classification training model obtained after training to perform prediction processing on the indoor scene image to be predicted, and output the corresponding predicted semantic segmentation image to achieve semantic segmentation of the indoor scene image; The step 2 is specifically as follows: The color image input layer and the depth image input layer are respectively input into the first neural network block in the color image processing module and the depth image processing module; The structure of the color image processing module and the depth image processing module is the same, specifically: One of the outputs of the first neural network block is input into the first fusion layer via the first feature re-extraction convolution block, and the other output of the first neural network block is input into the first fusion layer; one of the outputs of the second neural network block is input into the third fusion layer via the second feature re-extraction convolution block, and the other output of the second neural network block is input into the third fusion layer; one of the outputs of the third neural network block is input into the fifth fusion layer via the third feature re-extraction convolution block, and the other output of the third neural network block is input into the fifth fusion layer; one of the outputs of the fourth neural network block is input into the seventh fusion layer via the fourth feature re-extraction convolution block, and the other output of the fourth neural network block is input into the seventh fusion layer; one of the outputs of the fifth neural network block is input into the ninth fusion layer via the fifth feature re-extraction convolution block, and the other output of the fifth neural network block is input into the ninth fusion layer; the two inputs of each fusion layer are fused by element-by-element addition; The output of the first fusion layer is respectively input into the first block attention convolution block and the corresponding second fusion layer, the output of the third fusion layer is respectively input into the second block attention convolution block and the corresponding fourth fusion layer, the output of the fifth fusion layer is respectively input into the third block attention convolution block and the corresponding sixth fusion layer, the output of the seventh fusion layer is respectively input into the fourth block attention convolution block and the corresponding eighth fusion layer, and the output of the ninth fusion layer is respectively input into the fifth block attention convolution block and the corresponding tenth fusion layer; The two outputs of the first block attention convolution block are respectively input into the second fusion layer of the color image processing module and the depth image processing module, the two outputs of the second block attention convolution block are respectively input into the fourth fusion layer of the color image processing module and the depth image processing module, the two outputs of the third block attention convolution block are respectively input into the sixth fusion layer of the color image processing module and the depth image processing module, the two outputs of the fourth block attention convolution block are respectively input into the eighth fusion layer of the color image processing module and the depth image processing module, and the two outputs of the fifth block attention convolution block are respectively input into the tenth fusion layer of the color image processing module and the depth image processing module; The two inputs of the second fusion layer are fused by element-by-element addition and are respectively input into the eleventh fusion layer and the corresponding second neural network block. The two inputs of the fourth fusion layer are fused by element-by-element addition and are respectively input into the first upsampling layer and the corresponding third neural network block. The two inputs of the sixth fusion layer are fused by element-by-element addition and are respectively input into the second upsampling layer and the corresponding fourth neural network block. The two inputs of the eighth fusion layer are fused by element-by-element addition and are respectively input into the third upsampling layer and the corresponding fifth neural network block. The output of the tenth fusion layer is input into the fourth upsampling layer. The two inputs of the eleventh fusion layer, the first upsampling layer, the second upsampling layer, the third upsampling layer, and the fourth upsampling layer are fused by element-by-element addition and then input into the twelfth fusion layer; The twelfth fusion layer concatenates all inputs and outputs them through the output layer. The output layer is mainly composed of the convolutional layer and the fifth upsampling layer connected in sequence.

2. The method for indoor scene semantic segmentation based on an improved fully convolutional neural network according to claim 1, characterized in that: The five neural network blocks adopt the MobileNet V2 network structure. The first neural network block adopts 1 to 4 layers of MobileNetV2, the second neural network block adopts 5 to 7 layers of MobileNetV2, the third neural network block adopts 8 to 11 layers of MobileNetV2, the fourth neural network block adopts 12 to 14 layers of MobileNetV2, and the fifth neural network block adopts 15 to 17 layers of MobileNetV2.

3. The method for indoor scene semantic segmentation based on an improved fully convolutional neural network according to claim 1, characterized in that: Each of the feature re-extraction convolution blocks is composed of four re-extraction modules connected in sequence, and each re-extraction module includes a convolution layer, a normalization layer, and an activation layer connected in sequence; All activation layers use ReLU6 as the activation method; all convolutional layers have a step size of 1; the number of convolution kernels in each re-extraction module is the same as the normalization parameter of the normalization layer; The convolution kernel size of the convolution layer in the first re-extraction module, the third re-extraction module, and the fourth re-extraction module is the same, and the convolution kernel size of the convolution layer in the second re-extraction module is half of the convolution kernel size of the convolution layer in the first re-extraction module; The expansion factor of the convolutional layer in the first re-extraction module is 1, and the expansion factor of the convolutional layer in the third re-extraction module is 2; the zero-padding parameter of the convolutional layer in the first re-extraction module is 1, the zero-padding parameter of the convolutional layer in the third re-extraction module is 2, and the zero-padding parameters of the convolutional layer in the second and fourth re-extraction modules are both 0.

4. The method for indoor scene semantic segmentation based on improved fully convolutional neural network according to claim 3, characterized in that: The convolution kernel size of the convolution layer of the first re-extraction module in the first feature re-extraction convolution block is 24, the convolution kernel size of the convolution layer of the first re-extraction module in the second feature re-extraction convolution block is 32, the convolution kernel size of the convolution layer of the first re-extraction module in the third feature re-extraction convolution block is 64, the convolution kernel size of the convolution layer of the first re-extraction module in the fourth feature re-extraction convolution block is 96, and the convolution kernel size of the convolution layer of the first re-extraction module in the fifth feature re-extraction convolution block is 160; The grouping parameters of the four convolutional layers set sequentially in the first feature re-extraction convolution block are 24, 12, 24, and 24 respectively; the grouping parameters of the four convolutional layers set sequentially in the second feature re-extraction convolution block are 32, 16, 16, and 32 respectively; the grouping parameters of the four convolutional layers set sequentially in the third feature re-extraction convolution block are 64, 32, 32, and 64 respectively; the grouping parameters of the four convolutional layers set sequentially in the fourth feature re-extraction convolution block are 64, 48, 48, and 99 respectively; and the grouping parameters of the four convolutional layers set sequentially in the fifth feature re-extraction convolution block are 96, 80, 80, and 160 respectively.

5. The method for indoor scene semantic segmentation based on improved fully convolutional neural network according to claim 1, characterized in that: Each of the block attention convolution blocks includes a block layer, a channel attention layer and a spatial size attention layer. The input of the block attention convolution block is respectively input into the channel attention layer and the spatial size attention layer after passing through the block layer. The input of the block attention convolution block is multiplied with the output of the channel attention layer and the output of the spatial size attention layer respectively, and then the multiplication results are added as the output of the block attention convolution block.

6. The method for indoor scene semantic segmentation based on improved fully convolutional neural network according to claim 5, characterized in that: The block layer uses the split function provided by pytorch, and the parameter is half of the number of channels corresponding to the input feature map of the block attention convolution block; The channel attention layer includes an adaptive maximum pooling layer, a channel attention first convolution layer, a channel attention second convolution layer and a channel attention activation layer connected in sequence; the maximum pooling parameter of the adaptive maximum pooling layer is 1; the convolution kernel size of the channel attention first convolution layer and the channel attention second convolution layer are both 1x1, the step size is 1, the bias item is False, the grouping parameters are the same, and the number of convolution kernels in the channel attention first convolution layer is half of the number of convolution kernels in the channel attention second convolution layer; The number of convolution kernels of the first channel attention convolution layer of the channel attention layer in the first block attention convolution block is 12, and the grouping parameter is 12; the number of convolution kernels of the first channel attention convolution layer of the channel attention layer in the second block attention convolution block is 16, and the grouping parameter is 16; the number of convolution kernels of the first channel attention convolution layer of the channel attention layer in the third block attention convolution block is 32, and the grouping parameter is 32; the number of convolution kernels of the first channel attention convolution layer of the channel attention layer in the fourth block attention convolution block is 48, and the grouping parameter is 48; the number of convolution kernels of the first channel attention convolution layer of the channel attention layer in the fifth block attention convolution block is 80, and the grouping parameter is 80; The spatial-size attention layer includes a channel-wise maximization layer, a spatial-size attention convolution layer, and a spatial-size attention activation layer connected in sequence; the channel-wise maximization layer uses the max function that comes with pytorch; the convolution kernel size of the spatial-size attention convolution layer is 3×3, the number of convolution kernels is 1, the expansion factor is 2, the zero-padding parameter is 2, the grouping parameter is 1, and the bias term is False; The activation method used by the channel attention activation layer and the spatial size attention activation layer is the Sigmoid function.

7. The method for indoor scene semantic segmentation based on an improved fully convolutional neural network according to claim 1, characterized in that: The upsampling layer adopts the UpsamlingBilinear2d function in pytorch. The function parameter of the first upsampling layer is 2, and the function parameter of the second upsampling layer, the third upsampling layer, the fourth upsampling layer and the fifth upsampling layer is 4; the convolution kernel size of the convolution layer in the output layer is 1×1, the number of convolution kernels is 41, the step size is 1, and the bias item is False.

8. The method for indoor scene semantic segmentation based on improved fully convolutional neural network according to claim 3, characterized in that: The input end of the color image input layer receives the color image of the indoor scene, the input end of the depth image input layer receives the depth image of the indoor scene, and the output of the output layer is 41 semantic segmentation prediction images corresponding to the indoor scene images input by the input layer.

Images