Forest unstructured scene segmentation method based on multispectral image fusion

Abstract

The present invention discloses a forest unstructured scene segmentation method based on multispectral image fusion. Aiming at the situations of complex background, multiple light and shadow interference in forest unstructured scenes, the method introduces enhanced vegetation index (EVI) data, designs a parallel dual encoding structure to extract RGB and EVI features, and forms multimodal complementary features in the encoding process. In addition, the encoding stage also uses dilated convolution to increase the receptive field of the network and optimize the feature extraction process. Then, the decoded features are fused with the fused features generated by the encoding for a second time to obtain a multispectral fusion convolutional neural network, and then the network is trained and RGB and EVI images are input to realize the semantic segmentation of forest unstructured scenes. The present invention effectively solves the problems of poor adaptability and mis-segmentation of unstructured segmentation methods based on RGB images, and improves the accuracy and robustness of semantic segmentation of forest unstructured scenes.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and driverless car environment perception, and relates to an unstructured scene segmentation method, and in particular to a forest unstructured scene segmentation method based on multispectral image fusion. Background Art

[0002] In recent years, autonomous driving technology has developed rapidly and has become a hot area for scholars from all walks of life to study and solve traffic problems. During driving, providing comprehensive, accurate and reliable perception for autonomous driving vehicles is a prerequisite for achieving safe and reliable autonomous driving. Among them, semantic segmentation technology has been widely studied as an important part of autonomous driving scene perception technology. At present, in the field of autonomous driving vehicle perception, the more mature visual semantic segmentation technology is mainly designed for structured road environments, such as urban road environments. However, in real scenes, there are also a large number of unstructured scenes, such as forest scenes and off-road scenes. More complex unstructured environments will force autonomous driving vehicles to make more intelligent decisions, which puts higher requirements on semantic segmentation technology.

[0003] Considering factors such as cost and feasibility, most current semantic segmentation methods are based on computer vision. In unstructured environments, there are no structural features such as traffic lanes, paved roads, or guardrails. In addition, the drivable areas of unstructured scenes have fuzzy boundaries, complex textures, and complex features. They are also affected by natural factors such as shadows and glare. These characteristics pose considerable challenges to visual semantic segmentation. Traditional image segmentation methods mostly use the surface information of images and require artificial construction of appropriate target features. They are not applicable to segmentation tasks that require a large amount of semantic information and complex scenes, and cannot meet actual needs.

[0004] Compared with traditional image segmentation algorithms, methods based on deep convolutional neural networks can learn more complex and abstract high-level semantic features and can cope with more complex scenes. However, most of the current vision-based unstructured scene segmentation models only use a single RGB image, which is difficult to provide comprehensive and effective knowledge for complex unstructured scene segmentation. Moreover, methods based on a single RGB image are easily affected by light and it is difficult to obtain accurate and stable segmentation. Summary of the invention

[0005] To solve the above problems, the present invention discloses a forest unstructured scene segmentation method based on multispectral image fusion, which effectively solves the problem that most of the current segmentation methods based on RGB images are difficult to apply to complex unstructured scenes. By using different perception methods to construct a feature description of the same scene, the accuracy and robustness of forest unstructured scene segmentation are further improved.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] The forest unstructured scene segmentation method based on multispectral image fusion includes the following steps:

[0008] Step 1: Establish a multispectral segmentation dataset for unstructured forest scenes.

[0009] Step 2: Construct a multispectral feature fusion encoding network. First, design two parallel encoder branches for feature extraction of RGB data and EVI data respectively. During the encoding process, the RGB branch will hierarchically fuse the hierarchical features of the EVI (enhanced vegetation index data) branch to achieve complementary feature advantages. Then construct a jump connection structure to perform a secondary fusion of the fused features in the encoding stage with the decoded features to compensate for the spatial information features and edge detail features lost during the downsampling process. It includes the following sub-steps:

[0010] Sub-step 1: Design the EVI branch coding structure, including:

[0011] (1) Design and initialize the convolution layer. The number of initial convolution layers is 1, the number of convolution kernels is 16, the size of the convolution kernel is 7×7, and the step size is 2.

[0012] (2) Design residual modules. The number of residual modules is 3. The network structures of residual modules 1 to 3 are the same, where the number of convolution kernels is 64, 128, and 256 respectively. The number of convolution kernels is represented by n. The residual module includes a standard convolution module, a separation residual module, and a dilated convolution module. These three modules can be constructed as follows:

[0013] ①Build a standard convolution module:

[0014] Design standard convolution: The number of standard convolutions is 2, the number of convolution kernels is n, the size of the convolution kernel is 3×3, and the step sizes are 2 and 1 respectively.

[0015] Construct a standard convolution module a-1: Use n 3×3 standard convolution kernels to perform convolution with the input feature map with a step size of 2, and then pass through the BN layer and Re LU activation, and the output feature size is 1 / 2 of the input.

[0016] Construct standard convolution module a-2: Use n 3×3 convolution kernels to convolve with the feature map of standard convolution module a with a step size of 1, and then pass through the BN layer and Re LU activation to obtain a feature map with unchanged dimension.

[0017] Construct a standard convolution block: Cascade the standard convolution module a-1 and the standard convolution module a-2 to obtain a standard convolution block, and the output feature size is 1 / 2 of the input.

[0018] ②Build a separation residual module

[0019] Design separation convolution: the standard separation number is 4, where the number of convolution kernels is n, and the convolution kernel sizes are 3×1, 1×3, 3×1, 1×3, and the step size is 1.

[0020] Construct separation residual a-1: Use n 3×1 convolution kernels to convolve with the output feature map of the standard convolution block with a step size of 1, and then pass through the BN layer and Re LU activation to obtain a feature map with unchanged dimension;

[0021] Construct separation residual a-2: Use n 1×3 convolution kernels to convolve with the output feature map of separation residual a-1 with a step size of 1, and then pass through the BN layer and Re LU activation to obtain a feature map with unchanged dimension;

[0022] Construct separation residual a-3: Use n 3×1 convolution kernels to convolve with the output feature map of separation residual a-2 with a step size of 1, and then pass through the BN layer and Re LU activation to obtain a feature map with unchanged dimension;

[0023] Construct separation residual a-4: Use n 1×3 convolution kernels to convolve with the output feature map of separation residual a-3 with a step size of 1, and then pass through the BN layer and Re LU activation to obtain a feature map with unchanged dimension;

[0024] Finally, the output feature map of the standard convolutional block is added to the output of the separation residual a-4 to obtain the output of the separation residual module.

[0025] ③Construct dilated residual module

[0026] Design dilated convolutional layers, the number of dilated convolutional layers is 2, the dilation rates are 3 and 5 respectively, the number of convolution kernels is n, the size of convolution kernels is 3×3, and the step size is 1;

[0027] Construct the dilated residual module a-1: Use n 3×3 standard convolution kernels to convolve with the output feature map of the separation residual module with a step size of 1, and then pass through the BN layer and Re LU activation to obtain a feature map with unchanged dimension.

[0028] Construct the dilated residual module a-2: Use n 3×3 convolution kernels to perform convolution with the feature map of the dilated residual module a-1 with a step size of 1, and then pass through the BN layer and Re LU activation to obtain a feature map with unchanged dimension.

[0029] Finally, the output feature map of the separation residual module is added to the output of the dilated residual module a-2 to obtain the output of the dilated residual module.

[0030] ④Construct residual module 1:

[0031] Use a standard convolutional module with n=64 to convolve the output of the initialization convolutional layer of A / 2×A / 2 pixels to obtain a feature map with a dimension of A / 4×A / 4×64;

[0032] Convolve the output of the standard convolutional module with A / 4×A / 4 pixels using the separation residual module with n=64 to obtain a feature map with a dimension of A / 4×A / 4×64;

[0033] Convolve the output of the dilated residual module with n=64 pixels and the separated residual module with A / 4×A / 4 pixels to obtain a feature map with a dimension of A / 4×A / 4×64;

[0034] ⑤Build residual module 2:

[0035] Convolve the output of residual module 1 with A / 4×A / 4 pixels using a standard convolution module with n=128 to obtain a feature map with dimensions of A / 8×A / 8×128.

[0036] Convolve the output of the standard convolutional module with A / 8×A / 8 pixels using the separation residual module with n=128 to obtain a feature map with a dimension of A / 8×A / 8×128;

[0037] Convolve the output of the dilated residual module with n=128 pixels and the separated residual module with A / 8×A / 8 pixels to obtain a feature map with a dimension of A / 8×A / 8×128;

[0038] ⑥Build residual module 3:

[0039] Convolve the output of the residual module 2 with A / 8×A / 8 pixels using a standard convolution module with n=256 to obtain a feature map with dimensions of A / 16×A / 16×256;

[0040] Convolve the output of the standard convolutional module with A / 16×A / 16 pixels using the separation residual module with n=256 to obtain a feature map with dimensions of A / 16×A / 16×256;

[0041] Convolve the output of the dilated residual module with n=256 pixels and the separated residual module with A / 16×A / 16 pixels to obtain a feature map with the dimension of A / 16×A / 16×256.

[0042] (3) Constructing EVI branch coding structure

[0043] Convolve the initialization convolution layer with the EVI input sample of A×A pixels to obtain the feature map a1 with the dimension of A / 2×A / 2×16. The output of the initialization convolution layer passes through the residual module 1 to obtain the feature map a2 with the dimension of A / 4×A / 4×64. The output of the residual module 1 passes through the residual module 2 to obtain the feature map a3 with the dimension of A / 8×A / 8×128. The output of the residual module 2 passes through the residual module 3 to obtain the feature map a4 with the dimension of A / 16×A / 16×256.

[0044] Sub-step 2: Design the RGB branch coding structure, including:

[0045] (1) Design and initialize the convolutional layer, which is consistent with step (1) of sub-step 1.

[0046] (2) Design the residual module. This is the same as step (2) in sub-step 1.

[0047] (3) Constructing RGB branch coding structure

[0048] Convolve the initialized convolution layer with the RGB input sample of A×A pixels to obtain the feature map b1 with a dimension of A / 2×A / 2×16. Add a1 and b1 obtained from the EVI branch and pass through residual module 1 to obtain feature map b2 with a dimension of A / 4×A / 4×64. Add a2 and b2 obtained from the EVI branch and pass through residual module 2 to obtain feature map b3 with a dimension of A / 8×A / 8×128. Add a3 and b3 obtained from the EVI branch and pass through residual module 3 to obtain feature map b4 with a dimension of A / 16×A / 16×256.

[0049] Step 3: Construct a hierarchical feature fusion decoding structure. First, design an upsampling layer to restore the upsampling layer and dense prediction. Then introduce the fusion features of RGB and EVI in the encoder stage and fuse them with the decoding features for the second time. It includes the following sub-steps:

[0050] Sub-step 1: Design a standard convolution layer. The number of standard convolutions is 4, where the number of convolution kernels is 128, 64, 16, and class_num, the size of the convolution kernel is 3×3, and the step size is 1.

[0051] Sub-step 2: Construct upsampling layer 1. The encoder output feature b4 is subjected to bilinear interpolation to obtain a feature map with a dimension of A / 8×A / 8×256. The standard convolution layer is convolved with the bilinear interpolation output of A / 8×A / 8 pixels to obtain a feature map with a dimension of A / 8×A / 8×128.

[0052] Sub-step 3: Construct upsampling layer 2. The features output by upsampling layer 1 are first concatenated with b3 to obtain a feature map of A / 8×A / 8×256, and then bilinearly interpolated to obtain a feature map of dimension A / 4×A / 4×256. The standard convolutional layer is convolved with the bilinear interpolation output of A / 4×A / 4 pixels to obtain a feature map of dimension A / 4×A / 4×64;

[0053] Sub-step 4: Construct upsampling layer 3. The features output by upsampling layer 2 are first concatenated with b2 to obtain a feature map of A / 4×A / 4×128, and then bilinearly interpolated to obtain a feature map of dimension A / 2×A / 2×128. The standard convolution layer is convolved with the bilinear interpolation output of A / 2×A / 2 pixels to obtain a feature map of dimension A / 2×A / 2×16;

[0054] Sub-step 5: Construct upsampling layer 4. The features output by upsampling layer 1 are first concatenated with b1 to obtain a feature map of A / 2×A / 2×32, and then bilinear interpolation is performed to obtain a feature map of dimension A×A×32. The standard convolution layer is convolved with the bilinear interpolation output of A×A pixels to obtain a feature map of dimension A×A×class_num;

[0055] Step 4: Train the designed multispectral image fusion convolutional neural network to obtain network parameters;

[0056] Step 5: Use multispectral image fusion convolutional neural network to perform forest unstructured scene segmentation.

[0057] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0058] The present invention proposes a semantic segmentation method suitable for unstructured forest scenes, and introduces enhanced vegetation index data (EVI) based on visible light RGB data. On the one hand, EVI can eliminate common interference factors (such as glare, shadows, weather changes, etc.) in complex environments and enhance the robustness of the algorithm. On the other hand, EVI can enhance the identification of passable areas and obstacles in unstructured forest scenes by providing data information of vegetation in unstructured scenes.

[0059] The present invention designs a parallel dual-branch coding convolutional neural network acting on RGB and EVI data to extract features. In the encoding process, the dilated convolution is used to increase the receptive field of the network, optimize the feature extraction process, and construct a mixed complementary feature by layered fusion of RGB and EVI features by two branches; in the decoding stage, the lightweight decoder layered fuses the features generated in the encoding stage to achieve secondary feature fusion and obtain a multi-spectral data fusion convolutional neural network. This network not only greatly reduces the impact of common interference on segmentation accuracy, but also takes into account local detail features and global features of different scales. Compared with most of the current unstructured scene semantic segmentation methods based on single vision, it effectively solves the problems of poor network adaptability and mis-segmentation, and improves the accuracy and robustness of unstructured scene semantic segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 It is a design flow chart of the multispectral image fusion convolutional neural network of the present invention.

[0061] Figure 2 It is a schematic diagram of the multispectral image fusion convolutional neural network structure designed by the present invention.

[0062] Figure 3 It is a multispectral image fusion convolutional neural network training flow chart designed by the present invention. DETAILED DESCRIPTION

[0063] The technical solution provided by the present invention will be described in detail below in conjunction with specific embodiments. It should be understood that the following specific implementation methods are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0064] The present invention discloses a forest unstructured scene segmentation method based on multispectral image fusion. The method designs a parallel encoder and a two-level fusion strategy, and realizes hierarchical feature fusion in the encoding and decoding process. First, in the encoder stage, two encoding branches process RGB data and EVI data respectively, and fuse two-way features at each layer to obtain complementary features with rich information. The encoder also uses dilated convolution to increase the receptive field of the network and optimize the feature extraction process. Then, the high-semantic features are decoded, and the low-semantic, high-detail features generated by encoding are secondary fused with the high-semantic, low-detail features generated by decoding during the decoding process, and finally a convolutional neural network with multispectral image fusion is obtained, which effectively solves the problem that most of the current segmentation methods based on RGB images are difficult to apply to complex unstructured scenes, and improves the accuracy and robustness of segmentation.

[0065] Specifically, the forest unstructured scene segmentation method based on multispectral image fusion provided by the present invention comprises the following steps:

[0066] Step 1: Establish a multispectral dataset of unstructured forest scenes, annotate the unstructured scene samples or use existing data samples, and then adjust the sample size to 480×640 pixels and record it as D k .

[0067] Step 2: Construct a multi-spectral feature fusion encoding network architecture. The encoding structure consists of two encoders. The EVI encoder is used to extract the features of the EVI data, and the RGB encoder extracts the features of the RGB data and hierarchically fuses the two features during the encoding process. The two branch features are hierarchically fused to form information-rich complementary features. The design flow chart is as follows: Figure 1 As shown. It includes the following steps

[0068] Sub-step 1: Design the EVI branch coding structure, including:

[0069] (1) Design and initialize the convolution layer. In convolutional neural networks for semantic segmentation, one of the keys to achieving good performance and real-time operation is to handle large input frames. If the initial convolution kernel is too small, the subsequent convolution features will contain less global information. However, if the convolution kernel is too large, it will lead to more parameters and increase the computational burden. Therefore, the design takes these two aspects into consideration and initializes the number of convolution layers to 1, the number of convolution kernels to 16, the size of the convolution kernel to 7×7, and the step size to 2.

[0070] Use 16 7×7 convolution kernels to convolve with the 480×640 pixel input sample with a step size of 2, and then activate it with ReLU to obtain a feature map a1 with a dimension of 240×320×16;

[0071] (2) Design of residual module. In order to accurately extract features from images, a residual module consisting of a cascade of standard convolution, separation convolution and dilated convolution is designed. Generally speaking, the feature expression capability of a network increases with the number of convolutional layers, but stacking more convolutional layers increases the amount of computation, while a small number of convolutional layers makes it difficult to extract features with strong expression capabilities. Therefore, the present invention adds separation convolution, which reduces convolution parameters and computation while increasing network depth and ensuring feature extraction capabilities. In addition, in order to expand the convolution field of view and consider global information, a dilated convolution module is designed. The number of residual modules is 3, and the network structures of residual modules 1 to 3 are consistent, wherein the number of convolution kernels are 64, 128, and 256, respectively, and the number of convolution kernels is represented by n. Each residual module includes a standard convolution module, a separation residual module and a dilated convolution module, and these three modules can be constructed as follows:

[0072] ①Build a standard convolution module:

[0073] Design of standard convolution: In order to retain more feature information, increase network depth and learnability, the standard convolution block consists of 2 standard convolutions. After the first standard convolution, the input feature size is reduced to 1 / 2 of the original, and the feature dimension remains unchanged after the second convolution operation. The number of convolution kernels is n, the convolution kernel size is 3×3, and the step sizes are 2 and 1 respectively.

[0074] Construct a standard convolution module a-1: Use n 3×3 standard convolution kernels to perform convolution with the input feature map with a step size of 2, and then pass through the BN layer and Re LU activation, and the output feature size is 1 / 2 of the input.

[0075] Construct standard convolution module a-2: Use n 3×3 convolution kernels to convolve with the feature map of standard convolution module a with a step size of 1, and then pass through the BN layer and Re LU activation to obtain a feature map with unchanged dimension.

[0076] Construct a standard convolution block: Cascade the standard convolution module a-1 and the standard convolution module a-2 to obtain a standard convolution block, and the output feature size is 1 / 2 of the input.

[0077] ②Build a separation residual module

[0078] Design of separation convolution: As the network depth increases, the network becomes more complex and the amount of calculation increases. Decomposition convolution can compress a two-dimensional convolution kernel into two one-dimensional convolution kernels, greatly reducing feature redundancy and thus speeding up network operation. However, frequent use of decomposition convolution can easily cause excessive feature compression and destroy the continuity of features. The number of separation convolutions is 4, of which the number of convolution kernels is n, and the sizes of the convolution kernels are 3×1, 1×3, 3×1, and 1×3, respectively, with a step size of 1.

[0079] Construct separation residual a-1: Use n 3×1 convolution kernels to convolve with the output feature map of the standard convolution block with a step size of 1, and then pass through the BN layer and Re LU activation to obtain a feature map with unchanged dimension;

[0080] Construct separation residual a-2: Use n 1×3 convolution kernels to convolve with the output feature map of separation residual a-1 with a step size of 1, and then pass through the BN layer and Re LU activation to obtain a feature map with unchanged dimension;

[0081] Construct separation residual a-3: Use n 3×1 convolution kernels to convolve with the output feature map of separation residual a-2 with a step size of 1, and then pass through the BN layer and Re LU activation to obtain a feature map with unchanged dimension;

[0082] Construct separation residual a-4: Use n 1×3 convolution kernels to convolve with the output feature map of separation residual a-3 with a step size of 1, and then pass through the BN layer and Re LU activation to obtain a feature map with unchanged dimension;

[0083] Finally, the output feature map of the standard convolutional block is added to the output of the separation residual a-4 to obtain the output of the separation residual module.

[0084] ③Construct dilated residual module

[0085] Design of dilated convolution layer: Dilated convolution has the advantage of expanding the local receptive field of the network, and its optimized design can realize compensation for large target location information. Although stacking dilated convolution layers and using a large dilation rate increase the local receptive field, noise is introduced, and the receptive field is so large that the network ignores the local detail information of the road, resulting in discontinuous segmentation or even missed segmentation. More importantly, in order to eliminate the negative impact of the "grid effect" of dilated convolution on feature compensation, the dilation rate of dilated convolution is optimized to satisfy the sawtooth distribution. Based on the above considerations, the present invention sets the number of dilated convolution layers to 2, the dilation rates to 3 and 5 respectively, the number of convolution kernels to n, the convolution kernel size to 3×3, and the step size to 1;

[0086] Construct the dilated residual module a-1: Use n 3×3 standard convolution kernels to convolve with the output feature map of the separation residual module with a step size of 1, and then pass through the BN layer and Re LU activation to obtain a feature map with unchanged dimension.

[0087] Construct the dilated residual module a-2: Use n 3×3 convolution kernels to perform convolution with the feature map of the dilated residual module a-1 with a step size of 1, and then pass through the BN layer and Re LU activation to obtain a feature map with unchanged dimension.

[0088] Finally, the output feature map of the separation residual module is added to the output of the dilated residual module a-2 to obtain the output of the dilated residual module.

[0089] ④Construct residual module 1:

[0090] Use the standard convolution module with n=64 to convolve with the output of the initialized convolution layer to obtain a feature map a2_1 with a dimension of 120×160×64;

[0091] Convolve the output a2_1 of the standard convolutional module with the separation residual module of n=64 to obtain a feature map a2_2 with a dimension of 120×160×64;

[0092] Convolve the output a2_2 of the dilated residual module with n=64 and the separated residual module to obtain a feature map a2 with a dimension of 120×160×64;

[0093] ⑤Build residual module 2:

[0094] Use the standard convolution module with n=128 to convolve with the output of residual module 1 to obtain a feature map a3_1 with a dimension of 60×80×128;

[0095] Convolve the output a3_1 of the standard convolutional module with the separation residual module of n=128 to obtain a feature map a3_2 with a dimension of 60×80×128;

[0096] Convolve the output a3_2 of the dilated residual module with n=128 and the separated residual module to obtain a feature map a3 with a dimension of 60×80×128;

[0097] ⑥Build residual module 3:

[0098] Convolve the output of residual module 2 with the standard convolution module of n=256 to obtain a feature map a4_1 with a dimension of 30×40×256;

[0099] Convolve the output a4_1 of the standard convolutional module with the separation residual module of n=256 to obtain a feature map a4_2 with a dimension of 30×40×256;

[0100] Convolve the dilated residual module with n=256 and the output a4_2 of the separated residual module to obtain a feature map a4 with a dimension of 30×40×256;

[0101] (3) Construct EVI branch coding structure:

[0102] The initialization convolution layer is convolved with the EVI input sample of dimension 480×640×3 to obtain the feature map a1 of dimension 240×320×16. The output of the initialization convolution layer passes through residual module 1 to obtain the feature map a2 of dimension 120×160×64. The output of residual module 1 passes through residual module 2 to obtain the feature map a3 of dimension 60×80×128. The output of residual module 2 passes through residual module 3 to obtain the feature map a4 of dimension 30×40×256.

[0103] Sub-step 2: Design the RGB branch coding structure, including:

[0104] (1) Design and initialize the convolutional layer, which is consistent with step (1) of sub-step 1.

[0105] (2) Design the residual module. This is the same as step (2) in sub-step 1.

[0106] (3) Constructing RGB branch coding structure

[0107] Convolve the initialized convolutional layer with the RGB input sample of dimension 480×640×3 to obtain feature map b1 of dimension 240×320×16. Add a1 and b1 obtained from the EVI branch and pass through residual module 1 to obtain feature map b2 of dimension 120×160×64. Add a2 and b2 obtained from the EVI branch and pass through residual module 2 to obtain feature map b3 of dimension 60×80×128. Add a3 and b3 obtained from the EVI branch and pass through residual module 3 to obtain feature map b4 of dimension 30×40×256.

[0108] Step 3: Construct a hierarchical feature fusion decoding structure. In steps 1 and 2, the operation of reducing the feature size due to convolution occurs 4 times, so that the feature map output by the final encoder is 1 / 16 of the input. In order to restore the feature map to the original image size and avoid introducing a lot of noise, a quadratic interpolation and convolution cascade are used to form an upsampling layer for restoring resolution and dense prediction. In order to make up for the spatial feature information and edge detail feature information lost during the downsampling process, the fusion features of RGB and EVI in the encoder stage are introduced and fused with the decoding features for a second time. The design flow chart is as follows: Figure 1 As shown. It includes the following sub-steps:

[0109] Sub-step 1: Design a standard convolution layer. The number of standard convolutions is 4, where the number of convolution kernels is 128, 64, 16, and class_num, the size of the convolution kernel is 3×3, and the step size is 1.

[0110] Sub-step 2: Construct upsampling layer 1. The encoder output feature b4 is subjected to bilinear interpolation to obtain a feature map d1 with a dimension of 60×80×256. The standard convolution layer is convolved with the bilinear interpolation output d1 to obtain a feature map with a dimension of 60×80×128;

[0111] Sub-step 3: Construct upsampling layer 2. The features output by upsampling layer 1 are first concatenated with b3 to obtain a 60×80×256 feature map, and then bilinear interpolation is performed to obtain a feature map d2 with a dimension of 120×160×256. The standard convolution layer is convolved with the output d2 of bilinear interpolation to obtain a feature map with a dimension of 120×160×64;

[0112] Sub-step 4: Construct upsampling layer 3. The features output by upsampling layer 2 are first concatenated with b2 to obtain a feature map of 120×160×128, and then subjected to bilinear interpolation to obtain a feature map d3 of 240×320×128. The standard convolution layer is convolved with the output d3 of bilinear interpolation to obtain a feature map of 240×320×16;

[0113] Sub-step 5: Construct upsampling layer 4. The features output by upsampling layer 3 are first concatenated with b1 to obtain a feature map of 240×320×32, and then bilinear interpolation is performed to obtain a feature map d4 with a dimension of 480×640×32. The standard convolution layer is convolved with the output d4 of bilinear interpolation to obtain a feature map with a dimension of 480×640×nums_class, where nums_class represents the target category of semantic segmentation;

[0114] Step 4: Train the designed multispectral image fusion convolutional neural network, iterate and optimize the network parameters through the stochastic gradient descent method. During the training process, the sample set (x, y) is input into the network, where x is the input image and y is the corresponding label. The actual output is obtained by layer-by-layer operation of the network, and the cross-entropy loss function is used to measure the error between the ideal output y and the actual output p. The number of samples of each category in the data set is different, and the number of pixels occupied by each category is also different. In order to achieve better results on data sets with unbalanced sample numbers, the loss is assigned to different categories with different weights, and the weighted cross-entropy loss function is obtained:

[0115]

[0116] In formula (1), N represents the number of pixels in each image, and C represents the number of semantic categories in the sample. n,c is an indicator function, which takes the value 1 when the true category of sample n is equal to c, otherwise it takes the value 0. n,c represents the predicted probability that the observed sample n belongs to category c, w c represents the weight of class c, which is defined as:

[0117]

[0118] In formula (2), M is the total number of all pixels in the training set, y m,c is an indicator function, which takes the value of 1 when the true category of sample m is equal to c, otherwise it takes the value of 0. In the back-propagation stage, the network parameters are updated layer by layer from back to front through the stochastic gradient descent algorithm to minimize the error between the actual output and the ideal output. mIoU is used to evaluate the semantic segmentation performance, which is defined as follows:

[0119]

[0120] TP, FP, and FN represent the total number of true positives, false positives, and false negatives, respectively. Here, C represents the number of categories in the dataset.

[0121] After selecting the loss function, optimization method and performance evaluation index, the designed convolutional neural network is trained using the deep learning framework. In order to make the training results more accurate, pre-training is performed before formal training, and then the parameters obtained by pre-training are fine-tuned. The training process is as follows: Figure 3 As shown, it specifically includes the following sub-steps:

[0122] Sub-step 1: Select a multimodal dataset related to autonomous driving, such as ApolloScape and Cityscapes, and resize the sample to 480 × 640 pixels and record it as D c , then use D c Pre-train the designed convolutional neural network and set the pre-training hyperparameters respectively, where the maximum number of iterations is I c , the learning rates are α c , the weight decay is θ c , and finally save the pre-trained network parameters;

[0123] Sub-step 2: Use the dataset D created in step (1) k Fine-tune the network parameters obtained by pre-training in sub-step 1 of step (4), and set the maximum number of iterations to I. k , the learning rate is α k , the weight decays to θ k Then, according to the changes in the training loss curve, the verification loss curve and the verification accuracy mIou, the convolutional neural network model with the optimal network parameters is obtained.

[0124] Step 5: Use the multispectral image fusion convolutional neural network to perform unstructured scene segmentation. The acquired unstructured scene sample size is adjusted to 480×640 pixels and input into the trained multispectral image fusion convolutional neural network to obtain the semantic segmentation result of the forest unstructured scene.

Claims

1. A forest unstructured scene segmentation method based on multispectral image fusion includes the following steps: Step 1: Establish a multispectral segmentation dataset of forest unstructured scenes; Step 2: Construct a multi-spectral feature fusion encoding network. First, design two parallel encoder branches, which are used for feature extraction of RGB data and EVI data respectively. During the encoding process, the RGB branch will hierarchically fuse the hierarchical features of the EVI branch to obtain multimodal fusion features. Then, a skip connection structure is constructed to perform secondary fusion of the fusion features in the encoding stage with the decoding features, including the following sub-steps: Sub-step 1: Design the EVI branch coding structure, including: (1) Design and initialize the convolution layer. The number of initial convolution layers is 1, the number of convolution kernels is 16, the size of the convolution kernel is 7×7, and the step size is 2. (2) Design residual modules. The number of residual modules is 3. The network structures of residual modules 1 to 3 are consistent. The number of convolution kernels is 64, 128, and 256 respectively. The number of convolution kernels is represented by n. The residual module includes a standard convolution module, a separation residual module, and a dilated convolution module. The three modules are constructed as follows: ①Build a standard convolution module: Design standard convolution: The number of standard convolutions is 2, where the number of convolution kernels is n, the convolution kernel size is 3×3, and the step sizes are 2 and 1 respectively; Construct a-1: Convolve the input feature map with n 3×3 standard convolution kernels with a step size of 2, and then pass through the BN layer and ReLU activation. The output feature size is 1 / 2 of the input. Construct standard convolution module a-2: Use n 3×3 convolution kernels to convolve with the feature map of standard convolution module a with a step size of 1, and then pass through the BN layer and Re LU activation to obtain a feature map with unchanged dimension; Construct a standard convolution block: Cascade the standard convolution module a-1 and the standard convolution module a-2 to obtain a standard convolution block, and the output feature size is 1 / 2 of the input; ②Build a separation residual module Design separation convolution: the standard separation number is 4, where the number of convolution kernels is n, and the convolution kernel sizes are 3×1, 1×3, 3×1, 1×3, and the step size is 1; Construct separation residual a-1: Use n 3×1 convolution kernels to convolve with the output feature map of the standard convolution block with a step size of 1, and then pass through the BN layer and ReLU activation to obtain a feature map with unchanged dimension; Construct separation residual a-2: Use n 1×3 convolution kernels to convolve with the output feature map of separation residual a-1 with a step size of 1, and then pass through the BN layer and ReLU activation to obtain a feature map with unchanged dimension; Construct separation residual a-3: Use n 3×1 convolution kernels to convolve with the output feature map of separation residual a-2 with a step size of 1, and then pass through the BN layer and ReLU activation to obtain a feature map with unchanged dimension; Construct separation residual a-4: Use n 1×3 convolution kernels to convolve with the output feature map of separation residual a-3 with a step size of 1, and then pass through the BN layer and ReLU activation to obtain a feature map with unchanged dimension; Finally, the output feature map of the standard convolutional block is added to the output of the separation residual a-4 to obtain the output of the separation residual module; ③Construct dilated residual module Design dilated convolutional layers, the number of dilated convolutional layers is 2, the dilation rates are 3 and 5 respectively, the number of convolution kernels is n, the size of convolution kernels is 3×3, and the step size is 1; Construct the dilated residual module a-1: Use n 3×3 standard convolution kernels to convolve with the output feature map of the separation residual module with a step size of 1, and then pass through the BN layer and Re LU activation to obtain a feature map with unchanged dimension; Construct the dilated residual module a-2: Use n 3×3 convolution kernels to perform convolution with the feature map of the dilated residual module a-1 with a step size of 1, and then pass through the BN layer and Re LU activation to obtain a feature map with unchanged dimension; Finally, the output feature map of the separation residual module is added to the output of the dilated residual module a-2 to obtain the output of the dilated residual module; ④Construct residual module 1: Use a standard convolutional module with n=64 to convolve the output of the initialization convolutional layer of A / 2×A / 2 pixels to obtain a feature map with a dimension of A / 4×A / 4×64; Convolve the output of the standard convolutional module with A / 4×A / 4 pixels using the separation residual module with n=64 to obtain a feature map with a dimension of A / 4×A / 4×64; Convolve the output of the dilated residual module with n=64 pixels and the separated residual module with A / 4×A / 4 pixels to obtain a feature map with a dimension of A / 4×A / 4×64; ⑤Build residual module 2: Convolve the output of residual module 1 with A / 4×A / 4 pixels using a standard convolution module with n=128 to obtain a feature map with dimensions of A / 8×A / 8×128. Convolve the output of the standard convolutional module with A / 8×A / 8 pixels using the separation residual module with n=128 to obtain a feature map with a dimension of A / 8×A / 8×128; Convolve the output of the dilated residual module with n=128 pixels and the separated residual module with A / 8×A / 8 pixels to obtain a feature map with a dimension of A / 8×A / 8×128; ⑥Build residual module 3: Convolve the output of the residual module 2 with A / 8×A / 8 pixels using a standard convolution module with n=256 to obtain a feature map with dimensions of A / 16×A / 16×256; Convolve the output of the standard convolutional module with A / 16×A / 16 pixels using the separation residual module with n=256 to obtain a feature map with dimensions of A / 16×A / 16×256; Convolve the output of the dilated residual module with n=256 pixels and the separated residual module with A / 16×A / 16 pixels to obtain a feature map with the dimension of A / 16×A / 16×256. (3) Constructing EVI branch coding structure Convolve the initialization convolution layer with the EVI input sample of A×A pixels to obtain a feature map a1 with a dimension of A / 2×A / 2×16; the output of the initialization convolution layer passes through the residual module 1 to obtain a feature map a2 with a dimension of A / 4×A / 4×64; the output of the residual module 1 passes through the residual module 2 to obtain a feature map a3 with a dimension of A / 8×A / 8×128; the output of the residual module 2 passes through the residual module 3 to obtain a feature map a4 with a dimension of A / 16×A / 16×256; Sub-step 2: Design the RGB branch coding structure, including: (1) Design and initialize the convolutional layer, which is the same as step (1) of sub-step 1; (2) Design the residual module, which is the same as step (2) of sub-step 1; (3) Constructing RGB branch coding structure Convolve the initialized convolution layer with the RGB input sample of A×A pixels to obtain a feature map b1 with a dimension of A / 2×A / 2×16; add a1 and b1 obtained from the EVI branch and pass them through residual module 1 to obtain a feature map b2 with a dimension of A / 4×A / 4×64; add a2 and b2 obtained from the EVI branch and pass them through residual module 2 to obtain a feature map b3 with a dimension of A / 8×A / 8×128; add a3 and b3 obtained from the EVI branch and pass them through residual module 3 to obtain a feature map b4 with a dimension of A / 16×A / 16×256; Step 3: Construct a hierarchical feature fusion decoding structure: First, design an upsampling layer to restore the upsampling layer and dense prediction; then introduce the fusion features of RGB and EVI in the encoder stage and fuse them with the decoding features for the second time; including the following sub-steps: Sub-step 1: Design a standard convolution layer. The number of standard convolutions is 4, where the number of convolution kernels is 128, 64, 16, and class_num respectively, the size of the convolution kernel is 3×3, and the step size is 1; Sub-step 2: Construct upsampling layer 1, and perform bilinear interpolation on the encoder output feature b4 to obtain a feature map with a dimension of A / 8×A / 8×256; perform convolution on the standard convolution layer and the bilinear interpolation output of A / 8×A / 8 pixels to obtain a feature map with a dimension of A / 8×A / 8×128; Sub-step 3: Construct upsampling layer 2. The features output by upsampling layer 1 are first concatenated with b3 to obtain a feature map of A / 8×A / 8×256, and then bilinearly interpolated to obtain a feature map of dimension A / 4×A / 4×256. The standard convolutional layer is convolved with the bilinear interpolation output of A / 4×A / 4 pixels to obtain a feature map of dimension A / 4×A / 4×64. Sub-step 4: Construct upsampling layer 3. The features output by upsampling layer 2 are first concatenated with b2 to obtain a feature map of A / 4×A / 4×128, and then bilinearly interpolated to obtain a feature map of dimension A / 2×A / 2×128. The standard convolutional layer is convolved with the bilinear interpolation output of A / 2×A / 2 pixels to obtain a feature map of dimension A / 2×A / 2×16. Sub-step 5: Construct upsampling layer 4. The features output by upsampling layer 3 are first concatenated with b1 to obtain a feature map of A / 2×A / 2×32, and then bilinear interpolation is performed to obtain a feature map of dimension A×A×32. The standard convolution layer is convolved with the bilinear interpolation output of A×A pixels to obtain a feature map of dimension A×A×class_num. Step 4: Train the designed multispectral image fusion convolutional neural network to obtain network parameters; Step 5: Use multispectral image fusion convolutional neural network to perform forest unstructured scene segmentation.

Images