A field road segmentation method based on semantic segmentation network

Abstract

A field road segmentation method based on a semantic segmentation network includes the following steps: obtaining a field road image of a path to be planned; combining the detail information of the road edge in the field road image with the semantic information between the road and the crop through a multi-scale boundary feature fusion module to obtain a first feature map; processing the field road image through an ASPP module to obtain multi-scale features; performing space-divided and channel-divided parallel processing on the multi-scale features through a multi-receptive field attention module, and fusing the processing results with the multi-scale features to obtain a second feature map; after feature splicing the first feature map and the second feature map, a path image containing a road boundary curve is obtained. The multi-scale boundary feature fusion module and the multi-receptive field attention mechanism module are used to strengthen the road edge features and fine features. The two modules are processed in parallel in the encoder and fused in the decoder to obtain a field road feature map containing high-level and low-level features and more accurate.

Description

Technical Field

[0001] The invention belongs to the field of robot navigation path generation, and in particular relates to a field road segmentation method based on a semantic segmentation network. Background Art

[0002] In order to improve production efficiency and economic benefits, intelligent agricultural mechanization is an inevitable trend of future development. Field autonomous navigation technology is one of the key technologies of intelligent agricultural mechanization; in order to improve the quality and efficiency of field agricultural machinery and reduce the labor of agricultural machinery drivers, visual navigation technology has become one of the current mainstream technologies. The intelligent harvesting robot is a robot that performs autonomous navigation and harvesting in crop fields. In order to realize the autonomous harvesting function in the field, this type of robot must be able to realize autonomous navigation during the patrol process, and can complete the navigation task under the given patrol route through the visual navigation equipment it carries without external interference.

[0003] The intelligent harvesting robot is a robot that autonomously harvests crops in complex field environments. Due to the complex field environment, traditional GPS navigation methods are prone to damage to crops due to curved field roads and insufficient navigation accuracy. In addition, due to its own defects, the signal is blocked or even the satellite is lost, making it impossible for the robot to determine its own position and the exact field road. Summary of the invention

[0004] In view of the problems in the prior art, the present invention provides a field road segmentation method based on a semantic segmentation network, aiming to make up for the defects of traditional navigation and accurately identify the curves on both sides of the field road.

[0005] A field road segmentation method based on a semantic segmentation network comprises the following steps:

[0006] Step 1: Obtain the field road image of the path to be planned;

[0007] Step 2: The detailed information of the road edge in the field road image is combined with the semantic information between the road and the crop through a multi-scale boundary feature fusion module to obtain the first feature map;

[0008] Step 3: Use the ASPP module to extract multi-scale features from the field road image and supplement the context information of the image to obtain multi-scale features;

[0009] Step 4: Use the multi-receptive field attention module to perform parallel processing of multi-scale features in space and channels, and fuse the processing results with the multi-scale features to obtain the second feature map;

[0010] Step 5: Convolve the first feature map with a kernel size of The convolution operation is performed on the second feature map, and the convolution kernel size is After the convolution operation, a 4-fold upsampling operation is performed, and then the results of the two operations are feature concatenated and then upsampled 4 times to obtain a path image containing the road boundary curve.

[0011] Further: Step 2 is specifically:

[0012] Step 2.1: Downsample the field road image by 4 times to obtain the low-level features of the field road image;

[0013] Step 2.2: Perform multi-scale feature extraction on low-level features to obtain feature maps ;

[0014] Step 2.3: Feature map , After edge feature extraction block operation and introduction of loss function between feature map and label , get the feature map ;

[0015] Step 2.4: Feature map Perform feature fusion and obtain feature maps ;

[0016] Step 2.5: Feature map , After edge feature extraction block operation and introduction of loss function between feature map and label , get the feature map , which is the first feature map.

[0017] Further: Step 2.3 is specifically to convert the feature map , , The number of channels is adjusted to be consistent, and the feature map and feature map The size is adjusted to be consistent; the following steps are also included:

[0018] Step 2.5.1: Feature map and feature map After fusion, the fusion result is weighted to the region of interest through the sigmoid function to generate a boundary feature attention map , ;

[0019] Step 2.5.2: Boundary Feature Attention Map With the adjusted feature map , Perform feature fusion and introduce loss function , get the feature map ;

[0020] in, , is the true boundary of the road when training the network, is the prediction boundary of the network when training the network, Represents pixel points in semantic segmentation.

[0021] Further: Step 4 is specifically:

[0022] Step 4.1: Map the multi-scale features and obtain the specifications Map feature map, where , , , Represent the batch size, channel size, spatial height and width respectively;

[0023] Step 4.2: Use the size of , , , The convolution kernel performs two-dimensional convolution on the mapping feature map to obtain four tensors with different receptive fields , the sizes of these tensors are , and the tensor Feature fusion gets tensor ;

[0024] Step 4.3: Then divide each tensor into two groups of data, and connect the weights of each tensor in the channel dimension after weight extraction of the two groups of data;

[0025] Step 4.4: Perform feature fusion on the weights of all tensors and combine the fusion result with the multi-scale feature ( ) are fused and then the channels are shuffled to get the output , which is the second feature map.

[0026] Further: weight extraction includes the following steps:

[0027] Step 4.3.1: Record the input data as feature map , whose size is ;

[0028] Step 4.3.2: Global average pooling operation to obtain , ;

[0029] Step 4.3.3: Perform compression substitution to obtain , ;

[0030] Step 4.3.4: Exploitation Convolution kernel size The local cross-channel interaction is obtained , ;

[0031] Step 4.3.5: Use the sigmoid function to Process and obtain , ;

[0032] Step 4.3.6: Decompress and replace to get , ;

[0033] Step 4.3.7: The result of weight extraction.

[0034] Further: In step 4.3, the feature map include , ;

[0035] in, is the adaptive average pooling operator, yes Operator, yes Convolutional layers, is the decompression and exchange operator; the weight of each tensor , i is 1~5; .

[0036] Further: Road boundary curves in the path image for:

[0037]

[0038] in, It is a 4x upsampling operation; is the convolution operation; For feature stitching; is the first feature map, is the second feature map.

[0039] The beneficial effects of the present invention are as follows: a multi-scale boundary feature fusion module (MBF) and a multi-receptive field attention mechanism (MAM) module are used to enhance the road edge features and fine features. The two modules are processed in parallel in the encoder and fused in the decoder to obtain a more accurate field road feature map containing high- and low-level features. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a schematic diagram of the overall network model process of the present invention;

[0041] Figure 2 Schematic diagram of a multi-scale boundary feature fusion module in the present invention;

[0042] Figure 3 Schematic diagram of the multi-receptive field attention mechanism module in the present invention;

[0043] Figure 4 Schematic diagram of the weight extraction module in the present invention. DETAILED DESCRIPTION

[0044] The present invention is described in detail below in conjunction with the accompanying drawings. The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention. The directional terms such as left, middle, right, top, and bottom in the embodiments of the present invention are only relative concepts or are based on the normal use state of the product, and should not be considered as restrictive.

[0045] A field road segmentation method based on semantic segmentation network, such as Figure 1 As shown, the following steps are included:

[0046] Step 1: Obtain the field road image of the path to be planned; the camera used by the harvesting robot to take the field road image is higher than the crops on both sides of the field road, and can take field images near and far from the harvesting robot;

[0047] Step 2: Combine the detailed information of the road edge in the field road image with the semantic information between the road and the crop through the multi-scale boundary feature fusion module (MBF) to obtain the first feature map; Figure 2 As shown, specifically:

[0048] Step 2.1: Downsample the field road image by 4 times to obtain the low-level features of the field road image;

[0049] Step 2.2: Perform multi-scale feature extraction on low-level features to obtain feature maps ; Feature map Reflects the deep and shallow semantic information of field road images. , With more low-level detail information, Possess high-level semantic information;

[0050] Step 2.3: Feature map , After edge feature extraction block operation and introduction of loss function between feature map and label , get the feature map ; Specifically, for the feature map , , Perform the same number of convolution kernels Operation, the feature map , , The number of channels is adjusted to be consistent, and the feature map Perform a 2x bilinear upsampling UP operation to convert the feature map and feature map The size is adjusted to be consistent; the following steps are also included:

[0051] Step 2.3.1: Feature map and feature map After fusion, the fusion result is weighted to the region of interest through the sigmoid function to generate a boundary feature attention map , ;

[0052] Step 2.3.2: Boundary Feature Attention Map With the adjusted feature map , Perform feature fusion and introduce loss function , get the feature map ;

[0053] in, , is the true boundary of the road when training the network, is the prediction boundary of the network when training the network, Represents pixel points in semantic segmentation;

[0054] Step 2.4: Feature map Perform feature fusion and obtain feature maps ;

[0055] Step 2.5: Feature map , After edge feature extraction block operation and introduction of loss function between feature map and label , get the feature map , which is the first feature map.

[0056] Step 3: Use the ASPP module to extract multi-scale features from the field road image using convolution and use pooling to supplement the context information of the image to obtain multi-scale features;

[0057] Step 4: Use the multi-receptive field attention module (MAM) to perform parallel processing of multi-scale features in space and channels, and fuse the processing results with the multi-scale features to obtain the second feature map; combine Figure 3 As shown, specifically:

[0058] Step 4.1: Map the multi-scale features and obtain the specifications Map feature map, where , , , Represent the batch size, channel size, spatial height and width respectively;

[0059] Step 4.2: Use the size of , , , The convolution kernel performs two-dimensional convolution on the mapping feature map to obtain four tensors with different receptive fields , the sizes of these tensors are , and the tensor Feature fusion gets tensor ;

[0060] Step 4.3: Then divide each tensor into two groups of data, and connect the weights of each tensor in the channel dimension after weight extraction of the two groups of data; Figure 4 As shown, weight extraction includes the following steps:

[0061] Step 4.3.1: Record the input data as feature map , whose size is ;

[0062] Step 4.3.2: Global average pooling operation to obtain , ;

[0063] Step 4.3.3: Perform compression substitution to obtain , ;

[0064] Step 4.3.4: Exploitation Convolution kernel size The local cross-channel interaction is obtained , ;

[0065] Step 4.3.5: Use the sigmoid function to Process and obtain , ;

[0066] Step 4.3.6: Decompress and replace to get , ;

[0067] Step 4.3.7: is the result of weight extraction;

[0068] Step 4.4: Perform feature fusion on the weights of all tensors and combine the fusion result with the multi-scale feature ( ) are fused and then the channels are shuffled to get the output , i.e., the second feature map;

[0069] Among them, the feature map include , ;

[0070] in, is the adaptive average pooling operator, yes Operator, yes Convolutional layers, is the decompression and exchange operator; the weight of each tensor , i is 1~5; ;

[0071] Step 5: Convolve the first feature map with a kernel size of The convolution operation is performed on the second feature map, and the convolution kernel size is After the convolution operation and 4 times upsampling operation, the operation results of the two are then feature spliced ​​and then 4 times upsampling is performed to obtain a path image containing road boundary curves; the road boundary curves in the path image for:

[0072]

[0073] in, It is a 4x upsampling operation; is the convolution operation; For feature stitching; is the first feature map, is the second feature map.

[0074] In the processing of field images, field roads are easily affected by the growth of crops on both sides, resulting in interference and unclear road boundaries. Therefore, the present invention proposes a boundary feature extraction module to extract boundary features of the target area of ​​the image, enhance the ability to extract the edges of field roads in the image, and use the label edge detection results as loss supervision to iteratively update the attention feature map parameters and increase the pixel weights at the boundary positions. At the same time, the images taken by the harvesting robot have the situation that the near ones are large and the far ones are small, and features are lost. Therefore, in order to ensure the coherence of the near and far roads, the present invention proposes a multi-receptive field attention mechanism module, which is used to connect the contextual semantic relationship between the proximal roads, the distal roads and the crops, and process the field images with multi-scale receptive fields to obtain more refined segmentation area features. Therefore, the present invention adopts a multi-scale boundary feature fusion module (MBF) and a multi-receptive field attention mechanism (MAM) module to strengthen the road edge features and fine features. The two modules are processed in parallel in the encoder and fused in the decoder to obtain a field road feature map containing high-level and low-level features and more accurate. Finally, the segmented road image is fitted with a convex polygon fitting method for the area with large changes in the road edge, reducing the impact of road segmentation errors caused by uncertain factors, so that the navigation line can be extracted more accurately. The MBF module is used to strengthen the boundary features between crops and roads in field roads, and the detailed information of the road edge is combined with the semantic information between the road and the crops to obtain more accurate enhanced features, which improves the detection and segmentation capabilities of the neural network for fine edge features, so as to more accurately segment the field roads. The MAM module is used to perform parallel processing of the extracted feature maps in space and channels, and finally fuse them. The MAM module mainly targets the fuzzy boundaries between field roads and crops and the connection between proximal roads and distal roads. It can enhance the contextual semantic connection between the three and reduce segmentation errors caused by misidentification of neural networks. On the other hand, it also enhances the feature expression of the edges of roads and crops and the proximal and distal roads, and improves the segmentation accuracy of neural networks for roads. In addition, a loss function between the prediction results and labels is added. And the boundary loss function , optimized the training and improved the accuracy of field road segmentation.

[0075] The above shows and describes the basic principles, main features and advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A field road segmentation method based on a semantic segmentation network, characterized in that: The following steps are involved: Step 1: Obtain the field road image of the path to be planned; Step 2: Combine the detail information of the road edge in the field road image with the semantic information between the road and the crop through the multi-scale boundary feature fusion module to obtain the first feature map; specifically: Step 2.1: Downsample the field road image by 4 times to obtain the low-level features of the field road image; Step 2.2: Perform multi-scale feature extraction on low-level features to obtain feature maps F1, F2, F3, F4, F5; Step 2.3: The feature maps F1 and F2 are processed by edge feature extraction blocks and the loss function L between the feature maps and labels is introduced. B , and get the feature map F B1 ; Step 2.4: Perform feature fusion on feature map F5 to obtain feature map F 125 ; Step 2.5: Transform the feature map F B1 , F 125 After edge feature extraction block operation and introduction of loss function L between feature map and label B , and get the feature map F output , which is the first feature map; Step 3: Use the ASPP module to extract multi-scale features from the field road image and supplement the context information of the image to obtain multi-scale features; Step 4: Use the multi-receptive field attention module to perform parallel processing of multi-scale features in space and channels, and fuse the processing results with the multi-scale features to obtain the second feature map; specifically: Step 4.1: Map the multi-scale features and obtain a mapping feature map with a specification of [B, C, H, W], where B, C, H, and W represent the batch size, channel size, spatial height, and width, respectively; Step 4.2: Perform two-dimensional convolution on the mapping feature map with convolution kernels of sizes 1×1, 3×3, 5×5, and 7×7, respectively, to obtain four tensors [X1, X2, X3, X4] with different receptive fields. The sizes of these tensors are all [B, C, H, W], and the features of the tensors [X1, X2, X3, X4] are fused to obtain the tensor X5; Step 4.3: Then divide each tensor into two groups of data, and connect the weights of each tensor in the channel dimension after weight extraction of the two groups of data; Step 4.4: Perform feature fusion on the weights of all tensors, fuse the fusion result with the multi-scale features, and then perform channel shuffle to obtain the output X output , i.e., the second feature map; Step 5: Perform a convolution operation with a convolution kernel size of 1×1 on the first feature map, and perform a convolution operation with a convolution kernel size of 1×1 on the second feature map, and then perform a 4-fold upsampling operation. Then, after feature splicing, the operation results of the two are upsampled by 4 times to obtain a path image containing the road boundary curve.

2. The field road segmentation method based on semantic segmentation network according to claim 1, characterized in that: Step 2.3 specifically includes adjusting the number of channels of the feature maps F1, F2, and F5 to be consistent, and adjusting the size of the feature map F2 and the feature map F1 to be consistent; and further includes the following steps: Step 2.5.1: After fusing feature map F2 and feature map F1, use the sigmoid function to assign weights to the region of interest to generate a boundary feature attention map. Step 2.5.2: The boundary feature attention map E1 is fused with the adjusted feature maps F1 and F2, and the loss function L is introduced B , get the feature map F B1 ; in, is the true boundary of the road when training the network, P i b is the predicted boundary of the network when training the network, and i represents the pixel point in semantic segmentation.

3. The field road segmentation method based on semantic segmentation network according to claim 1, characterized in that: Weight extraction includes the following steps: Step 4.3.1: Record the input data as feature map X CH , whose size is [B,C,H,W]; Step 4.3.2: Global average pooling operation to obtain X a , X a ∈R (B,C,1,1) ; Step 4.3.3: X a Perform compression substitution to obtain X s , X s ∈R (B,1,C) ; Step 4.3.4: Use a k×k convolution kernel to calibrate X s The local cross-channel interaction is carried out to obtain X c , X c ∈R (B,1,C) ; Step 4.3.5: Use the sigmoid function to transform X c Process and get X sg , X sg ∈R (B,1,C) ; Step 4.3.6: X sg Decompress and replace to get F emk×k , F emkxk ∈R (B,C,1,1) ; Step 4.3.7: F emk×k The result of weight extraction.

4. The field road segmentation method based on semantic segmentation network according to claim 3 is characterized in that: In step 4.3, the feature map X CH Include X 51 ,X 52 , Femk × k(X CH )=FunFsgW1dFsFaX CH ; Among them, F a is the adaptive average pooling operator, F sg is the Sigmoid operator, W 1d is a k×k convolutional layer, F un is the decompression and exchange operator; the weight W of each tensor i , i is 1 to 5; W i =concat(F emk×k (X 51 ),F emk×k (X 52 )).

5. The field road segmentation method based on semantic segmentation network according to claim 1, characterized in that: The road boundary curve P in the path image is: P=Up4(Conv3(Concat(Conv(F output )Conv(Up4×X output )))) Among them, Up4 is a 4-fold upsampling operation; Conv is a convolution operation; Concat is feature concatenation; X output is the first feature map, F output is the second feature map.

Images