A method for calculating vehicle deflection angle on off-road vehicles based on semantic segmentation

Abstract

The present invention discloses a method for calculating the deflection angle of a vehicle traveling on a wild road based on semantic segmentation. The present invention first collects pictures of wild roads in different seasons and under different light intensities, and labels the road areas and non-road areas. Secondly, a semantic segmentation network model consisting of an information sharing layer, a two-way semantic reasoning layer, a detail guidance module and an aggregation layer is constructed and trained. The camera distortion is then corrected, and the corrected picture is input into the trained model to obtain a wild road segmentation result image. Finally, the center line of the wild road in the segmentation result is extracted, and the deflection angle is calculated. The present invention enables the network to further improve the segmentation accuracy of wild roads while ensuring real-time performance, and solves the problems of practical application availability of the overall network algorithm and poor network calculation results due to distortion of the input image.

Description

Technical Field

[0001] The present invention belongs to the field of methods for calculating the driving deflection angle of unmanned vehicles on off-road roads, and in particular to a method for calculating the driving deflection angle of vehicles on off-road roads based on semantic segmentation. Background Art

[0002] To improve vehicle safety and driving experience, AI-based autonomous driving technology has begun to assist and gradually replace manual driving. As the core of autonomous driving technology, road detection technology performs well on urban roads, which are mainly structured roads, but its robustness in the wild still needs to be improved. As an extreme case of unstructured roads, wild roads have complex road surface features, blurred road boundaries, and background interference features similar to the road surface. Model- and feature-based road recognition methods have problems such as weak generalization and poor robustness when identifying unstructured roads. While the current mainstream semantic segmentation network performs well in urban road recognition, its performance in the wild is suboptimal and its real-time performance is poor. Summary of the Invention

[0003] In response to the above-mentioned problems and technical requirements, the present invention proposes a method for calculating the deflection angle of vehicles traveling on off-road roads based on semantic segmentation. The present invention adopts an information sharing layer to promote the information exchange of feature parameters of the dual-stream network (dual-path semantic reasoning layer) in the semantic segmentation network model at a shallow level; utilizes a multi-information splicing module to alleviate the problem of shallow information being lost in the deep layer of the network; constructs a dual-path semantic reasoning layer to enhance the network's contextual information extraction capability; adds a detail guidance module to promote the network's capture of road detail features; and finally uses camera distortion correction algorithms and perspective transformation methods to realize the conversion of input images and the deflection angle required for the car to travel along the road on an unmanned vehicle. This solves the problem of weak generalization ability of traditional model- and feature-based road recognition methods, and compared with mainstream semantic segmentation networks, it is more adaptable to accurate road recognition in off-road environments and has a faster detection speed.

[0004] The technical solution of the present invention comprises the following steps:

[0005] Step 1: Dataset Collection and Image Annotation. Collect images of outdoor roads in different seasons and lighting intensities, and manually annotate the road and non-road areas in all images. Then, divide the images into training, validation, and test sets.

[0006] Step 2: Build a semantic segmentation network model. This network model primarily consists of an information sharing layer, a two-way semantic reasoning layer, a detail guidance module, and an aggregation layer. Unless otherwise specified, standard convolutional layers use a 3×3 kernel size. The standard convolutional layer consists of a convolutional layer, a batch normalization layer, and a ReLU activation function.

[0007] (1) The information sharing layer has three layers in total. The first layer consists of two standard convolutional layers: the first standard convolutional layer has 3 input channels, 32 output channels, and a stride of 2; the second standard convolutional layer has 32 input channels, 32 output channels, and a stride of 1. The second layer of the information sharing layer consists of three standard convolutional layers: the first standard convolutional layer has 32 input channels, 64 output channels, and a stride of 2; the second standard convolutional layer has 64 input channels, 64 output channels, and a stride of 1; the structure of the third standard convolutional layer is the same as the second layer. The third layer of the information sharing layer also consists of three standard convolutional layers: the first standard convolutional layer has 64 input channels, 128 output channels, and a stride of 2; the second standard convolutional layer has 128 input channels, 128 output channels, and a stride of 1; the structure of the third standard convolutional layer is the same as the second.

[0008] (2) The dual-path semantic reasoning layer is divided into two parts: the spatial branch and the semantic branch. The spatial branch does not process the input feature map and directly outputs it. The semantic branch mainly consists of 8 layers: the first layer is a standard convolution layer with a convolution kernel size of 1×1, 128 input channels, 32 output channels, and a stride of 1; the second layer consists of a multi-information splicing module with 32 input channels, 64 output channels, and downsampling; the third layer is an attention optimization module with 64 input and output channels; the fourth layer is a multi-information splicing module with 64 input channels, 128 output channels, and downsampling; the fifth layer is a multi-information splicing module with 128 input and output channels and no downsampling; the network structure of the sixth layer is the same as that of the fifth layer; the seventh layer is an attention optimization module with 128 input and output channels; the eighth layer is a context embedding module with 128 input and output channels. Finally, the output feature maps of the spatial branch and the semantic branch of the information sharing layer are fused and upsampled through the aggregation layer.

[0009] The multi-information splicing module, a major component of the dual-path semantic reasoning layer, is described in detail as follows:

[0010] The multi-information stitching module first uses a 3×3 convolutional layer with a stride of 2 to quickly downsample the feature map. It then consists of four main paths: Path1, Path2, Path3, and Path4. From left to right, the semantic information contained in each path becomes increasingly advanced. Path4 is the backbone path, and the inputs of the other three paths all come from Path4 at different depths. This is done to obtain multi-scale receptive fields and multi-depth semantic information on the module's backbone path. The channel configuration on the backbone path of the multi-information stitching module follows a strategy of gradually decreasing from wide to narrow. Assuming the output channels of the multi-information stitching module are M and the input channels are C, the channel number of Path4 decreases from M / 2 to M / 8. Path4 consists of five layers: the first is a standard convolutional layer with C input channels, M / 2 output channels, and a stride of 1 or 2. The output of this layer serves as the input to Path1. The second layer consists of a 1×1 convolutional layer and a batch normalization layer. The input channels are M / 2, M / 4, and the stride is 1. The output of this layer serves as the input to Path2. The third layer uses a ReLU activation function. The fourth layer is a standard convolutional layer with M / 4 input channels, M / 8 output channels, and a stride of 1. The output of this layer serves as the input to Path3. The fifth layer also consists of a standard convolutional layer with M / 8 input channels, M / 8 output channels, and a stride of 1. The input feature maps of Path1, Path2, and Path3 all come from layers of Path4 at different depths, and their output channels are M / 2, M / 4, and M / 8, respectively. Path 1 consists of a convolutional layer with a 1×1 kernel size, M / 2 input channels, M / 2 output channels, a stride of 1, and a batch normalization layer. Path 2 performs no processing and directly outputs the result. Path 3 consists of a convolutional layer with a 1×1 kernel size, M / 8 input channels, M / 8 output channels, a stride of 1, and a batch normalization layer. All paths are aggregated through a simple and time-saving splicing operation into a feature map with M channels that incorporates multi-scale receptive fields and multi-level semantic information. This feature map, after passing through the ReLU activation function, becomes the output of the multi-information splicing module. It is worth mentioning that to alleviate the information incompatibility problem caused by directly splicing feature information of different depths within the module, all paths in the multi-information splicing module, except for Path 4, end with a 1×1 convolution operation.

[0011] The detailed description of the attention optimization module (ARM) of the above two-way semantic reasoning layer is as follows:

[0012] The input of this module undergoes a global maximum pooling, a standard convolution layer with a convolution kernel size of 1×1, and a Sigmoid activation function, and then is multiplied with the original input of the module directly in the channel domain before being output to the next module.

[0013] The context embedding module (CE) of the two-way semantic reasoning layer is described as follows:

[0014] The input of this module passes through a global average pooling, a batch normalization, and a standard convolution layer with a convolution kernel size of 1×1, and is summed with the original module input. Finally, it is directly output after a convolution with a convolution kernel size of 3×3.

[0015] (3) The left branch of the aggregation layer inputs the detail branch of the two-way semantic reasoning layer, and the right branch inputs the semantic branch of the two-way semantic reasoning layer. The left branch obtains two different outputs through two different branches: Branch 1 first performs a depthwise separable convolution (DWConv) with a stride of 1 and batch normalization on the input, and then uses a 1×1 convolution kernel to obtain output 1; Branch 2 first performs a depthwise separable convolution with a stride of 2 and batch normalization on the input, and then uses global average pooling (APooling) with a convolution kernel size of 3 to obtain output 2. The right branch of the aggregation layer also obtains two different outputs through two different branches: Branch 3 first uses a convolution kernel size of 3 and batch normalization, and then uses upsampling (Up) and a Sigmoid activation function to obtain output 3; Branch 4 first uses a depthwise separable convolution of size 3×3 and batch normalization, and then uses a convolution kernel size of 1 (Down) and a Sigmoid activation function to obtain output 4. Output 1 is multiplied by output 3, and output 2 is multiplied by output 4. The sum of the two multiplication results is then subjected to a 3×3 ordinary convolution and a batch normalization to obtain the final output of the aggregation layer.

[0016] (4) The detail guidance module first uses the Laplacian Conv operator to convolve the label image with a step size of 1 to extract the binary detail feature map of the road. The Laplacian convolution kernel used in the present invention can be described by the following formula:

[0017]

[0018] The obtained detail feature map is then dilated using a max pooling layer with a convolution kernel size of 5. This further fills the number of pixels near the road edge to enrich the edge information. Finally, all pixels are converted to a binary image containing only 0s and 1s using a threshold of 0.1. This image guides the information sharing layer to learn road edge detail information. It's worth noting that the detail guidance module consumes computational resources during training, but not during inference.

[0019] Step 3: Train the semantic segmentation model. Preprocess the image. After reading the sample image, first randomly adjust its saturation, brightness, and contrast. After color dithering, randomly flip the image horizontally and randomly scale it. Finally, randomly crop the scaled image. The preprocessed image is fed into the semantic segmentation network model for training. During training, the segmentation loss function (Seg Loss) uses a binary cross entropy loss function.

[0020] Step 4: Perform camera distortion correction on the camera to address the barrel distortion problem in the output image. The images input to the network are all taken by cameras. Due to the problems in the camera lens manufacturing process, the images have barrel distortion, which needs to be corrected.

[0021] Step 5: Convert the trained pt-format model from Step 3 into a .onnx model using the Open Neural Network Exchange (ONNE) tool. Then, use the model parsing tool to convert the .onnx model into a ready-to-use .trt model. Convert the rectified image from Step 4 between BGR and RGB, normalize it, and then perform dimension expansion. Finally, input it into the .trt model to obtain the resulting field road segmentation image.

[0022] Step 6: Extract the center line of the wild road in the segmentation result image, and select a suitable pixel point on the center line as the target point (x, y) for the car to move forward. The method for extracting the center line of the road is to calculate the midpoint of the longest area of ​​pixels belonging to the wild road in each row of the image segmentation result, and connect the midpoints of all rows of the pixel image to form the center line of the road. Then, through perspective transformation, the pixel coordinates of the target point in the image under the horizontal perspective are converted into coordinate points with higher precision under the top-down perspective (x r ,y r ). Its coordinate transformation formula is:

[0023]

[0024] Among them, the perspective transformation matrix H is:

[0025]

[0026] The H parameter can be obtained by first actually measuring the pixel coordinate values ​​of the target point in the image under the horizontal viewing angle and its corresponding coordinate values ​​under the overhead viewing angle, a total of four sets, and substituting them into the above conversion formula to obtain it.

[0027] The deflection angle required by the car is the angle α between the line connecting the target point and the origin and the positive direction of the camera optical axis. Its mathematical calculation formula is:

[0028]

[0029] Among them, α is the vehicle's steering angle.

[0030] The beneficial technical effects of the present invention are:

[0031] The present invention solves the problem of insufficient publicly available field road datasets by independently producing a field road semantic segmentation dataset. A field road semantic segmentation network is independently constructed, and by optimizing the network structure and rationally adjusting the network detail parameters, the network further improves the field road segmentation accuracy while ensuring real-time performance. At the same time, the network segmentation results are converted into directional angles that can actually guide the car along the centerline of the road to solve the practical application of the overall network algorithm. Furthermore, image distortion correction is performed on the camera output image to solve the problem of poor network calculation results due to distortion in the input image. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is the overall program operation flow chart of the present invention;

[0033] Figure 2 It is the overall semantic segmentation network model diagram of the present invention;

[0034] Figure 3 This is a network structure diagram of the multi-information splicing module of the present invention;

[0035] Figure 4 is the attention optimization module of the present invention;

[0036] Figure 5 is the context embedding module of the present invention;

[0037] Figure 6 is a polymeric layer of the present invention;

[0038] Figure 7 The image obtained by the camera before correction has obvious barrel distortion.

[0039] Figure 8 It is a display in which all distortions are corrected after distortion correction;

[0040] Figure 9 is the perspective transformation template image;

[0041] Figure 10 It is the effect of converting the flat view image into a top view using perspective transformation;

[0042] Figure 11 It is a visual display of the intermediate results in actual operation of the present invention;

[0043] Figure 12 These are the test results of the present invention and the BiSeNet series of algorithms on some field road images;

[0044] Figure 13 These are the test results of STDC and Fast-SCNN on the wild road dataset. DETAILED DESCRIPTION

[0045] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0046] like Figure 1 As shown in the figure, process 1 is responsible for storing images captured by the camera into the image storage queue, while process 2 reads images from the queue and uses inference calculation to calculate the required front wheel deflection angle for the robot. Specifically, process 1 continuously stores images captured by the camera at the end of the image storage queue. The queue can store a maximum of three images; images older than three are discarded. Process 2 first reads the latest image from the end of the storage queue and clears the queue to ensure that the images in the queue are up to date. Next, distortion correction is performed on the acquired image and the corrected image is fed into the TensorRT-accelerated model for road segmentation. The centerline of the road is then extracted from the segmentation result, and a suitable pixel on the centerline is selected as the target point for the robot to move forward. A perspective transformation is then used to convert the pixel coordinates of the target point in the image into highly accurate coordinates from a bird's-eye view. Finally, a simple angle conversion is performed on the coordinates to obtain the required deflection angle for the robot. The specific design steps are as follows:

[0047] Step 1: Dataset Collection and Image Annotation. Wild road images were collected in different seasons and at varying light intensities. These images were manually labeled according to road and non-road categories. The dataset was then divided into training, validation, and test sets in a 7:1.5:1.5 ratio.

[0048] Step 2: Build a semantic segmentation network model. Figure 2 As shown in the figure, the network model mainly consists of an information sharing layer, a two-way semantic reasoning layer, a detail guidance module, and an aggregation layer. Unless otherwise specified, the convolutional layers used have a kernel size of 3×3. The standard convolutional layer consists of a convolutional layer, a batch normalization layer, and a ReLU activation function.

[0049] (1) The information sharing layer has three layers in total. The first layer consists of two standard convolutional layers: the first standard convolutional layer has 3 input channels, 32 output channels, and a stride of 2; the second standard convolutional layer has 32 input channels, 32 output channels, and a stride of 1. The second layer of the information sharing layer consists of three standard convolutional layers: the first standard convolutional layer has 32 input channels, 64 output channels, and a stride of 2; the second standard convolutional layer has 64 input channels, 64 output channels, and a stride of 1; the structure of the third standard convolutional layer is the same as that of the second layer. The third layer of the information sharing layer also consists of three standard convolutional layers: the first standard convolutional layer has 64 input channels, 128 output channels, and a stride of 2; the second standard convolutional layer has 128 input channels, 128 output channels, and a stride of 1; the structure of the third standard convolutional layer is the same as that of the second layer.

[0050] (2) The dual-path semantic reasoning layer is divided into two parts: the spatial branch and the semantic branch. The spatial branch does not process the input feature map and directly outputs it. The semantic branch mainly consists of 8 layers: the first layer is a standard convolution layer with a convolution kernel size of 1×1, 128 input channels, 32 output channels, and a stride of 1; the second layer consists of a multi-information splicing module with 32 input channels, 64 output channels, and downsampling; the third layer is an attention optimization module with 64 input and output channels; the fourth layer is a multi-information splicing module with 64 input channels, 128 output channels, and downsampling; the fifth layer is a multi-information splicing module with 128 input and output channels and no downsampling; the network structure of the sixth layer is the same as that of the fifth layer; the seventh layer is an attention optimization module with 128 input and output channels; the eighth layer is a context embedding module with 128 input and output channels. Finally, the output feature maps of the spatial branch and the semantic branch of the information sharing layer are fused and upsampled through the aggregation layer.

[0051] The multi-information splicing module, a major component of the dual-path semantic reasoning layer, is described in detail as follows:

[0052] The structure of the multi-information splicing module is as follows Figure 3As shown in the figure, the module first uses a 3×3 convolution with a stride of 2 to quickly downsample the feature map. The module consists of four main paths: Path1, Path2, Path3, and Path4. From left to right, the semantic information contained in each path becomes increasingly advanced. Path4 is the backbone path, and the inputs of the other three paths all come from Path4 at different depths. This is done to obtain multi-scale receptive fields and multi-depth semantic information on the module's backbone path. The channel configuration on the backbone path of the multi-information splicing module follows a strategy of gradually decreasing from wide to narrow. Assume that the multi-information splicing module has M output channels, C input channels, and the height and width of the input feature map are H and W. The number of channels in Path4 decreases gradually from M / 2 to M / 8. Path4 consists of five layers: the first is a standard convolutional layer with C input channels, M / 2 output channels, and a stride of 1 or 2. The output of this layer serves as the input to Path1. The second layer consists of a convolutional layer (Conv) with a kernel size of 1×1 and a batch normalization layer (BN). The input channels are M / 2, M / 4, and the output channels are M / 4 with a stride of 1. The output of this layer serves as the input to Path2. The third layer uses a ReLU activation function. The fourth layer is a standard convolutional layer with M / 4 input channels, M / 8 output channels, and a stride of 1. The output of this layer serves as the input to Path3. The fifth layer also consists of a standard convolutional layer with M / 8 input channels, M / 8 output channels, and a stride of 1. The input feature maps of Path1, Path2, and Path3 all come from layers of Path4 at different depths, with output channels of M / 2, M / 4, and M / 8, respectively. Path1 consists of a convolution layer with a convolution kernel size of 1×1, M / 2 input channels, M / 2 output channels, a stride of 1, and a batch normalization. Path2 does not perform any processing and outputs directly. Path3 consists of a convolution layer with a convolution kernel size of 1×1, M / 8 input channels, M / 8 output channels, a stride of 1, and a batch normalization. All paths are concatenated (Concatenated) into a feature map with M channels and a fusion of multi-scale receptive fields and multi-level semantic information through a simple and time-saving splicing operation. After passing through the ReLU activation function, this feature map becomes the output of the multi-information splicing module. It is worth mentioning that in order to alleviate the problem of information incompatibility caused by directly splicing feature information of different depths within the module, all paths of the multi-information splicing module except Path4 end with a 1×1 convolution (Conv) operation. The output of the multi-information splicing module can be described by the following formula:

[0053]

[0054] Where, X outrepresents the output of the multi-information splicing module, M represents the number of channels, F represents the fusion operation, and P1, P2, P3, and P4 are the outputs of each path.

[0055] The specific description of the attention optimization module ARM of the above two-way semantic reasoning layer is as follows:

[0056] like Figure 4 As shown in the figure, the input of this module passes through a global maximum pooling, a standard convolution layer with a convolution kernel size of 1×1, and a Sigmoid activation function, and is directly multiplied with the original input of the module in the channel domain before being output to the next module.

[0057] The context embedding module CE of the two-way semantic reasoning layer is described as follows:

[0058] like Figure 5 As shown in the figure, the input of this module passes through a global average pooling, a batch normalization, and a standard convolution layer with a convolution kernel size of 1×1, and is summed with the original module input. Finally, it is directly output after a normal convolution with a convolution kernel size of 3×3.

[0059] (3) Figure 6 As shown in the figure, the left branch of the aggregation layer inputs the detail branch of the two-way semantic reasoning layer, and the right branch inputs the semantic branch of the two-way semantic reasoning layer. The left branch obtains two different outputs through two different branches: Branch 1 first performs a depthwise separable convolution (DWConv) with a stride of 1, batch normalization, and then a 1×1 convolution kernel to obtain output 1; Branch 2 first performs a depthwise separable convolution with a stride of 2, batch normalization, and then global average pooling (APooling) with a kernel size of 3 to obtain output 2. The right branch of the aggregation layer also obtains two different outputs through two different branches: Branch 3 first performs a convolution with a kernel size of 3, batch normalization, upsampling (Up), and a sigmoid activation function to obtain output 3; Branch 4 first performs a depthwise separable convolution of 3×3, batch normalization, and then a normal convolution with a kernel size of 1 (Up) and a sigmoid activation function to obtain output 4. Output 1 is multiplied by output 3, and output 2 is multiplied by output 4. The sum of the two multiplication results is then subjected to a 3×3 ordinary convolution and a batch normalization to obtain the final output of the aggregation layer.

[0060] (3) The detail guidance module first uses the Laplacian operator (Laplacian Conv) to convolve the label image with a step size of 1 to extract the binary detail feature map of the road. The Laplacian convolution kernel used in this paper can be described by the following formula:

[0061]

[0062] Subsequently, a maximum pooling layer with a convolution kernel size of 5 is used to dilate the obtained detail feature map, which can further fill the number of pixels on the road edge to enrich the edge information. Finally, all the pixels are converted into binary images with only the two numbers 0 and 1 using a threshold of 0.1, and this is used to guide the information sharing layer to learn the road edge detail information. It is worth mentioning that the detail guidance module consumes computing resources during training, but not in the inference stage. The detail guidance module loss function (Detial Loss) uses a combination (addition) of the binary cross entropy loss function and the Dice loss function to optimize detail training. The segmentation head (Seg Head) mainly consists of a 3×3 convolution layer, a batch normalization layer, a ReLU activation function, and a 1×1 convolution layer.

[0063] Step 3: Train the semantic segmentation model. After reading the sample images, first randomly adjust their saturation, brightness, and contrast. After color dithering, the images are randomly horizontally flipped and randomly scaled using the following scaling ratios: 0.5, 0.75, 1.0, 1.25, 1.5, and 1.75. Finally, the scaled images are randomly cropped to a size of 480×640. During training, 32 sample images are extracted at a time and batch normalized. The iteration cycle is set to 40K. The segmentation loss function (Seg Loss) uses the binary cross entropy loss function.

[0064] Step 4: First, perform camera distortion correction on the camera to address the barrel distortion problem in the output image. The images input to the network are all taken by cameras. Due to the manufacturing process of the camera lens, the images have barrel distortion, which needs to be corrected.

[0065] Step 5: Convert the trained pt-format model from Step 3 into a .onnx model using Nvidia's Open Neural Network Exchange tool. Then, use the model parsing tool provided by TensorRT to convert the .onnx model into a .trt model that can be directly used by TensorRT. Convert the rectified image obtained in Step 4 between BGR and RGB, normalize it, and finally perform dimension expansion. Finally, input it into the .trt model to obtain the resulting field road segmentation image.

[0066] Step 6: Extract the center line of the field road in the segmentation result and select a suitable pixel point on the center line as the target point (x, y) for the car to move forward. Then, through perspective transformation, the pixel coordinates of the target point in the image are converted into a coordinate point with higher accuracy in a bird's-eye view (x r ,y r ). Its coordinate transformation formula is:

[0067]

[0068] Among them, the perspective transformation matrix H is:

[0069]

[0070] The H parameter can be obtained by first actually measuring the pixel coordinate values ​​of the target point in the image under the horizontal viewing angle and its corresponding coordinate values ​​under the overhead viewing angle, a total of four sets, and substituting them into the above conversion formula to obtain it.

[0071] The deflection angle required by the car is the angle α between the line connecting the target point and the origin and the positive direction of the camera optical axis. Its mathematical calculation formula is:

[0072]

[0073] In order to further demonstrate the feasibility and effectiveness of the present invention, the actual effects of the method used in the present invention will be shown in the form of charts below.

[0074] Camera distortion correction is to alleviate the problem of barrel distortion in the output image caused by camera production process problems. This article applies Zhang Zhengyou's distortion correction method to field road detection. Since the objects on the field road are irregular in shape, it is difficult to see the effect of camera distortion correction. Therefore, the camera distortion correction effect is demonstrated in an indoor environment (because it is common sense that indoor objects such as beams are straight, so it is easy to see the barrel distortion problem of the camera and the actual effect after correction). Figure 7 As shown in the figure, before correction, the top beam and the edge of the fence in the lower part of the image obtained by the camera have obvious barrel distortion. Figure 8 As shown, all of the above distortions have been corrected.

[0075] In order to better demonstrate the effect of perspective transformation, a perspective transformation template image is selected, such as Figure 9 The effect of using perspective transformation to convert the flat view image into a top view is shown as follows. Figure 10 As shown in the figure, it can be seen Figure 9 The part in the middle frame is converted into a top view through perspective transformation. Figure 10 , Figure 9The coordinates of the four corner points on the framed part are (215,399), (386,399), (563,462), (55,462) in the pixel coordinate system, sorted by upper left, upper right, lower right, and lower left. After perspective transformation, these four points will be mapped to (0,0), (300,0), (300,900), and (0,900) respectively. Substituting these four groups of points into the perspective transformation formula, the perspective transformation matrix H can be obtained. In actual operation, the patent code of this invention only converts the coordinates of the pixel points on the center line of the road under a head-on perspective into the coordinates under a top-down perspective. This is to alleviate the overhead of computing resources, like Figures 9 to 10 The conversion requires converting the coordinates of all image pixels.

[0076] Table 1 shows the deflection angle α of the car obtained after using perspective transformation and the deflection β calculated directly using pixel points without using perspective transformation, as well as the actual deflection angle required for the car to drive to the target point (x, y). In the table, (x, y) are the coordinate values ​​before perspective transformation, (x r ,y r ) is the coordinate value after perspective transformation. As can be seen from the table, the deflection angle α obtained after perspective transformation is closer to the actual deflection angle than the deflection angle β calculated directly using pixel coordinates.

[0077] Table 1

[0078]

[0079]

[0080] Figure 11 The visualization of the intermediate results of the code of the present invention in actual operation is demonstrated. From left to right, the three columns are respectively the effect of distortion correction of the camera image, the inference result of the field semantic segmentation network after TensorRT acceleration, and the effect of extracting the centerline of the field road in the inference result.

[0081] The code of this invention runs on the Nvidia Jetson Xavier NX embedded development board. Figure 12 The code of the present invention has an FPS of 23 on this embedded platform, and TensorRT actually only takes 0.0152 seconds to predict each image, meeting the real-time requirements.

[0082] To better demonstrate the superiority of our semantic segmentation network algorithm over other network algorithms in identifying road surfaces, we compared it with BiSeNetV1, BiSeNetV2, Fast-SCNN, and the latest STDC real-time semantic segmentation network on the same server. The results are shown in Table 2. As can be seen, our semantic segmentation network improved Intersection over Union (IoU) accuracy by 0.9% on the validation set and 1.2% on the test set compared to BiSeNetV2. The algorithm also reduced the number of parameters by 0.6M. In terms of inference speed, our algorithm increased the number of frames per second by 262, a 59.2% improvement. Compared with Fast-SCNN, BiSeNetV1, and STDC, the IoU accuracy was 2.3%, 0.7%, and 0.7% higher on the validation set, and 1.5%, 0.8%, and 1% higher on the test set, respectively. Although Fast-SCNN has the lowest number of parameters, this low number of parameters also resulted in the worst performance of all the networks.

[0083] Table 2

[0084]

[0085] Figure 12 The test results of the semantic segmentation algorithm of the present invention and the BiSeNet series algorithm on some wild road pictures are shown. The four columns from left to right are the input picture, the image obtained by the present invention, the image obtained by BiSeNetV2, and the image obtained by BiSeNetV1. For the convenience of comparison, the detailed differences in the segmentation results of different algorithms are marked with boxes and dotted lines. By comparing the framed areas of the first row of pictures, the semantic segmentation algorithm of the present invention can effectively segment the road blocked by tree branches in the lower left corner. BiSeNetV1 recognizes the part of the road surface not blocked by tree branches, while BiSeNetV1 fails to effectively recognize it. In the second row of pictures, through the auxiliary comparison of dotted lines, the algorithm of the present invention and BiSeNetV1 can recognize the narrower road area in the distance from the camera's perspective, while BiSeNetV2's recognition ability is relatively limited. The third row of pictures shows that the algorithm of the present invention has a stronger edge fitting ability for distant bends. BiSeNetV1 has a good effect, while BiSeNetV2 has the worst. In the red framed part of the last row of pictures, the top of the road segmented by the algorithm of the present invention is flatter, which is closer to the shape of the road in the distance, and the effect of BiSeNetV2 is slightly inferior. However, BiSeNetV1 has some defects and does not fit the distant road edges well. In general, the semantic segmentation network of the present invention has the best effect on segmenting road details, followed by BiSeNetV1, and BiSeNetV2 is the worst.

[0086] Figure 13Use with Figure 12 The same input picture shows the test results of STDC and Fast-SCNN on the wild road dataset. The four columns from left to right are the input picture, the image obtained by MICNet, the image obtained by STDC, and the image obtained by Fast-SCNN. Through the comparison of the first row of pictures, it also shows the advantages of the algorithm of the present invention in identifying partially obscured roads and narrow roads in the distance. The segmentation effect of STDC is suboptimal, and Fast-SCNN is the worst. At the same time, there is also a problem of being unable to identify narrow roads in the distance. In the second row of pictures, the algorithm of the present invention can identify roads farther away than STDC and Fast-SCNN. In the third row of pictures, Fast-SCNN, which has the least number of parameters, is completely unable to fit the edges of distant roads. At the same time, the algorithm of the present invention still shows the strongest fitting ability for the edges of distant roads in the fourth row of pictures. Figure 13 Compared with the results of the previous studies, the proposed algorithm still has the best segmentation effect on wild roads, followed by STDC, and Fast-SCNN is the worst.

[0087] The above shows and describes the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions merely illustrate the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the present invention. The scope of protection claimed by the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for calculating the deflection angle of a vehicle traveling on an off-road road based on semantic segmentation, characterized in that: The steps include: Step 1: Collect field road images in different seasons and light intensities, and manually label the road and non-road areas in all images; Divide the training set, validation set and test set into proportions; Step 2: Build a semantic segmentation network model; The semantic segmentation network model mainly consists of an information sharing layer, a two-way semantic reasoning layer, a detail guidance module, and an aggregation layer; The information sharing layer consists of three layers in total. The first layer consists of two standard convolutional layers, the second layer consists of three standard convolutional layers, and the third layer also consists of three standard convolutional layers. The two-way semantic reasoning layer is divided into two parts: spatial branch and semantic branch; The left branch of the aggregation layer inputs the spatial branch of the two-way semantic reasoning layer, and the right branch of the aggregation layer inputs the semantic branch of the two-way semantic reasoning layer; Step 3: Train the semantic segmentation network model; Preprocess the image: After reading the sample image, first randomly adjust its saturation, brightness, and contrast, then randomly flip the image horizontally and randomly scale it, and finally randomly crop the scaled image. The preprocessed images are input into the semantic segmentation network model for training. During the training process, the segmentation loss function uses the binary cross entropy loss function; Step 4: Perform camera distortion correction on the camera to solve the barrel distortion problem in the output image; Step 5: Convert the pt format model trained in step 3 into a .onnx model using the Open Neural Network Exchange tool, and then use the model parsing tool to convert the .onnx model into a .trt model that can be directly used; Convert the corrected image obtained in step 4 between BGR and RGB, normalize it, expand its dimension, and finally input it into the .trt model to obtain the field road segmentation result map; Step 6: Extract the center line of the field road in the field road segmentation result image, and select a suitable pixel point on the center line as the target point (x, y) for the car to move forward. Then, through perspective transformation, convert the pixel coordinates of the target point in the image under the horizontal perspective into a coordinate point with higher precision under the top perspective (x r ,y r ); The mathematical formula for the angle α between the line connecting the target point and the origin and the positive direction of the camera optical axis is: Among them, α is the vehicle's steering angle.

2. The method for calculating the deflection angle of a vehicle traveling on an off-road road based on semantic segmentation according to claim 1, characterized in that: In step 2, the information sharing layer, two-way semantic reasoning layer, detail guidance module, and aggregation layer are specifically as follows: The spatial branch of the two-way semantic reasoning layer does not process the input feature map and outputs it directly; The first layer of the semantic branch of the two-way semantic reasoning layer is a standard convolutional layer. The second layer consists of a multi-information splicing module that performs downsampling. The third layer consists of an attention optimization module. The fourth layer consists of a multi-information splicing module that performs downsampling. The fifth layer consists of a multi-information splicing module that does not perform downsampling. The network structure of the sixth layer is the same as that of the fifth layer. The seventh layer consists of an attention optimization module, and the eighth layer consists of a contextual embedding module; The left branch of the aggregation layer obtains two different outputs through two different branches: Branch 1 performs depth-wise separable convolution and batch normalization on the input, and then uses a 1×1 convolution kernel to obtain output 1; Branch 2 performs depth-wise separable convolution and batch normalization on the input, and then uses global average pooling with a convolution kernel size of 3 to obtain output 2; The right branch of the aggregation layer also obtains two different outputs through two different branches: branch 3 uses a convolution kernel with a convolution kernel size of 3 and batch normalization, and then uses upsampling and a Sigmoid activation function to obtain output 3; branch 4 uses a depth-wise separable convolution of size 3×3 and a batch normalization, and then uses a convolution kernel size of 1 and a Sigmoid activation function to obtain output 4; Output 1 is multiplied by output 3, and output 2 is multiplied by output 4. The sum of the two multiplication results is then subjected to a normal convolution and a batch normalization to obtain the final output of the aggregation layer. The detail guidance module first uses the Laplacian operator to convolve the label image with a step size of 1 to extract the binary detail feature map of the road; Secondly, a maximum pooling layer with a convolution kernel size of 5 is used to dilate the obtained detail feature map. Finally, a threshold of 0.1 is used to convert all pixel points into a binary image with only the numbers 0 and 1, which guides the information sharing layer to learn the road edge detail information.

3. The method for calculating the deflection angle of a vehicle traveling on an off-road road based on semantic segmentation according to claim 2, characterized in that: The multi-information splicing module is specifically described as follows: The multi-information stitching module first uses a convolutional layer to quickly downsample the feature map. It then consists of four paths: Path 1, Path 2, Path 3, and Path 4. From left to right, the semantic information contained in each path becomes increasingly advanced. Path 4 is the backbone path, and the inputs of the other three paths all come from Path 4 at different depths. Assume that the number of output channels of the multi-information splicing module is M, the number of input channels is C, and the number of channels of the Path4 path decreases gradually from M / 2 to M / 8. Path4 consists of 5 layers in total: the first layer is a standard convolutional layer with C input channels and M / 2 output channels. The output of this layer is the input of Path1; the second layer consists of a convolutional layer and a batch normalization layer with M / 2 input channels and M / 4 output channels. The output of this layer is the input of Path2; the third layer is a ReLU activation function; the fourth layer is a standard convolutional layer with M / 4 input channels and M / 8 output channels. The output of this layer is the input of Path3; the fifth layer also consists of a standard convolutional layer with M / 8 input channels and M / 8 output channels. Path 1 consists of a convolutional layer with M / 2 input channels and M / 2 output channels, followed by a batch normalization. Path 2 does not perform any processing and directly outputs the results. Path 3 consists of a convolutional layer with M / 8 input channels and M / 8 output channels, followed by a batch normalization. After the splicing operation, all paths are aggregated into a feature map with M channels that integrates multi-scale receptive fields and multi-level semantic information. After the ReLU activation function, this feature map becomes the output of the multi-information splicing module; The specific description of the attention optimization module of the two-way semantic reasoning layer is as follows: The input of this module undergoes a global maximum pooling, a standard convolution layer, and a Sigmoid activation function, and then is multiplied with the original input of the module directly in the channel domain and output to the next module; The context embedding module of the two-way semantic reasoning layer is described in detail as follows: The input of this module passes through a global average pooling, a batch normalization, and a standard convolution layer, and is summed with the original module input, and finally directly output after a convolution.

4. The method for calculating the deflection angle of a vehicle traveling on an off-road road based on semantic segmentation according to claim 3, characterized in that: The detail guidance module loss function optimizes the detail guidance module training by combining the binary cross entropy loss function and the Dice loss function.

5. The method for calculating the deflection angle of a vehicle traveling on an off-road road based on semantic segmentation according to claim 3, characterized in that: All paths except Path 4 of the multi-information splicing module end with a 1×1 convolution operation.

6. The method for calculating the deflection angle of a vehicle traveling on an off-road road based on semantic segmentation according to claim 5, characterized in that: In step 2, the standard convolutional layer consists of a convolutional layer, a batch normalization layer, and a ReLU activation function.

7. The method for calculating the deflection angle of a vehicle traveling on an off-road road based on semantic segmentation according to claim 1, characterized in that: In step 6, the method for extracting the road centerline is: calculating the midpoint of the longest area of ​​pixels belonging to the wild road in each row of the image segmentation result, and connecting the midpoints of all rows of the pixel image is the road centerline.

8. The method for calculating the deflection angle of a vehicle traveling on an off-road road based on semantic segmentation according to claim 1, characterized in that: In step 6, the coordinate conversion formula of the perspective transformation is: Among them, the target point of the car is (x, y), and the coordinate point after conversion is (x r ,y r ), H is the perspective transformation matrix. Its parameters are obtained by first measuring the pixel coordinate values ​​of the target point in the image under the horizontal perspective and its corresponding coordinate values ​​under the top-down perspective, a total of four groups, and substituting them into the above coordinate transformation formula to obtain them.

Images