A method for detecting the smoothness of a road ahead

Abstract

The present invention discloses a method for detecting the smoothness of a road ahead, comprising: using a monocular camera to collect video stream data of the road ahead of a vehicle and extracting frames of the video stream data; constructing a deep residual network, and using the deep residual network to perform depth estimation operations on the extracted frame images to obtain a depth map; performing 3D reconstruction on the depth map to obtain a 3D cloud map of the road ahead of the vehicle; based on the 3D cloud map, calculating the distances between the vehicle and other vehicles, obstacles, road markings, and signs captured in the frame images of the road ahead of the vehicle, and converting the pixel coordinate system into a world coordinate system in combination with the depth map; detecting whether the road ahead of the vehicle is flat in the world coordinate system by using a hypothesis test method; and if the road ahead of the vehicle is not flat, further detecting whether the road ahead of the vehicle is a slope by using a hypothesis test method. The method of the present invention can detect whether the road ahead of the vehicle is flat or sloped while the vehicle is driving, providing convenience and safety for the vehicle owner.

Description

Technical Field

[0001] The invention relates to a method for detecting the flatness of a front road surface, and belongs to the technical field of road surface flatness detection. Background Art

[0002] In recent years, artificial intelligence has been widely used. Deep learning models based on neural networks, combined with autonomous driving, are highly effective in acquiring relevant road information. The acquisition and analysis of road information is crucial for autonomous driving. Due to the enormous potential of neural networks, researchers are constantly exploring and optimizing how to accurately acquire road information using monocular depth estimation.

[0003] Currently, the method for detecting the road surface ahead in autonomous driving generally uses lidar and millimeter-wave radar. First, the vehicle road target image and point cloud data are obtained, and the road target image and point cloud data are preprocessed. The target category label information and 2D bounding box information are obtained through the target detection network. The road target image and point cloud data are fused and processed to obtain a point cloud with image label information in the 2D bounding box. The target point cloud is obtained through target detection, and the distance to the target point cloud center of mass is calculated to obtain the current target information.

[0004] However, there's no specific method for specifically identifying the smoothness and slope of the road surface in autonomous driving. Many patents and papers use expensive, computationally intensive LiDAR (LiDAR) to analyze and predict road conditions ahead. Some papers also incorporate LiDAR and camera fusion, but these fusion methods are computationally intensive and time-consuming to detect road conditions, making them inconvenient. Furthermore, using cloud platforms can lead to significant real-time latency. Due to technical limitations, monocular depth estimation is currently rarely used to extract road surface information ahead. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a method for detecting the flatness of the road ahead, which uses a probability statistics method to detect the flatness information of the road ahead while the vehicle is driving, thereby providing convenience and safety for the vehicle owner.

[0006] The present invention adopts the following technical solutions to solve the above technical problems:

[0007] A method for detecting the flatness of a road ahead comprises the following steps:

[0008] Step 1: using a monocular camera to collect video stream data of the road ahead of the vehicle, and performing frame extraction processing on the video stream data of the road ahead of the vehicle;

[0009] Step 2: construct a deep residual network, and use the deep residual network to perform depth estimation operation on the frame image extracted in step 1 to obtain a depth map;

[0010] Step 3: Perform 3D reconstruction on the depth map obtained in step 2 to obtain a 3D cloud map of the road ahead of the vehicle;

[0011] Step 4: Calculate the distances between the vehicle and other vehicles, obstacles, road markings, and signs captured in the road frame image in front of the vehicle based on the 3D cloud image, and convert the pixel coordinate system into the world coordinate system in combination with the depth map;

[0012] Step 5: In the world coordinate system, a hypothesis-checking method is used to detect whether the road ahead of the vehicle is a plane. If the road ahead of the vehicle is not a plane, a hypothesis-checking method is further used to detect whether the road ahead of the vehicle is a slope.

[0013] As a preferred solution of the present invention, in step 1, the frame extraction interval of the video stream data of the road ahead of the vehicle is the sum of the depth estimation operation time in step 2 and the 3D reconstruction time in step 3.

[0014] As a preferred solution of the present invention, the frame extraction interval of the video stream data of the road ahead of the vehicle is 10ms.

[0015] As a preferred embodiment of the present invention, the specific process of step 2 is as follows:

[0016] Step 2.1: Use the Laplacian pyramid structure to build a deep residual network and use the pyramid decomposition to define the deep residual;

[0017] In step 2.2, the activation function of the deep residual network is the GeLU activation function, which is defined as:

[0018]

[0019] Where x represents the input of the activation function;

[0020] Step 2.3: Calculate the conversion relationship between the pixels in the frame image and the actual distance to obtain the depth map. The calculation formula is as follows:

[0021]

[0022] Among them, (x c ,y c , z c ) is the pixel point P in the depth map obtained by depth estimation operation c Coordinates, (x, y, z) are coordinates in the world coordinate system, k and l are camera internal parameters, f is camera external parameter, c x is the offset of pixel coordinate system coordinate x, c y is the offset of the pixel coordinate system y.

[0023] As a preferred solution of the present invention, in step 4, the formula for converting the pixel coordinate system into the world coordinate system is as follows:

[0024]

[0025] Among them, (x, y, z) is the coordinate information of the pixel point P in the world coordinate system, k and l are the camera internal parameters, f is the camera external parameter, c x is the offset of the pixel coordinate system coordinate x, (x c ,y c , z c ) is the pixel point P in the depth map obtained by depth estimation operation c coordinate.

[0026] As a preferred embodiment of the present invention, the specific process of step 5 is as follows:

[0027] Step 5.1, select any point in the 3D cloud image (X i , Y i , Z i ), on the XOZ plane, with point (X i , Y i , Z i ) is the center of the circle, d standard Draw a circle with a radius of 1, and sample 8 points (X i+n , Y i , Z i+n ), n∈(1, 2, 3, 4, 5, 6, 7, 8), the angle between two adjacent sampling points and the center of the circle is 45°;

[0028] X i+n 、Z i+n It is obtained by the following formula:

[0029]

[0030] d standard =0.03m

[0031] (X i , Y i , Z i ) is the coordinate in the world coordinate system, X i+n 、Z i+n are the values ​​of the sampling points on the X-axis and Z-axis respectively;

[0032] On the 3D cloud map, find (X i+n , Z i+n ) and remap it into a three-dimensional coordinate system to obtain its value Y on the Y axis i+n ;

[0033] Calculate the sample mean:

[0034]

[0035] Calculate the sample variance:

[0036]

[0037] The road plane is regarded as a single normal population Y~N(μ, σ 2 ), (μ, σ 2 ) obeys the normal distribution, μ, σ 2 are the population mean and population variance respectively. When the confidence level α = 0.05, the test hypothesis H0: σ 2 =0, H1:σ 2 ≠0; the confidence interval is χ 0.025 (7), χ 0.975 (7) is the chi-square distribution;

[0038] Assume that the road ahead of the vehicle is flat, that is, the overall variance is 0. If 0 is within the above confidence interval, the assumption is established, that is, (X i , Y i , Z i ) as the center, d standard The area with radius is a plane and the height is If 0 is not within the above confidence interval, the hypothesis is not established, that is, (X i , Y i , Z i ) as the center, d standard The area with radius is not a plane;

[0039] Step 5.2: If the road ahead of the vehicle is not flat, select any point (X5, Y5, Z5) in the 3D cloud map, draw a square with a side length of D = 0.3m on the XOZ plane with point (X5, Y5, Z5) as the center, and sample the center, four vertices, and center points of the four sides of the square. j , Y5, Z j ) are numbered from left to right and from top to bottom, j = 1, ..., 9;

[0040] On the 3D cloud map, find (X j , Z j ) and remap it into a three-dimensional coordinate system to obtain its value Y on the Y axis j ;

[0041] For sampling points 1, 4, and 7 with the same value on the X axis, calculate the angle formed by sampling points 1 and 4 The angle formed by sampling points 1 and 7 The angle formed by sampling points 4 and 7

[0042] For sampling points 2, 5, and 8 with the same value on the X axis, calculate the angle formed by sampling points 2 and 5 The angle formed by sampling points 2 and 8 The angle formed by sampling points 5 and 8

[0043] For sampling points 3, 6, and 9 with the same value on the X axis, calculate the angle formed by sampling points 3 and 6 The angle formed by sampling points 3 and 9 The angle formed by sampling points 6 and 9

[0044] Set the angle α 14 , α 17 , α 47 , α 25 , α 28 , α 58 , α 36 , α 39 , α 69 Renumbered α1 to α9;

[0045] Calculate the sample mean:

[0046]

[0047] Calculate the sample variance:

[0048]

[0049] The road slope is regarded as a single normal population Y~N(μ, σ 2 ), (μ, σ 2 ) obeys the normal distribution, μ, σ 2 are the population mean and population variance respectively. When the confidence level α = 0.05, the test hypothesis H0: σ 2 =0, H1:σ 2 ≠0; the confidence interval is χ 0.025 (8), χ 0.975 (8) is the chi-square distribution;

[0050] Assume that the road ahead of the vehicle is a slope with a slope of That is, the overall variance is 0. If 0 is within the above confidence interval, the hypothesis is established, that is, the slope of the square XOZ surface with (X5, Y5, Z5) as the center and a side length of D = 0.3m is If 0 is not within the above confidence interval, the assumption is not true, that is, the square XOZ surface with (X5, Y5, Z5) as the center and a side length of D = 0.3m is irregular and has no slope.

[0051] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects:

[0052] 1. The present invention adopts a monocular camera, which has higher timeliness and can reflect road information in a shorter time. The monocular camera is easy to install and has low cost.

[0053] 2. The present invention uses a deep residual network to perform depth estimation and three-dimensional image reconstruction to obtain real-time information of the road.

[0054] 3. The present invention uses a probability statistics method to detect the road surface flatness, which is simple and easy to implement. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 This is a workflow diagram of a method for detecting the flatness of a front road surface according to the present invention;

[0056] Figure 2 It is a structural diagram of the deep residual network used in the present invention;

[0057] Figure 3 This is a rendering of the depth estimation algorithm used in the present invention;

[0058] Figure 4 It is a sampling example in the slope detection of the present invention. DETAILED DESCRIPTION

[0059] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be interpreted as limiting the present invention.

[0060] Acquiring depth images of road conditions is a key technology for autonomous driving at this stage. Using a monocular camera to obtain road depth images through depth estimation for road prediction is a key project for future research. This paper uses a deep residual network based on a Laplacian pyramid structure to perform depth estimation to estimate and judge road conditions. Figure 1 It is a flow chart of the present invention, combined with Figure 1 The main steps of the present invention are as follows:

[0061] Step 1: Use a camera to collect road video stream data information and extract frames from the video stream data. The camera uses a monocular camera. Extracting frames is to make the processing smoother and better use tracking algorithms to calculate the path. According to the operation time t involved in taking a frame of image, n It is determined by the depth estimation operation time and 3D reconstruction time in the subsequent steps. In the present invention, the sampling time t n The value is about 10ms.

[0062] Step 2: Use depth residual technology to perform depth estimation on the extracted frame image. The depth estimation process has the following steps:

[0063] (1) Constructing a convolutional neural network using the Laplacian pyramid structure

[0064] The Laplacian pyramid structure is used in the decoder, and the depth residual is defined by the decomposition of the pyramid. The output of the convolutional network is gradually integrated from a small coarse-scale depth map to a final comprehensive and detailed depth map, which can accurately estimate the depth boundaries of targets in the road situation and understand the overall situation. Figure 2 This is the structure diagram of the deep residual network.

[0065] (2) Modification of activation function

[0066] The activation function is used to transform the nonlinear combination and the hidden layer into a linear one. In the present invention, the activation function is the GeLU activation function, which is defined as:

[0067]

[0068] Like the original ReLU activation function, GeLU has unilateral inhibition and is equivalent to ReLU when the variance is infinite and the mean is 0. However, ReLU can only input 0 and positive numbers. GeLU introduces the idea of ​​random regularization in activation, which is a probabilistic description of neuron input. It is more intuitive and natural, and the experimental results are better than ReLU and ELU.

[0069] (3) Calculate the conversion relationship between the camera and the actual distance

[0070] Due to the different cameras used, the distance that can be reflected by depth estimation is also different. The formula for the actual distance corresponding to the pixels in the photo is:

[0071]

[0072] Among them, (x c ,y c , z c ) is the pixel point P in the depth map obtained by depth estimation operation c Coordinates, (x, y, z) are coordinates in the world coordinate system, k and l are camera internal parameters, which are the actual distance (meters) converted to pixels; f is the camera external parameter, which is the actual focal length of the camera; c x is the offset of pixel coordinate system coordinate x, c y is the offset of the pixel coordinate system y. Figure 3 This is the effect diagram of the depth estimation operation.

[0073] Step 3: Reconstruct the estimated image into a 3D cloud map of the complete road conditions.

[0074] The specific steps are to iteratively match and integrate the depth images taken in front to form a 3D reconstructed road information summary centered on the vehicle itself, making analysis more convenient.

[0075] Step 4: Calculate the distance between the vehicle and other vehicles, roadside obstacles, road markings, and signs captured in the 3D cloud image. Based on the resulting depth map, convert the pixel coordinate system into the world coordinate system.

[0076]

[0077] In (x, y, z), z represents the distance information in the depth map, x represents the width, and y represents the height. (x, y, z) represents the coordinate information in the world coordinate system. Based on this coordinate information, the distance between the camera and the preceding vehicle and obstacle can be determined.

[0078] Step 5: In the horizontal road detection, first select a point in the 3D cloud map (X i , Y i , Z i ), d standard A value representing how many m are sampled:

[0079] d standard =0.03m

[0080]

[0081] In the XOZ plane, (X i , Y i , Z i ) as the center and sample 8 points at 45 degrees to find the corresponding (X i+n , Y i , Z i+n ), find (X i+n , Z i+n ) remapped to a three-dimensional coordinate system, and obtained the inner Y i+n , where n∈(1, 2, 3, 4, 5, 6, 7, 8).

[0082] Calculate the sample mean:

[0083]

[0084] Calculate the sample variance:

[0085]

[0086] By default, the road plane is used as a single normal population Y~N(μ,σ 2 ), (μ, σ 2) is unknown and obeys normal distribution, μ, σ 2 are the population mean and population variance respectively. When the confidence level α = 0.05, the test hypothesis H0: σ 2 =0, H1:σ 2 ≠0;

[0087] The confidence interval is χ 0.025 (7), χ 0.975 (7) is the chi-square distribution;

[0088] When 0 is within this confidence interval, it is considered that H0: σ 2 =0 is established, it can be considered that (X i , Y i , Z i ) as the center and radius d standard The area is a plane, and the height is If 0 is not in this confidence interval, it can be rejected and it can be considered that (X i , Y i , Z i ) as the center and radius d standard The region is not a plane.

[0089] In the prediction of the road slope detection ahead, first select any point (X5, Y5, Z5) in the 3D cloud map, and set up a square sampling with a side length of D = 0.3m in the XOZ plane with (X5, Y5, Z5) as the geometric center, as shown in the following example: Figure 4 shown.

[0090] On the cloud map, find (X i , Z i ) is remapped into a three-dimensional coordinate system, and the corresponding point i obtains the inner Y i , when the x values ​​are equal, that is, 1, 4, and 7, get the angle value

[0091] Similarly, we can obtain α 25 , α 28 , α 58 , α 36 , α 39 , α 69 .

[0092] Renumber all nine angles from α1 to α9.

[0093] Calculate the sample mean:

[0094]

[0095] Calculate the sample variance:

[0096]

[0097] The default road slope is a single normal population Y~N(μ,σ 2 ), (μ, σ 2 ) is unknown and obeys normal distribution, μ, σ 2 are the population mean and population variance respectively. When the confidence level α = 0.05, the test hypothesis H0: σ 2 =0, H1:σ 2 ≠0;

[0098] The confidence interval is χ 0.025 (8), χ 0.975 (8) is the chi-square distribution;

[0099] When 0 is within this confidence interval, it is considered that H0: σ 2 = 0, it can be considered that a square XOZ plane with a side length of D = 0.3m is established with point 5 (X5, Y5, Z5) as the geometric center, and the slope is When 0 is not in this confidence interval, it can be rejected. A square XOZ with a side length of D = 0.3m is established with point 5 (X5, Y5, Z5) as the geometric center. The plane is irregular and has no slope.

[0100] The above embodiments are only for illustrating the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the present invention.

Claims

1. A method for detecting the flatness of a road ahead, characterized in that: The steps include: Step 1: using a monocular camera to collect video stream data of the road ahead of the vehicle, and performing frame extraction processing on the video stream data of the road ahead of the vehicle; Step 2: construct a deep residual network, and use the deep residual network to perform depth estimation operation on the frame image extracted in step 1 to obtain a depth map; Step 3: Perform 3D reconstruction on the depth map obtained in step 2 to obtain a 3D cloud map of the road ahead of the vehicle; Step 4: Calculate the distances between the vehicle and other vehicles, obstacles, road markings, and signs captured in the road frame image in front of the vehicle based on the 3D cloud image, and convert the pixel coordinate system into the world coordinate system in combination with the depth map; Step 5: Detecting whether the road ahead of the vehicle is a plane by using a hypothesis testing method in the world coordinate system. If the road ahead of the vehicle is not a plane, further detecting whether the road ahead of the vehicle is a slope by using a hypothesis testing method. The specific process of detecting whether the road ahead of the vehicle is flat is as follows: Select any point in the 3D cloud image , on the XOZ plane with point is the center of the circle, Draw a circle with a radius of , and sample 8 points on the circle drawn above , n , the angle between two adjacent sampling points and the center of the circle is 45°; 、 It is obtained by the following formula: ， ， is the coordinate in the world coordinate system, 、 are the values ​​of the sampling points on the X-axis and Z-axis respectively; On the 3D cloud map, find ( ) and remap it into a three-dimensional coordinate system to obtain its value on the Y axis ; Calculate the sample mean: ， Calculate the sample variance: ， The road plane is regarded as a single normal population Y , Obey the normal distribution, are the population mean and population variance respectively, when the confidence level When testing the hypothesis : , : ; where the confidence interval is , 、 is the chi-square distribution; Assume that the road ahead of the vehicle is flat, that is, the overall variance is 0. If 0 is within the above confidence interval, the assumption is established, that is, Centered on The area with radius is a plane and the height is ; If 0 is not within the above confidence interval, the hypothesis is not established, that is, Centered on The area with radius is not a plane.

2. A method for detecting the flatness of a road ahead according to claim 1, characterized in that: In step 1, the frame extraction interval of the video stream data of the road ahead of the vehicle is the sum of the depth estimation operation time in step 2 and the 3D reconstruction time in step 3.

3. The method for detecting the flatness of the road ahead according to claim 2, wherein: The frame extraction interval of the video stream data of the road ahead of the vehicle is 10ms.

4. The method for detecting the flatness of a road ahead according to claim 1, wherein: The specific process of step 2 is as follows: Step 2.1: Use the Laplacian pyramid structure to build a deep residual network and use the pyramid decomposition to define the deep residual; In step 2.2, the activation function of the deep residual network is the GeLU activation function, which is defined as: ， Where, Represents the input of the activation function; Step 2.3: Calculate the conversion relationship between the pixels in the frame image and the actual distance to obtain the depth map. The calculation formula is as follows: ， in, Pixel points in the depth map obtained by depth estimation operation coordinate, is the coordinate in the world coordinate system, All are camera internal parameters. is the camera extrinsic parameter, is the pixel coordinate system coordinate The bias, is the pixel coordinate system coordinate Bias.

5. The method for detecting the flatness of a road ahead according to claim 1, wherein: In step 4, the formula for converting the pixel coordinate system to the world coordinate system is as follows: ， in, is the coordinate information of pixel point P in the world coordinate system, All are camera internal parameters. is the camera extrinsic parameter, is the pixel coordinate system coordinate The bias, Pixel points in the depth map obtained by depth estimation operation coordinate.

6. The method for detecting the flatness of a road ahead according to claim 1, wherein: In step 5, whether the road ahead of the vehicle is a slope is detected. The specific process is as follows: If the road ahead of the vehicle is not flat, select any point in the 3D cloud map. , on the XOZ plane with point As the center, draw a line with a side length of The center, four vertices and the center points of the four sides of the square are sampled. Number them in order from left to right and from top to bottom. ; On the 3D cloud map, find ( ) and remap it into a three-dimensional coordinate system to obtain its value on the Y axis ; For sampling points 1, 4, and 7 with the same value on the X axis, calculate the angle formed by sampling points 1 and 4 , the angle formed by sampling points 1 and 7 , the angle formed by sampling points 4 and 7 ; For sampling points 2, 5, and 8 with the same value on the X axis, calculate the angle formed by sampling points 2 and 5 , the angle formed by sampling points 2 and 8 , the angle formed by sampling points 5 and 8 ; For sampling points 3, 6, and 9 with the same value on the X axis, calculate the angle formed by sampling points 3 and 6 , the angle formed by sampling points 3 and 9 , the angle formed by sampling points 6 and 9 ; The angle 、 、 、 、 、 、 、 、 Renumbered to arrive ; Calculate the sample mean: ， Calculate the sample variance: ， Take the road slope as a single normal population Y , Obey the normal distribution, are the population mean and population variance respectively, when the confidence level When testing the hypothesis : , : ; where the confidence interval is , 、 is the chi-square distribution; Assume that the road ahead of the vehicle is a slope with a slope of , that is, the population variance is 0. If 0 is within the above confidence interval, the hypothesis is established, that is, The slope of the square XOZ surface with a center of D=0.3m is ; If 0 is not within the above confidence interval, the hypothesis is not established, that is, The square XOZ surface with a center of D and a side length of D=0.3m is irregular and has no slope.

Images