Road model extension method and device based on multi-sensor fusion, medium and computer program product

Abstract

The invention discloses a road model extension method and device based on multi-sensor fusion, a medium and a computer program product. The road model extension method comprises the following steps: acquiring a center line equation of a visual lane; judging whether the preceding vehicle is an effective preceding vehicle or not, and if yes, detecting whether the distance between the preceding vehicle and the own vehicle is larger than the sight distance of the lane line or not; if the distance between the front vehicle and the own vehicle is greater than the sight distance of the lane line, extending the center line of the visual lane to an extension point, and comparing the transverse distance difference between the extension point and the front vehicle; if the transverse distance difference is smaller than a set threshold value, an extension line of the visual lane center line extending from the sight distance point to the extension point serves as an extended lane center line, and if the transverse distance difference is larger than or equal to the set threshold value, the front vehicle position serves as an extended point, and the sight distance point and the extended point serve as two end points of the extended lane center line; and obtaining an equation of the extended lane center line. According to the method, the beyond-visual-range road model can be obtained, and the implementation cost is low.

Description

Technical Field

[0001] The present invention relates to the technical field of autonomous driving. Background Art

[0002] At present, vehicles equipped with autonomous driving functions obtain real-time road models through the road lane equations sent by the on-board camera. Due to the detection performance of the camera, the longitudinal view distance of the obtained road model is limited.

[0003] When the vehicle turns on the automatic driving function, it is necessary to control the vehicle to stay in the middle of the lane and maintain the set cruising speed according to the road markings and surrounding vehicles. At the same time, in order to ensure the timeliness and comfort of vehicle control, it is necessary to preview the road structure at a certain distance ahead of the road to perform feedforward control of the vehicle speed and steering. At present, most vehicles equipped with automatic driving functions obtain real-time road models through the road lane line equations emitted by the on-board forward-looking camera. For roads with large changes in curvature of curves, such as straight roads into curves, curves into straight roads, sharp turns, etc., the longitudinal sight distance of the obtained road model is limited due to the detection performance of the camera sensor and the possible obstruction of surrounding vehicles. There may be deviations between the judgment of the road structure near the preview point and the actual situation, which will cause the automatic driving vehicle to adjust the speed in time, inaccurate steering control, or even run out of the lane.

[0004] To cope with this scenario, high-level autonomous driving functions (such as navigation and driving assistance) use high-precision maps and positioning modules to predict the upcoming curves in advance, and adjust the vehicle's actuators accordingly to slow down and adjust the steering range in advance. However, high-precision maps are expensive and not all autonomous driving vehicles are equipped with them. In addition, high-precision maps also have the disadvantages of long update cycles and limited coverage. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a road model expansion method based on multi-sensor fusion, which can obtain a road model beyond visual range and is easy to implement with low implementation cost.

[0006] According to a first aspect of the present invention, a road model expansion method based on multi-sensor fusion is provided, comprising the following steps:

[0007] Get the centerline equation of the visual lane;

[0008] Determine whether the preceding vehicle is a valid preceding vehicle. If so, check whether the distance between the preceding vehicle and the vehicle is greater than the lane line sight distance.

[0009] If the distance between the front vehicle and the vehicle is greater than the lane line sight distance, extend the center line of the visual lane to the extension point, and compare the lateral distance difference between the extension point and the front vehicle position; the x coordinate of the extension point is equal to the x coordinate of the front vehicle position, and the lateral distance difference is the absolute value of the difference between the y coordinate of the extension point and the y coordinate of the front vehicle position;

[0010] If the lateral distance difference is less than the set threshold, the extension line of the visual lane centerline from the sight point to the extension point is used as the extended lane centerline; if the lateral distance difference is greater than or equal to the set threshold, the position of the front vehicle is used as the extension point, and the sight point and the extension point are used as the two endpoints of the extended lane centerline to obtain the equation of the extended lane centerline.

[0011] According to a second aspect of the present invention, an intelligent driving device is provided, comprising at least one processor and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor; and the at least one processor is used to execute the instructions to implement the steps of the aforementioned method.

[0012] According to a third aspect of the present invention, a readable storage medium is provided, in which computer instructions are stored. When the computer instructions are executed by a processor, the steps of the aforementioned method are implemented.

[0013] According to a fourth aspect of the present invention, there is provided a computer program product, comprising computer instructions, which, when executed by a processor, implement the steps of the aforementioned method.

[0014] The present invention has at least the following advantages:

[0015] 1. The embodiment of the present invention processes the surrounding vehicle targets and guardrail information obtained by the vehicle-mounted multi-sensor based on the road model obtained by the camera, expands the road model of a certain sight distance when conditions are met, and better expresses the road shape of special scenes, thereby helping the autonomous driving vehicle to control the vehicle more smoothly;

[0016] 2. Low implementation cost. The basic configuration of autonomous driving is a camera + millimeter wave radar solution. By fully tapping the potential of this solution, the practicality and comfort of autonomous driving can be expanded without the need for additional configuration;

[0017] 3. Low computational complexity and small amount of code. Based on the characteristics of the road transition curve, two key points are used to calculate the equation of the extended lane centerline, and the road model is expressed using a cubic polynomial, which has low computational complexity. In addition, by comparing the extension point of the visual lane centerline with the position of the vehicle in front, the changes in the road structure ahead can be quickly determined. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1A schematic flow chart of a road model expansion method based on multi-sensor fusion according to an embodiment of the present invention is shown.

[0019] Figure 2 A schematic diagram showing a visual lane centerline and an extended lane centerline in a method according to an embodiment of the present invention is shown.

[0020] Figure 3 A schematic diagram showing a method for determining a road structure change when a visual lane centerline is a curved line in an embodiment of the present invention. DETAILED DESCRIPTION

[0021] See also Figure 1 According to an embodiment of the present invention, the road model expansion method based on multi-sensor fusion includes the following steps:

[0022] Get the centerline equation of the visual lane;

[0023] Determine whether the preceding vehicle is a valid preceding vehicle. If so, check whether the distance between the preceding vehicle and the vehicle is greater than the lane line sight distance.

[0024] If the distance between the front vehicle and the self-vehicle is greater than the lane line sight distance, the center line of the visual lane is extended to the extension point, and the lateral distance difference between the extension point and the position of the front vehicle is compared; the x coordinate of the extension point is equal to the x coordinate of the position of the front vehicle, and the lateral distance difference is the absolute value of the difference between the y coordinate of the extension point and the y coordinate of the position of the front vehicle; the aforementioned x coordinate and y coordinate are the coordinates of the self-vehicle in the vehicle coordinate system;

[0025] If the lateral distance difference is less than the set threshold, the extension line of the visual lane centerline from the sight point to the extension point is used as the centerline of the extended lane. If the lateral distance difference is greater than or equal to the set threshold, the position of the front vehicle is used as the extension point, and the sight point and the extension point are used as the two endpoints of the centerline of the extended lane to obtain the centerline equation of the extended lane.

[0026] In this embodiment, the center line equation of the visual lane is obtained by calculating the center line equation of the visual lane according to the polynomial coefficients of the left lane line and the right lane line sent by the front view camera of the vehicle. The center line equation of the visual lane is: y = c 10 +c 11 .x+c 12 .x 2 +c 13 .x 3 ; Where y is the y coordinate in the xoy coordinate system, and x is the x coordinate in the xoy coordinate system. c li is the polynomial coefficient of the left lane line sent by the front-view camera, c riis the polynomial coefficient of the right lane line emitted by the front camera. The above xoy coordinate system is the vehicle coordinate system of the vehicle, and the vehicle coordinate system takes the direction of vehicle forward movement as the positive direction of the x-axis, and the y-axis is perpendicular to the x-axis. Preferably, the above-mentioned left lane line and right lane line are the left lane line and right lane line of the lane where the vehicle is located, respectively.

[0027] The centerline equation of the visual lane mentioned above refers to the equation of the centerline of the lane where the vehicle is located within the sight range of the on-board front-view camera. This embodiment is based on the premise that the visual lane line is valid, and uses the lane line equation obtained by the on-board front-view camera to calculate the centerline equation of the lane. The front-view camera is usually installed on the front windshield of the vehicle. The lane line equation obtained by the front-view camera includes:

[0028] Left lane line equation: y = c l0 +c l1 .x+c l2 .x 2 +c l3 .x 3

[0029] Right lane line equation: y = c r0 +c r1 .x+c r2 .x 2 +c r3 .x 3

[0030] Where: c l0 ,c l1 ,c l2 ,c l3 is the polynomial coefficient of the left lane line, which is sent by the front-view camera;

[0031] c r0 ,c r1 ,c r2 ,c r3 is the polynomial coefficient of the right lane line, which is sent by the front-view camera;

[0032] The lane in which the vehicle is currently located is the host lane, the first lane on the left adjacent to the host lane is the left lane, and the first lane on the right adjacent to the host lane is the right lane.

[0033] The center line equation of the visual lane can be calculated by the above formula: y = c 10 +c 11 .x+c 12 .x 2 +c 13 .x 3 ,in:

[0034] The preceding vehicle in this embodiment is a target vehicle located in the lane where the vehicle is located and closest to the vehicle. In this embodiment, the determination of whether the preceding vehicle is a valid preceding vehicle includes the following steps:

[0035] S21, multi-sensor detection and tracking of all vehicle trajectories within the field of view, fitting the guardrail equation through the guardrail radar reflection points

[0036] In this embodiment, the multi-sensor includes at least a forward-looking camera and a millimeter-wave radar, that is, at least all vehicle trajectories within the field of view of the forward-looking camera and the millimeter-wave radar are detected and tracked. The aforementioned "field of view" refers to the FOV (Field Of View) of the sensor. Preferably, the multi-sensor also includes a lidar, an ultrasonic sensor, etc., in which case all vehicle trajectories within a wider field of view including the forward-looking camera and the millimeter-wave radar can be detected. The application of multi-sensor fusion technology in the field of autonomous driving is to improve the vehicle's perception of the surrounding environment. It achieves more comprehensive and accurate perception by integrating data from different sensors.

[0037] The vehicle position within the field of view is detected and tracked by the on-board camera and millimeter-wave radar, and the record is updated and unified to the current vehicle coordinate system. Then, the least square method is used to fit the three-dimensional polynomials to obtain the vehicle trajectory. Most highways and urban expressways have guardrails (generally speaking, the front-view camera can detect the guardrails nearby, but the guardrails farther away require millimeter-wave radar to detect). The millimeter-wave radar detection reflection point can be fitted into a three-dimensional polynomial equation. In this way, several cubic polynomials can be used to express the surrounding environment information.

[0038] S22, respectively calculating the similarity between the trajectory of the front vehicle and the trajectory of other vehicles within the field of view, and the similarity between the trajectory of the front vehicle and the guardrail

[0039] The cubic polynomial of the front vehicle trajectory and the cubic polynomial of the other vehicle trajectory and the cubic polynomial of the guardrail 3 are respectively calculated for similarity. To calculate the similarity of two cubic polynomials, the following methods in the prior art can be used:

[0040] Compare the coefficients of two cubic polynomials and calculate their Euclidean distance;

[0041] Take several sample points within a certain range and calculate the values ​​and mean square error (MSE) of the two polynomials at these points;

[0042] Treat polynomial coefficients as vectors, compute their cosine similarity, etc.

[0043] By setting a similarity threshold, it is determined whether the vehicle trajectory and guardrail in the surrounding environment are similar to the trajectory of the preceding vehicle.

[0044] S23: If the sum of the number of similarities between the front vehicle trajectory and other vehicle trajectories and the number of similarities between the front vehicle trajectory and the guardrail exceeds a set threshold, the front vehicle is determined to be a valid front vehicle.

[0045] If the number of cubic polynomials of surrounding vehicles and guardrails that are similar to the trajectory of the front vehicle is greater than the set threshold, it means that the driving trajectory of the front vehicle is consistent with the surrounding traffic flow and the driving direction matches the guardrail. Then it can be considered that the front vehicle is driving stably without changing lanes, and the front vehicle has a stable and effective front vehicle target. On this basis, it can be judged whether the distance between the front vehicle and the vehicle is greater than the lane line sight distance. The lane line sight distance refers to the length of the front lane line that can be detected by the front-view camera.

[0046] The range of the extended road model in the embodiment of the present invention is the distance from the lane line sight distance to the front vehicle. Only when the distance between the front vehicle and the vehicle is greater than the lane line sight distance, the extension condition is met and the subsequent steps are continued.

[0047] Figure 2 A schematic diagram showing a visual lane centerline and an extended lane centerline in a method according to an embodiment of the present invention is shown. Figure 2 The scene shown is a straight road entering a curve. In fact, the method of the present invention is also applicable to scenes where the road curvature changes greatly, such as a curve entering a straight road or a sharp turn.

[0048] exist Figure 2 In the xoy coordinate system (i.e., the vehicle coordinate system of the ego vehicle), the center of the rear axle of the ego vehicle 1 is the origin o, the direction along the vehicle is x, and the direction perpendicular to the direction is y. The lane line sight distance of the front camera (i.e., the longitudinal sight distance of the lane line) is x1, and the coordinates of the sight distance point P of the center line of the visual lane are (x 1 ,y 1 ), the effective position of the preceding vehicle 2 is (x 2 ,y 2 ), the coordinates of the extension point M extending from the center line L1 of the visual lane are (x 2 ,y 2 '), the x coordinate of the extension point M is equal to the x coordinate of the front vehicle position, both are x 2 The lateral distance difference Δy between the extension point M and the position of the front vehicle 2 is Δy 2 =|y′ 2 -y 2 |.

[0049] If the lateral distance difference Δy 2 If the value is less than the set threshold, the road structure beyond the camera's sight range can be judged and the center line of the visual lane can be used for extension. The extended center line L2 from the sight point P to the extension point M is used as the center line of the extended lane, which is not much different from the actual road structure.

[0050] If the lateral distance difference Δy 2 If the value is greater than or equal to the set threshold, it can be judged that for road structures beyond the camera's viewing range, if the visual lane line is simply extended, there will be a large gap between the actual road structure and the position of the front vehicle. In this case, the center line L3 of the extended lane is calculated using the position of the front vehicle as the extension point N.

[0051] The greater the lateral distance difference, the greater the change in the road structure ahead. The center line of the extended lane calculated by the embodiment of the present invention will be closer to the actual road structure, which can provide a more accurate preview control basis for the autonomous driving and improve the performance in such scenarios. In some specific implementations, the above-mentioned set threshold is related to the curvature of the current road. For example, when the curvature radius of the current road is 2000m, the threshold is set to 0.5m.

[0052] In this embodiment, the centerline equation of the extended lane is: y = c 20 +c 21 .x+c 22 .x 2 +c 23 .x 3 y is the y coordinate in the xoy coordinate system (i.e., the vehicle coordinate system of the vehicle), and x is the x coordinate in the xoy coordinate system.

[0053] Based on the requirement of road curvature continuity, the first-order and second-order derivatives of the road model at the sight point P are equal.

[0054]

[0055] where y′ 1 and y″ 1 They represent the first-order and second-order derivatives of the centerline of the extended lane at the sight point P respectively.

[0056] According to the above formula, the coefficient of the center line of the extended lane can be calculated:

[0057]

[0058] Where Δy = y 2 -y 1 , Δx=x 2 -x 1 .

[0059] Furthermore, the road model expansion method of the embodiment of the present invention further includes determining the lateral distance difference Δy 2 The step of judging the road structure change after the value is greater than or equal to the set threshold value specifically includes:

[0060] If the lateral distance difference Δy 2If the value is greater than or equal to the set threshold and the center line L1 of the visual lane is a straight line, it is judged that the road ahead is about to enter a curve. For the principle diagram, please refer to Figure 2 ;

[0061] If the lateral distance difference Δy 2 If the value is greater than or equal to the set threshold and the center line of the visual lane L1 is a curve, the road structure is judged based on whether the position of the front vehicle is offset toward the inside or outside of the curve: if the position of the front vehicle is offset toward the inside of the curve, the curvature of the road ahead tends to increase; if the position of the front vehicle is offset toward the outside of the curve, the curvature of the road ahead tends to decrease.

[0062] Using the above steps, a quick qualitative judgment of road structure changes can be achieved. Figure 3 A schematic diagram of judging changes in road structure when the visual lane centerline L1 is a curve in the method of an embodiment of the present invention is shown. When the visual lane centerline L1 is a curve, if the position of the effective target (the front vehicle 2) is offset toward the inside of the curve, it indicates that the curvature of the road ahead has a tendency to increase (common scenario: the vehicle is in the process of entering a curve). If the position of the effective target (the front vehicle 2) is offset toward the outside of the curve, it indicates that the curvature of the road ahead has a tendency to decrease (common scenario: the vehicle is in the process of exiting a curve).

[0063] According to another embodiment of the present invention, an intelligent driving device includes at least one processor and a memory communicatively connected to the at least one processor; the memory stores instructions that can be executed by the at least one processor; and the at least one processor is used to execute the instructions to implement the steps of the aforementioned method.

[0064] According to another embodiment of the present invention, a readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the steps of the aforementioned method are implemented.

[0065] According to yet another embodiment of the present invention, a computer program product includes computer instructions. When the computer instructions are executed by a processor, the steps of the aforementioned method are implemented.

[0066] The embodiment of the present invention processes the surrounding vehicle target and guardrail information obtained by the on-board multi-sensor based on the road model obtained by the camera, expands the road model of a certain sight distance when conditions are met, and better expresses the road shape of special scenes, thereby helping the autonomous driving vehicle to control the vehicle more smoothly.

[0067] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A road model expansion method based on multi-sensor fusion, characterized in that: The following steps are involved: Get the centerline equation of the visual lane; Determine whether the preceding vehicle is a valid preceding vehicle. If so, check whether the distance between the preceding vehicle and the vehicle is greater than the lane line sight distance. If the distance between the front vehicle and the vehicle itself is greater than the lane line sight distance, the center line of the visual lane is extended to the extension point, and the lateral distance difference between the extension point and the position of the front vehicle is compared; the x coordinate of the extension point is equal to the x coordinate of the position of the front vehicle, and the lateral distance difference is the absolute value of the difference between the y coordinate of the extension point and the y coordinate of the position of the front vehicle; If the lateral distance difference is less than the set threshold, the extension line of the visual lane centerline from the sight point to the extension point is used as the extended lane centerline; if the lateral distance difference is greater than or equal to the set threshold, the position of the preceding vehicle is used as the extension point, and the sight point and the extension point are used as the two endpoints of the extended lane centerline to obtain the equation of the extended lane centerline.

2. The road model expansion method based on multi-sensor fusion according to claim 1, characterized in that: The method for obtaining the center line equation of the visual lane includes: calculating the center line equation of the visual lane according to the polynomial coefficients of the left lane line and the right lane line sent by the vehicle's front-view camera, and the center line equation of the visual lane is: y = c 10 +c 11 .x+c 12 .x 2 +c 13 .x 3 ; Where y is the y coordinate in the xoy coordinate system, and x is the x coordinate in the xoy coordinate system. i=0~3,c li is the polynomial coefficient of the left lane line sent by the front-view camera, c ri The polynomial coefficients of the right lane line sent by the front-view camera.

3. The road model expansion method based on multi-sensor fusion as claimed in claim 1, characterized in that: The determining whether the preceding vehicle is a valid preceding vehicle comprises: Multi-sensor detection and tracking of all vehicle trajectories within the field of view, and fitting the guardrail equation through the guardrail radar reflection points; Calculate the similarity between the front vehicle trajectory and other vehicle trajectories, and the similarity between the front vehicle trajectory and the guardrail; If the sum of the number of similarities between the front vehicle's trajectory and other vehicle's trajectories and the number of similarities between the front vehicle's trajectory and the guardrail exceeds a set threshold, the front vehicle is determined to be a valid front vehicle.

4. The road model expansion method based on multi-sensor fusion as claimed in claim 1, characterized in that: The centerline equation of the extended lane is: y = c 20 +c 21 .x+c 22 .x 2 +c 23 .x 3 ; Among them, y is the y coordinate in the xoy coordinate system, x is the x coordinate in the xoy coordinate system, the coordinates of the sight point of the visual lane centerline are (x1, y1), the coordinates of the extended point are (x2, y2), y′1 and y″1 respectively represent the first-order and second-order derivatives of the centerline of the extended lane at the sight point, Δy=y2-y1, Δx=x2-x1.

5. The road model expansion method based on multi-sensor fusion as claimed in claim 1, characterized in that: The road model extension method further includes: If the lateral distance difference is greater than or equal to the set threshold, and the visual lane centerline is a straight line, it is determined that a curve is about to be entered ahead; If the lateral distance difference is greater than or equal to the set threshold value, and the center line of the visual lane is a curve, the road structure is judged based on whether the position of the front vehicle is offset toward the inside or outside of the curve: if the position of the front vehicle is offset toward the inside of the curve, it is judged that the curvature of the road ahead has a tendency to increase; if the position of the front vehicle is offset toward the outside of the curve, it is judged that the curvature of the road ahead has a tendency to decrease.

6. The road model expansion method based on multi-sensor fusion as claimed in claim 1, characterized in that: The multi-sensor includes a forward-looking camera and a millimeter-wave radar.

7. An intelligent driving device, comprising at least one processor and a memory in communication with the at least one processor; the memory stores instructions executable by the at least one processor; characterized in that: The at least one processor is configured to execute the instructions to implement the steps of the method according to any one of claims 1 to 6.

8. A readable storage medium, characterized in that: The readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images