Target 3D position prediction method based on 2D image and high-precision map fusion

Abstract

The invention discloses a target 3D position prediction method based on 2D image and high-precision map fusion. The method comprises the following steps: acquiring an image in a certain range in front of a vehicle; marking a corresponding target bounding box for each target obstacle on the image; based on the positioning information of the vehicle, a high-precision map in a certain range in front of the vehicle is obtained; acquiring all lanes in the same direction as the own vehicle on the high-precision map, and projecting feature points of left and right lane lines of each lane to the acquired image; performing secondary projection on all target bounding boxes on the acquired image and feature points of left and right lane lines of all lanes to a pseudo overlook map; matching and associating each target obstacle with all lanes on the pseudo overlook plane map; and determining the position information of each target obstacle according to the relative position relationship between the target obstacle and the lane. According to the invention, through fusing high-resolution image target detection and high-precision map information, accurate position information of middle and long-distance target obstacles is obtained.

Description

Technical Field

[0001] The present invention belongs to the technical field of autonomous driving, and specifically relates to a method for predicting the 3D position of a target based on the fusion of 2D images and high-precision maps in an autonomous driving scenario. Background Art

[0002] 3D target detection is the main task of perception in autonomous driving scenarios. It can simultaneously detect all obstacles and targets within a large range around the vehicle, making it possible to achieve autonomous driving functions and goals including close-range obstacle avoidance, autonomous lane change, and speed adjustment to enhance physical perception. Mainstream sensor perception technologies include visual image perception, microwave perception, lidar perception, and so on. Among them, visual perception is still widely used for its wide detection range and high efficiency. The technology of image target recognition based on deep learning has become increasingly mature. How to infer the 3D position information of the target obstacle from the 2D image target detection results is a key problem that needs to be solved in practical applications. Monocular ranging and binocular ranging are the two common types of ranging solutions based on visual perception.

[0003] LiDAR sensing technology usually uses a sensor solution of multiple radars. It obtains point cloud information (a series of points and reflection intensity values) by scanning the surrounding environment. This is used as the input of the model for target detection. It can directly obtain the 3D information of the target and obtain a high-precision ranging result. However, the field of view of the radar sensor is small, and multiple radars need to be used in combination; and the computing power required for point cloud processing is also large, which has a high application cost in the field of autonomous driving. When detecting targets with small reflective surfaces at a distance, it will be difficult to identify due to the weak reflection wave.

[0004] Visual perception technology uses cameras as sensors and images as model inputs for target detection. Among them, the system structure of the monocular ranging method is relatively simple. The position and posture of the target are estimated through the detection results of a single camera, and the resource requirements are not high. Usually, the prior size of the target, the size of the target in the image, and the internal and external parameters of the camera are used to perform coordinate conversion operations from 2D pixels to the 3D world. The target position obtained is not a true distance measurement, and the results are prone to serious deviations. The binocular ranging solution directly uses the parallax of the two cameras to calculate the distance from the target to the camera based on the principle of triangulation, which is more accurate and stable than the monocular ranging result. However, the binocular system has a large amount of calculation and high performance requirements for the computing unit, making it difficult to commercialize this solution in the field of autonomous driving.

[0005] In addition, the technical solution of multimodal fusion can also complete the target ranging task. For example, the point cloud data and image data collected by the radar are preprocessed, and the preprocessed data is used as the input of the model to directly obtain the 3D bounding box of the target obstacle. However, in the scenario of autonomous driving, the accuracy, real-time and robustness of the fusion of different modal inputs still face great challenges, and the training cost of its model is also relatively high. Summary of the invention

[0006] In order to solve the above technical problems, the present invention proposes a method for predicting the 3D position of a target based on the fusion of 2D images and high-precision maps.

[0007] In order to achieve the above object, the technical solution of the present invention is as follows:

[0008] In a first aspect, the present invention discloses a method for predicting a 3D position of a target based on fusion of a 2D image and a high-precision map, comprising:

[0009] Step S1: using a camera to collect images within a certain range in front of the vehicle;

[0010] Step S2: Detect target obstacles in the captured image using the target detection model, and mark each target obstacle with its corresponding target bounding box on the image;

[0011] Step S3: Based on the positioning information of the vehicle, a high-precision map within a certain range in front of the vehicle is obtained;

[0012] Step S4: Obtain all lanes on the high-precision map that are in the same direction as the vehicle, and project the feature points of the left and right lane lines of each lane onto the collected image;

[0013] Step S5: reprojecting the feature points of all target bounding boxes and left and right lane lines of all lanes on the captured image onto the pseudo top-down view;

[0014] Step S6: on the pseudo top-down map, each target obstacle is matched and associated with all lanes, and the relative position relationship between each target obstacle and the lane is determined to be any of the following: the target obstacle is in any lane, the target obstacle is on the lane line of any lane, and the target obstacle is not in any lane;

[0015] Step S7: Determine the position information of each target obstacle according to the relative position relationship between the target obstacle and the lane.

[0016] Based on the above technical solution, the following improvements can be made:

[0017] As a preferred solution, step S6 specifically includes:

[0018] Step S6.1: establishing a pseudo top-down coordinate system with the width direction of the lane where the vehicle is located as the horizontal axis direction and the length direction of the lane where the vehicle is located as the vertical axis direction;

[0019] Step S6.2: In the pseudo top-down coordinate system, a judgment is made based on the ordinate values ​​of the two target projection points of the bottom edge of the target bounding box of each target obstacle. If the ordinate value of any one of the two target projection points is within the interval of the ordinate values ​​of the two lane projection points of the lane line of any lane, then the lane is the associated lane of the corresponding target obstacle.

[0020] Step S6.3: among all lane projection points of the left and right lane lines of the associated lane, find the point p1 on the left lane line closest to the starting target projection point, the point p2 on the right lane line closest to the starting target projection point, the point p3 on the left lane line closest to the ending target projection point, and the point p4 on the right lane line closest to the ending target projection point in the vertical axis direction of the pseudo top view coordinate system;

[0021] Step S6.4: Determine whether the abscissa value of the starting target projection point is within the interval range of the abscissa values ​​of points p1 and p2, and whether the abscissa value of the ending target projection point is within the interval range of the abscissa values ​​of points p3 and p4.

[0022] If both are present, it indicates that the target obstacle is within the associated lane;

[0023] If the abscissa value of the starting target projection point is within the interval of the abscissa values ​​of points p1 and p2, and the abscissa value of the ending target projection point is not within the interval of the abscissa values ​​of points p3 and p4, it indicates that the target obstacle is on the left lane line of the associated lane;

[0024] If the abscissa value of the starting target projection point is not within the interval of the abscissa values ​​of points p1 and p2, and the abscissa value of the ending target projection point is within the interval of the abscissa values ​​of points p3 and p4, it indicates that the target obstacle is on the right lane line of the associated lane;

[0025] If neither is the case, it indicates that the target obstacle is not in the associated lane.

[0026] As a preferred solution, step S7 includes:

[0027] Step S7.1: Obtain the projection points on the left and right lane lines of the associated lane closest to the start and end target projection points of the target bounding box of each target obstacle in step S6;

[0028] Step S7.2: converting the obtained projection points on the left and right lane lines of the nearest associated lane into real feature points of the left and right lane lines in the corresponding world coordinate system;

[0029] Step S7.3: Based on the relative position relationship between the target obstacle and the associated lane and the real feature points of the left and right lane lines in the world coordinate system, the position information of the target obstacle is obtained.

[0030] As a preferred solution, step S7.3 includes:

[0031] Determine the relative position relationship between the target obstacle and the associated lane;

[0032] If the target obstacle is in the associated lane, the center point of the target obstacle is the midpoint of the line connecting the real feature points of the left and right lane lines of the associated lane in the world coordinate system;

[0033] If the target obstacle is on any lane line of the associated lane, the center point of the target obstacle is the real feature point of the corresponding lane line of the associated lane in the world coordinate system.

[0034] In a second aspect, the present invention discloses a device for predicting a 3D position of a target based on the fusion of a 2D image and a high-precision map, comprising:

[0035] A collection module, used to use a camera to collect images within a certain range in front of the vehicle;

[0036] The target detection module is used to detect the target obstacles in the captured image using the target detection model, and mark the corresponding target bounding box for each target obstacle on the image;

[0037] A high-precision map acquisition module is used to obtain a high-precision map within a certain range in front of the vehicle based on the vehicle's positioning information;

[0038] The first projection module is used to obtain all lanes in the same direction as the vehicle on the high-precision map, and project the feature points of the left and right lane lines of each lane onto the collected image;

[0039] The second projection module is used to re-project the feature points of all target bounding boxes and left and right lane lines of all lanes on the captured image onto a pseudo top-down view.

[0040] A matching and associating module is used to match and associate each target obstacle with all lanes on the pseudo top-down map, and determine whether the relative position relationship between each target obstacle and the lane is any of the following: the target obstacle is in any lane, the target obstacle is on the lane line of any lane, or the target obstacle is not in any lane;

[0041] The target position determination module is used to determine the position information of each target obstacle according to the relative position relationship between the target obstacle and the lane.

[0042] As a preferred solution, the matching association module specifically includes:

[0043] A coordinate system establishing unit, used for establishing a pseudo top-down coordinate system with the width direction of the lane where the vehicle is located as the horizontal axis direction and the length direction of the lane where the vehicle is located as the vertical axis direction;

[0044] An associated lane acquisition unit is used to make a judgment based on the ordinate values ​​of the starting and ending target projection points of the bottom edge of the target bounding box of each target obstacle in a pseudo top-down coordinate system. If the ordinate value of any one of the starting and ending target projection points is within the interval range of the ordinate values ​​of the starting and ending lane projection points of the lane line of any lane, then the lane is the associated lane corresponding to the target obstacle;

[0045] A searching unit is used to search, from all lane projection points of the left and right lane lines of the associated lane, a point p1 on the left lane line closest to the starting target projection point, a point p2 on the right lane line closest to the starting target projection point, a point p3 on the left lane line closest to the ending target projection point, and a point p4 on the right lane line closest to the ending target projection point in the vertical axis direction of the pseudo top-down coordinate system;

[0046] The matching unit is used to determine whether the horizontal coordinate value of the starting target projection point is within the interval range of the horizontal coordinate values ​​of points p1 and p2, and whether the horizontal coordinate value of the ending target projection point is within the interval range of the horizontal coordinate values ​​of points p3 and p4.

[0047] If both are present, it indicates that the target obstacle is within the associated lane;

[0048] If the abscissa value of the starting target projection point is within the interval of the abscissa values ​​of points p1 and p2, and the abscissa value of the ending target projection point is not within the interval of the abscissa values ​​of points p3 and p4, it indicates that the target obstacle is on the left lane line of the associated lane;

[0049] If the abscissa value of the starting target projection point is not within the interval of the abscissa values ​​of points p1 and p2, and the abscissa value of the ending target projection point is within the interval of the abscissa values ​​of points p3 and p4, it indicates that the target obstacle is on the right lane line of the associated lane;

[0050] If neither is the case, it indicates that the target obstacle is not in the associated lane.

[0051] As a preferred solution, the target location determination module includes:

[0052] An acquisition unit, used for acquiring the projection points on the left and right lane lines of the associated lane closest to the start and end target projection points of the target bounding box of each target obstacle obtained by the matching association module;

[0053] A conversion unit, used for converting the obtained projection points on the left and right lane lines of the nearest associated lane into real feature points of the left and right lane lines in the corresponding world coordinate system;

[0054] The target position determination unit is used to obtain the position information of the target obstacle based on the relative position relationship between the target obstacle and the associated lane and the real feature points of the left and right lane lines in the world coordinate system.

[0055] As a preferred solution, the target position determination unit is used to perform the following method:

[0056] Determine the relative position relationship between the target obstacle and the associated lane;

[0057] If the target obstacle is in the associated lane, the center point of the target obstacle is the midpoint of the line connecting the real feature points of the left and right lane lines of the associated lane in the world coordinate system;

[0058] If the target obstacle is on any lane line of the associated lane, the center point of the target obstacle is the real feature point of the corresponding lane line of the associated lane in the world coordinate system.

[0059] In a third aspect, the present invention discloses a computing device, comprising:

[0060] one or more processors;

[0061] Memory;

[0062] And one or more programs, wherein the one or more programs are stored in a memory and configured to be executed by one or more processors, and the one or more programs include instructions for any of the above-mentioned methods for predicting the 3D position of a target based on the fusion of a 2D image and a high-precision map.

[0063] In a fourth aspect, the present invention discloses a storage medium storing one or more computer-readable programs, wherein the one or more programs include instructions suitable for being loaded by a memory and executing any of the above-mentioned methods for predicting the 3D position of a target based on the fusion of a 2D image and a high-precision map.

[0064] The present invention discloses a method for predicting the 3D position of a target based on the fusion of 2D images and high-precision maps. The present invention obtains accurate position information of medium- and long-distance target obstacles by fusing high-resolution image target detection with high-precision map information, thereby assisting the vehicle in making the optimal road selection and making reasonable lane change behavior. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0066] Figure 1 A flowchart of a method for predicting the 3D position of a target based on the fusion of 2D images and high-precision maps provided in an embodiment of the present invention.

[0067] Figure 2 A schematic diagram of an image with a target bounding box and lane line feature points provided by an embodiment of the present invention.

[0068] Figure 3 A schematic diagram of a pseudo top view of the lane where the vehicle is located after secondary projection provided by an embodiment of the present invention.

[0069] Figure 4 A schematic diagram of a pseudo top view of a target bounding box and all lanes after secondary projection provided by an embodiment of the present invention.

[0070] Figure 5 A real-world coordinate top view with target obstacle location information provided by an embodiment of the present invention.

[0071] Figure 6 A block diagram of a device for predicting the 3D position of a target based on the fusion of a 2D image and a high-precision map provided in an embodiment of the present invention.

[0072] Figure 7 A schematic diagram of a computing device provided for an embodiment of the present invention. DETAILED DESCRIPTION

[0073] The preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0074] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0075] Using ordinal numbers “first,” “second,” “third,” etc. to describe common objects merely indicates that different instances of similar objects are involved and is not intended to imply that the objects so described must have a given order in time, space, order, or in any other manner.

[0076] In addition, the expression of “comprising” an element is an “open” expression, which merely means that corresponding components or steps exist, and should not be interpreted as excluding additional components or steps.

[0077] In order to achieve the purpose of the present invention, in some embodiments, a method for predicting a 3D position of a target based on the fusion of a 2D image and a high-precision map includes:

[0078] Step S101: using a camera to collect images within a certain range (e.g., 80 to 200 meters) in front of the vehicle;

[0079] Step S102: Detect target obstacles in the captured image using the target detection model, and mark each target obstacle with its corresponding target bounding box on the image;

[0080] Step S103: Based on the positioning information of the vehicle, a high-precision map within a certain range (e.g., 200 meters) in front of the vehicle is obtained;

[0081] Step S104: Obtain all lanes on the high-precision map that are in the same direction as the vehicle, and project the feature points of the left and right lane lines of each lane onto the collected image;

[0082] Step S105: reprojecting the feature points of all target bounding boxes and left and right lane lines of all lanes on the captured image onto a pseudo top-down view;

[0083] Step S106: on the pseudo top-down map, each target obstacle is matched and associated with all lanes, and it is determined that the relative position relationship between each target obstacle and the lane is any of the following: the target obstacle is in any lane, the target obstacle is on the lane line of any lane, and the target obstacle is not in any lane;

[0084] Step S107: Determine the position information of each target obstacle according to the relative position relationship between the target obstacle and the lane.

[0085] Each of the above steps is explained in detail below.

[0086] Step S101 is an image acquisition step. In this embodiment, a high-resolution camera can be selected as the main sensor to acquire 8MP images to ensure that target obstacles at medium and long distances (eg, 80 to 200 meters) can be detected.

[0087] Step S102 is a target detection step. The target detection model is a relatively mature model in the art and will not be described in detail here. The present invention only needs to use the target detection model to detect the target boundary box (pixel coordinates of the four vertices of the rectangular box) of the target obstacle.

[0088] Step S103 is a high-precision map acquisition step. The present invention acquires the positioning information of the vehicle from the IMU sensor of the vehicle in real time, so as to search for a high-precision map within the range of the vehicle's driving concern (200m in front of the vehicle).

[0089] Step S104 is a projection step.

[0090] The specific methods and formats of obtaining map information for different high-precision map software interfaces may be slightly different. However, the principles and main contents are generally the same. Through the high-precision map software interface, all lanes within a certain range near the vehicle positioning information are searched.

[0091] In this embodiment, in order to reduce the computational cost, the present invention only focuses on obstacles on the same lane that are likely to affect the vehicle, and ignores the acquired opposite lane information. The lane information of the map includes the feature points of the left and right lane lines of the lane.

[0092] In order to obtain the relative relationship between the target obstacle and the lane, that is, to find the lane where the target is located, the present invention projects the lane line feature points in the 3D world coordinate system onto the image plane. The relative relationship between the target bounding box and the lane line (within the range of interest) is determined on the 2D image plane. This projection process is actually a conversion from world coordinates to pixel coordinates. Specifically, the feature points of the left and right lane lines of the high-precision map in the three-dimensional world coordinate system are projected onto the two-dimensional captured image through the following formula;

[0093]

[0094] Where: X w , Y w , Z w is the world coordinate;

[0095] X c , Y c , Z c is the camera coordinate;

[0096] u, v are pixel coordinates;

[0097] R, t is the camera extrinsic parameter;

[0098] f x , f y , u0, v0 are the camera internal parameters.

[0099] The projection result is as follows Figure 2 As shown in the figure, the white dots are the lane feature points projected onto the image. It is worth noting that there are actually hundreds or thousands of feature points on the map, but this is only a simplified illustration to help understanding.

[0100] Step S105 is a secondary projection operation step. Figure 2 In the projection result of the lane line on the image, it can be seen intuitively that due to the perspective relationship, the map lane line feature points sampled at equal intervals in the world coordinate system become uneven after being projected onto the image, that is, sparse near and dense far away; the left and right boundary lines of the lane also change from two parallel straight lines to a trapezoid that is narrow at the top and wide at the bottom. Such a situation is not conducive to unified calculation processing. Therefore, the present invention will perform secondary projection on the lane line feature points.

[0101] According to the shape of the projection of the current lane line feature point on the image, find an appropriate transformation matrix so that the lane line feature point (pixel coordinates) can present an approximately uniform distribution after transformation, and the left and right boundary lines of the lane can be restored to an approximately parallel state.

[0102] The transformed projection surface is a pseudo top-down plane (i.e., a bird's-eye view plane), which is the visual effect when the camera is facing the road. This transformation process is a perspective transformation, which gives the pixel coordinates of four points on the original plane and the pixel coordinates of the corresponding four points on the target plane, and obtains a 3×3 transformation matrix. Here, you can directly call functions in method libraries such as opencv for calculation.

[0103] It is worth noting that in this embodiment, the starting and ending points of the left and right lane line feature point sets of the lane where the vehicle is located are used as the four points on the original plane. After projection, the lane where the vehicle is located occupies the central position of the pseudo top-down plane map, as shown in FIG. Figure 3 shown.

[0104] Figure 3 In the figure, the starting and ending projection points of the left lane line of the lane where the vehicle is located are The starting and ending projection points of the right lane line are The length and width of the lanes on the pseudo-top-down map are set according to the size of the image, and their coordinates can be written directly.

[0105] Use the calculated projection matrix M to perform projection transformation operations on all lane feature points Get the projection of all lanes on the pseudo top-down map (specifically: l 1 , l 2 , l 3 is the projection of three lanes), such as Figure 4 shown.

[0106] In order to calculate the relative position relationship between the target detection bounding box and the lane, the target bounding box (such as Figure 2 The two vertices of the bottom edge of the white rectangular box shown is the target bounding box) are also projected to the pseudo top-down plane, and the perspective transformation method is the same as above. Figure 4 o1 and o2 are projections of two target obstacle bounding boxes. In the present invention, only two vertices at the bottom of the target bounding box need to be paid attention to, which can better represent the target grounding point information.

[0107] Step S106 is a target and lane relationship matching and associating step. The present invention divides the relative position relationship between the target and the lane line into three categories: the target obstacle is in any lane, the target obstacle is on the lane line of any lane, and the target obstacle is not in any lane.

[0108] The present invention performs calculations on a pseudo top view, and the determination method is relatively simple. Step S106 specifically includes:

[0109] Step S106.1: Establish a pseudo top-down coordinate system with the width direction of the lane where the vehicle is located as the horizontal axis and the length direction of the lane where the vehicle is located as the vertical axis (e.g. Figure 4 u, v directions in );

[0110] Step S106.2: In the pseudo top view coordinate system, based on the ordinate values ​​(v values) of the two target projection points at the bottom edge of the target bounding box of each target obstacle, a judgment is made. If the ordinate value (v value) of any one of the two target projection points at the bottom edge is within the interval of the ordinate values ​​(v values) of the two lane projection points at the top edge and the bottom edge of the lane line of any lane, then the lane is the associated lane of the corresponding target obstacle (the associated lanes may be one or more).

[0111] Step S106.3: Among all the lane projection points of the left and right lane lines of the associated lane, find the point on the left lane line that is closest to the target projection point in the vertical axis direction of the pseudo-top view coordinate system The point on the right lane line closest to the starting and target projection points The point on the left lane line closest to the projection point of the stop target The point on the right lane line closest to the projection point of the stop target Where: i is the corresponding lane number;

[0112] Step S106.4: Determine whether the horizontal coordinate value (u value) of the starting target projection point is at point point Is the horizontal coordinate value (u value) of the target projection point within the range of the horizontal coordinate value (u value) of the point point Within the range of the horizontal axis value (u value),

[0113] If both are present, it indicates that the target obstacle is within the associated lane;

[0114] If the horizontal coordinate value (u value) of the starting target projection point is at point point The horizontal coordinate value (u value) of the target projection point is not within the range of the horizontal coordinate value (u value) of the point point If the target obstacle is within the range of the horizontal coordinate value (u value), it indicates that the target obstacle is on the left lane line of the associated lane;

[0115] If the horizontal coordinate value (u value) of the starting target projection point is not at point point The horizontal coordinate value (u value) of the target projection point is within the range of the horizontal coordinate value (u value) of the target projection point. point If the target obstacle is within the range of the horizontal coordinate value (u value), it means that the target obstacle is on the right lane line of the associated lane;

[0116] If neither is the case, it indicates that the target obstacle is not in the associated lane.

[0117] Step S107 is a step for determining the location information of each target obstacle. Step S107 includes:

[0118] Step S107.1: Obtain the projection points on the left and right lane lines of the associated lane closest to the start and end target projection points of the target bounding box of each target obstacle in step S106;

[0119] Step S107.2: converting the obtained projection points on the left and right lane lines of the nearest associated lane into real feature points of the left and right lane lines in the corresponding world coordinate system;

[0120] Step S107.3: Based on the relative position relationship between the target obstacle and the associated lane and the real feature points of the left and right lane lines in the world coordinate system, the position information of the target obstacle is obtained.

[0121] Specifically, step S107.3 includes:

[0122] Determine the relative position relationship between the target obstacle and the associated lane;

[0123] If the target obstacle is in the associated lane, the center point of the target obstacle is the midpoint of the line connecting the real feature points of the left and right lane lines of the associated lane in the world coordinate system;

[0124] If the target obstacle is on any lane line of the associated lane, the center point of the target obstacle is the real feature point of the corresponding lane line of the associated lane in the world coordinate system.

[0125] Furthermore, the orientation of the target object is consistent with the orientation of the lane (local curvature of the lane). The prior size of the target obstacle type can be used to assist in restoring the 3D bounding box.

[0126] Obstacles that are not in any lane are not in the focus range and no output is made.

[0127] Output diagram in real world coordinate top view as follows Figure 5 As shown, l1, l2, and l3 are three lanes, and o1 and o2 are the bounding boxes of two target obstacles.

[0128] In summary, most of the existing target distance measurement solutions based on 2D image detection are based on the back-projection process from pixel coordinate points to world coordinate points. This process requires a lot of prior information and preset conditions, and is extremely dependent on the accuracy of camera calibration parameters. The results obtained have great uncertainty, which has a great impact on the stability of the system. The present invention changes and innovates in the overall thinking, projects reference points in the world coordinate system onto the image, and this calculation process does not require preset conditions, which greatly improves the error tolerance of camera calibration parameters.

[0129] The present invention integrates the information of high-precision maps to perform target ranging, making full use of the resources on the autonomous driving vehicle.

[0130] The present invention projects lane feature points onto an image, which is more intuitive and can provide a reference for subsequent optimal path planning of the vehicle.

[0131] The present invention uses high-resolution images as the basis for target detection. On the premise of completing the task of long-distance target ranging, it greatly saves the system hardware cost and avoids the huge consumption of system resources by large models.

[0132] The present invention can detect the target position of target obstacles within a certain range (e.g. 80-200m) in front of the vehicle, with an average error of 5% and high stability. The location information of the target obstacle can effectively provide a reference for the vehicle route planning downstream, which has great benefits for practical applications in the field of autonomous driving.

[0133] In addition, in some embodiments, the present invention discloses a method for predicting the 3D position of a target based on the fusion of a 2D image and a high-precision map, such as Figure 6 As shown, including:

[0134] The acquisition module 201 is used to use a camera to acquire images within a certain range in front of the vehicle;

[0135] The target detection module 202 is used to detect the target obstacles in the captured image using the target detection model, and mark the corresponding target bounding box for each target obstacle on the image;

[0136] A high-precision map acquisition module 203 is used to acquire a high-precision map within a certain range in front of the vehicle based on the positioning information of the vehicle;

[0137] The first projection module 204 is used to obtain all lanes in the same direction as the vehicle on the high-precision map, and project the feature points of the left and right lane lines of each lane onto the collected image;

[0138] The second projection module 205 is used to re-project the feature points of all target bounding boxes and left and right lane lines of all lanes on the captured image onto a pseudo top-down view;

[0139] The matching and associating module 206 is used to match and associate each target obstacle with all lanes on the pseudo top-down map, and determine whether the relative position relationship between each target obstacle and the lane is any of the following: the target obstacle is in any lane, the target obstacle is on the lane line of any lane, and the target obstacle is not in any lane;

[0140] The target position determination module 207 is used to determine the position information of each target obstacle according to the relative position relationship between the target obstacle and the lane.

[0141] Furthermore, the matching association module specifically includes:

[0142] A coordinate system establishing unit, used for establishing a pseudo top-down coordinate system with the width direction of the lane where the vehicle is located as the horizontal axis direction and the length direction of the lane where the vehicle is located as the vertical axis direction;

[0143] An associated lane acquisition unit is used to make a judgment based on the ordinate values ​​of the starting and ending target projection points of the bottom edge of the target bounding box of each target obstacle in a pseudo top-down coordinate system. If the ordinate value of any one of the starting and ending target projection points is within the interval range of the ordinate values ​​of the starting and ending lane projection points of the lane line of any lane, then the lane is the associated lane corresponding to the target obstacle;

[0144] A searching unit is used to search, from all lane projection points of the left and right lane lines of the associated lane, a point p1 on the left lane line closest to the starting target projection point, a point p2 on the right lane line closest to the starting target projection point, a point p3 on the left lane line closest to the ending target projection point, and a point p4 on the right lane line closest to the ending target projection point in the vertical axis direction of the pseudo top-down coordinate system;

[0145] The matching unit is used to determine whether the horizontal coordinate value of the starting target projection point is within the interval range of the horizontal coordinate values ​​of points p1 and p2, and whether the horizontal coordinate value of the ending target projection point is within the interval range of the horizontal coordinate values ​​of points p3 and p4.

[0146] If both are present, it indicates that the target obstacle is within the associated lane;

[0147] If the abscissa value of the starting target projection point is within the interval of the abscissa values ​​of points p1 and p2, and the abscissa value of the ending target projection point is not within the interval of the abscissa values ​​of points p3 and p4, it indicates that the target obstacle is on the left lane line of the associated lane;

[0148] If the abscissa value of the starting target projection point is not within the interval of the abscissa values ​​of points p1 and p2, and the abscissa value of the ending target projection point is within the interval of the abscissa values ​​of points p3 and p4, it indicates that the target obstacle is on the right lane line of the associated lane;

[0149] If neither is the case, it indicates that the target obstacle is not in the associated lane.

[0150] Further, the target location determination module includes:

[0151] An acquisition unit, used for acquiring the projection points on the left and right lane lines of the associated lane closest to the start and end target projection points of the target bounding box of each target obstacle obtained by the matching association module;

[0152] A conversion unit, used for converting the obtained projection points on the left and right lane lines of the nearest associated lane into real feature points of the left and right lane lines in the corresponding world coordinate system;

[0153] The target position determination unit is used to obtain the position information of the target obstacle based on the relative position relationship between the target obstacle and the associated lane and the real feature points of the left and right lane lines in the world coordinate system.

[0154] Further, the target position determination unit is used to perform the following method:

[0155] Determine the relative position relationship between the target obstacle and the associated lane;

[0156] If the target obstacle is in the associated lane, the center point of the target obstacle is the midpoint of the line connecting the real feature points of the left and right lane lines of the associated lane in the world coordinate system;

[0157] If the target obstacle is on any lane line of the associated lane, the center point of the target obstacle is the real feature point of the corresponding lane line of the associated lane in the world coordinate system.

[0158] Furthermore, it should be noted that: the target 3D position prediction device based on the fusion of 2D images and high-precision maps provided in the above embodiments only uses the division of the above-mentioned functional modules as an example when predicting the 3D position of the target obstacle. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the target 3D position prediction device based on the fusion of 2D images and high-precision maps is divided into different functional modules to complete all or part of the functions described above.

[0159] In addition, the target 3D position prediction device based on the fusion of 2D images and high-precision maps provided in the above-mentioned embodiments and the target 3D position prediction method based on the fusion of 2D images and high-precision maps belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0160] In addition, in some other embodiments, Figure 7 As shown, the present invention also discloses a computing device, including:

[0161] One or more processors 301;

[0162] Memory 302;

[0163] And one or more programs, wherein the one or more programs are stored in the memory 302 and are configured to be executed by one or more processors 301, and the one or more programs include instructions of the method for predicting the 3D position of the target based on the fusion of 2D image and high-precision map disclosed in the above embodiment.

[0164] The processor 301 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 301 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 301 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 301 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 301 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0165] The memory 302 may include one or more computer-readable storage media, which may be non-transitory. The memory 302 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 302 is used to store at least one instruction, which is used to be executed by the processor 301 to implement the method for predicting the target 3D position based on the fusion of 2D images and high-precision maps provided in the method embodiment of the present invention.

[0166] In addition, the computing device may also optionally include: a peripheral device interface and at least one peripheral device. The processor 301, the memory 302 and the peripheral device interface may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface via a bus, a signal line or a circuit board. Schematically, the peripheral devices include but are not limited to: a radio frequency circuit, a touch display screen, an audio circuit, and a power supply.

[0167] Of course, the computing device may also include fewer or more components, which is not limited in this embodiment.

[0168] In addition, in some other embodiments, the present invention also discloses a storage medium, which stores one or more computer-readable programs, and the one or more programs include instructions, which are suitable for being loaded by a memory and executing the method for predicting the 3D position of a target based on the fusion of 2D images and high-precision maps disclosed in the above embodiments.

[0169] The present invention discloses a method for predicting the 3D position of a target based on the fusion of 2D images and high-precision maps. The present invention obtains accurate position information of medium- and long-distance target obstacles by fusing high-resolution image target detection with high-precision map information, thereby assisting the vehicle in making the optimal road selection and making reasonable lane change behavior.

[0170] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for illustrating the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which shall fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A method for predicting the 3D position of a target based on the fusion of 2D images and high-precision maps, characterized in that: include: Step S1: using a camera to collect images within a certain range in front of the vehicle; Step S2: Detect target obstacles in the captured image using the target detection model, and mark each target obstacle with its corresponding target bounding box on the image; Step S3: Based on the positioning information of the vehicle, a high-precision map within a certain range in front of the vehicle is obtained; Step S4: Obtain all lanes on the high-precision map that are in the same direction as the vehicle, and project the feature points of the left and right lane lines of each lane onto the collected image; Step S5: reprojecting the feature points of all target bounding boxes and left and right lane lines of all lanes on the captured image onto the pseudo top-down view; Step S6: on the pseudo top-down map, each target obstacle is matched and associated with all lanes, and the relative position relationship between each target obstacle and the lane is determined to be any of the following: the target obstacle is in any lane, the target obstacle is on the lane line of any lane, and the target obstacle is not in any lane; Step S7: Determine the position information of each target obstacle according to the relative position relationship between the target obstacle and the lane.

2. The prediction method according to claim 1, characterized in that: The step S6 specifically includes: Step S6.1: establishing a pseudo top-down coordinate system with the width direction of the lane where the vehicle is located as the horizontal axis direction and the length direction of the lane where the vehicle is located as the vertical axis direction; Step S6.2: In the pseudo top-down coordinate system, a judgment is made based on the ordinate values ​​of the two target projection points of the bottom edge of the target bounding box of each target obstacle. If the ordinate value of any one of the two target projection points is within the interval of the ordinate values ​​of the two lane projection points of the lane line of any lane, then the lane is the associated lane of the corresponding target obstacle. Step S6.3: among all lane projection points of the left and right lane lines of the associated lane, find the point p1 on the left lane line closest to the starting target projection point, the point p2 on the right lane line closest to the starting target projection point, the point p3 on the left lane line closest to the ending target projection point, and the point p4 on the right lane line closest to the ending target projection point in the vertical axis direction of the pseudo top view coordinate system; Step S6.4: Determine whether the abscissa value of the starting target projection point is within the interval range of the abscissa values ​​of points p1 and p2, and whether the abscissa value of the ending target projection point is within the interval range of the abscissa values ​​of points p3 and p4. If both are present, it indicates that the target obstacle is within the associated lane; If the abscissa value of the starting target projection point is within the interval of the abscissa values ​​of points p1 and p2, and the abscissa value of the ending target projection point is not within the interval of the abscissa values ​​of points p3 and p4, it indicates that the target obstacle is on the left lane line of the associated lane; If the abscissa value of the starting target projection point is not within the interval of the abscissa values ​​of points p1 and p2, and the abscissa value of the ending target projection point is within the interval of the abscissa values ​​of points p3 and p4, it indicates that the target obstacle is on the right lane line of the associated lane; If neither is the case, it indicates that the target obstacle is not in the associated lane.

3. The prediction method according to claim 2, characterized in that: The step S7 comprises: Step S7.1: Obtain the projection points on the left and right lane lines of the associated lane closest to the start and end target projection points of the target bounding box of each target obstacle in step S6; Step S7.2: converting the obtained projection points on the left and right lane lines of the nearest associated lane into real feature points of the left and right lane lines in the corresponding world coordinate system; Step S7.3: Based on the relative position relationship between the target obstacle and the associated lane and the real feature points of the left and right lane lines in the world coordinate system, the position information of the target obstacle is obtained.

4. The prediction method according to claim 3, characterized in that: The step S7.3 comprises: Determine the relative position relationship between the target obstacle and the associated lane; If the target obstacle is in the associated lane, the center point of the target obstacle is the midpoint of the line connecting the real feature points of the left and right lane lines of the associated lane in the world coordinate system; If the target obstacle is on any lane line of the associated lane, the center point of the target obstacle is the real feature point of the corresponding lane line of the associated lane in the world coordinate system.

5. A device for predicting the 3D position of a target based on the fusion of a 2D image and a high-precision map, characterized in that: include: A collection module, used to use a camera to collect images within a certain range in front of the vehicle; The target detection module is used to detect the target obstacles in the captured image using the target detection model, and mark the corresponding target bounding box for each target obstacle on the image; A high-precision map acquisition module is used to obtain a high-precision map within a certain range in front of the vehicle based on the vehicle's positioning information; The first projection module is used to obtain all lanes in the same direction as the vehicle on the high-precision map, and project the feature points of the left and right lane lines of each lane onto the collected image; The second projection module is used to re-project the feature points of all target bounding boxes and left and right lane lines of all lanes on the captured image onto a pseudo top-down view. A matching and associating module is used to match and associate each target obstacle with all lanes on the pseudo top-down map, and determine whether the relative position relationship between each target obstacle and the lane is any of the following: the target obstacle is in any lane, the target obstacle is on the lane line of any lane, or the target obstacle is not in any lane; The target position determination module is used to determine the position information of each target obstacle according to the relative position relationship between the target obstacle and the lane.

6. The prediction device according to claim 5, characterized in that The matching association module specifically includes: A coordinate system establishing unit, used for establishing a pseudo top-down coordinate system with the width direction of the lane where the vehicle is located as the horizontal axis direction and the length direction of the lane where the vehicle is located as the vertical axis direction; An associated lane acquisition unit is used to make a judgment based on the ordinate values ​​of the starting and ending target projection points of the bottom edge of the target bounding box of each target obstacle in a pseudo top-down coordinate system. If the ordinate value of any one of the starting and ending target projection points is within the interval range of the ordinate values ​​of the starting and ending lane projection points of the lane line of any lane, then the lane is the associated lane corresponding to the target obstacle; A searching unit is used to search, from all lane projection points of the left and right lane lines of the associated lane, a point p1 on the left lane line closest to the starting target projection point, a point p2 on the right lane line closest to the starting target projection point, a point p3 on the left lane line closest to the ending target projection point, and a point p4 on the right lane line closest to the ending target projection point in the vertical axis direction of the pseudo top-down coordinate system; The matching unit is used to determine whether the horizontal coordinate value of the starting target projection point is within the interval range of the horizontal coordinate values ​​of points p1 and p2, and whether the horizontal coordinate value of the ending target projection point is within the interval range of the horizontal coordinate values ​​of points p3 and p4. If both are present, it indicates that the target obstacle is within the associated lane; If the abscissa value of the starting target projection point is within the interval of the abscissa values ​​of points p1 and p2, and the abscissa value of the ending target projection point is not within the interval of the abscissa values ​​of points p3 and p4, it indicates that the target obstacle is on the left lane line of the associated lane; If the abscissa value of the starting target projection point is not within the interval of the abscissa values ​​of points p1 and p2, and the abscissa value of the ending target projection point is within the interval of the abscissa values ​​of points p3 and p4, it indicates that the target obstacle is on the right lane line of the associated lane; If neither is the case, it indicates that the target obstacle is not in the associated lane.

7. The prediction device according to claim 6, characterized in that The target location determination module comprises: An acquisition unit, used for acquiring the projection points on the left and right lane lines of the associated lane closest to the start and end target projection points of the target bounding box of each target obstacle obtained by the matching association module; A conversion unit, used for converting the obtained projection points on the left and right lane lines of the nearest associated lane into real feature points of the left and right lane lines in the corresponding world coordinate system; The target position determination unit is used to obtain the position information of the target obstacle based on the relative position relationship between the target obstacle and the associated lane and the real feature points of the left and right lane lines in the world coordinate system.

8. The prediction device according to claim 7, characterized in that The target position determination unit is used to perform the following method: Determine the relative position relationship between the target obstacle and the associated lane; If the target obstacle is in the associated lane, the center point of the target obstacle is the midpoint of the line connecting the real feature points of the left and right lane lines of the associated lane in the world coordinate system; If the target obstacle is on any lane line of the associated lane, the center point of the target obstacle is the real feature point of the corresponding lane line of the associated lane in the world coordinate system.

9. A computing device, characterized in that include: one or more processors; Memory; And one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors, and one or more of the programs include instructions for the method for predicting the 3D position of a target based on the fusion of 2D images and high-precision maps as described in any of claims 1-4 above.

10. A storage medium, characterized in that The storage medium stores one or more computer-readable programs, and the one or more programs include instructions, which are suitable for being loaded by the memory and executing the method for predicting the 3D position of the target based on the fusion of 2D image and high-precision map as described in any of the above claims 1-4.

Images