High-precision map lane line location determination method, device and autonomous vehicle

Abstract

This disclosure provides a method, apparatus, and autonomous vehicle for determining lane line positions in high-precision maps, relating to the field of image processing technology, particularly to autonomous driving and high-precision mapping. The specific implementation scheme is as follows: Acquire image frames containing lane lines and corresponding radar point cloud frames; for each image frame, fit the ground plane equation in the vehicle coordinate system based on the corresponding radar point cloud frame; for each image frame, sample the lane lines in that image frame to obtain sampling points; project the sampling points of the image frame onto the corresponding ground plane equation to obtain the ground three-dimensional coordinates of the sampling points in the vehicle coordinate system; transform each sampling point to the world coordinate system based on its ground three-dimensional coordinates; and obtain the position of the lane line in the world coordinate system based on the world coordinates of each sampling point. This achieves the determination of the lane line's position in the world coordinate system.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to the fields of autonomous driving and high-precision maps. Background Technology

[0002] High-precision maps, also known as high-resolution electronic maps, are used in autonomous vehicles. They possess accurate vehicle location information and rich road element data, helping cars anticipate complex road conditions such as slope, curvature, and heading, thus better avoiding potential risks. In autonomous driving scenarios, a vehicle's perception of its surroundings relies heavily on high-precision electronic maps. These maps significantly contribute to the safety and stability of autonomous driving tasks, especially the position of lane lines, which has a crucial impact on autonomous driving. Summary of the Invention

[0003] This disclosure provides a method, apparatus, and autonomous vehicle for determining lane line positions in high-precision maps.

[0004] According to one aspect of this disclosure, a method for determining the position of a lane line is provided, the method comprising:

[0005] Acquire image frames containing lane lines and radar point cloud frames corresponding to those image frames;

[0006] For each image frame, the ground plane equation in the vehicle coordinate system corresponding to that image frame is obtained by fitting the radar point cloud frame corresponding to that image frame.

[0007] For each image frame, the lane lines in the image frame are sampled to obtain the sampling points of the image frame; the sampling points of the image frame are projected onto the ground plane equation corresponding to the image frame to obtain the ground three-dimensional coordinates of the sampling points of the image frame in the vehicle coordinate system corresponding to the image frame.

[0008] Based on the ground three-dimensional coordinates of each sampling point, the world coordinates of each sampling point are obtained by transforming each sampling point into the world coordinate system.

[0009] Based on the world coordinates of each sampling point, the position of the lane line in the world coordinate system is obtained.

[0010] According to another aspect of this disclosure, a lane line position determination device is provided, the device comprising:

[0011] The relevant data acquisition module is used to acquire image frames containing lane lines and radar point cloud frames corresponding to the image frames;

[0012] The ground plane equation fitting module is used to fit the ground plane equation in the vehicle coordinate system corresponding to each image frame based on the radar point cloud frame corresponding to that image frame.

[0013] The ground 3D coordinate determination module is used to sample the lane lines in each image frame to obtain the sampling points of the image frame; and to project the sampling points of the image frame onto the ground plane equation corresponding to the image frame to obtain the ground 3D coordinates of the sampling points of the image frame in the vehicle coordinate system corresponding to the image frame.

[0014] The world coordinate transformation module is used to transform each sampling point into the world coordinate system based on the ground three-dimensional coordinates of each sampling point to obtain the world coordinates of each sampling point;

[0015] The lane line position determination module is used to obtain the position of the lane line in the world coordinate system based on the world coordinates of each sampling point.

[0016] According to another aspect of this disclosure, an electronic device is provided, comprising:

[0017] At least one processor; and

[0018] A memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the lane line position determination method described in any one of the present applications.

[0020] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are configured to cause the computer to perform the lane line position determination method described in any one of this application.

[0021] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the lane line position determination method described in any one of this application.

[0022] According to another aspect of this disclosure, an autonomous vehicle is provided, including any of the electronic devices described in this application.

[0023] In this embodiment of the disclosure, the position of the lane line in the world coordinate system is determined.

[0024] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0025] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0026] Figure 1 This is a schematic diagram of a lane line position determination method according to an embodiment of the present disclosure;

[0027] Figure 2 This is a schematic diagram of a possible implementation of step S102 in an embodiment of this disclosure;

[0028] Figure 3 This is a schematic diagram of a possible implementation of step S103 in an embodiment of this disclosure;

[0029] Figure 4 This is a schematic diagram of a possible implementation of step S104 in an embodiment of this disclosure;

[0030] Figure 5 This is a schematic diagram of a bird's-eye view of the lane line sampling points in an embodiment of this disclosure;

[0031] Figure 6 This is a schematic diagram of a lane line position determination device according to an embodiment of the present disclosure;

[0032] Figure 7 This is another schematic diagram of the lane line position determination device according to an embodiment of the present disclosure;

[0033] Figure 8 This is a block diagram of an electronic device used to implement the lane line position determination method of the embodiments of this disclosure. Detailed Implementation

[0034] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0035] In scenarios such as autonomous driving, determining the position of lane lines is crucial for the normal driving of vehicles. To determine the position of lane lines, this application provides a method for determining lane line positions. The method includes: acquiring an image frame containing lane lines and a radar point cloud frame corresponding to the image frame; for each image frame, fitting the ground plane equation in the vehicle coordinate system corresponding to the image frame based on the radar point cloud frame corresponding to the image frame; for each image frame, sampling the lane lines in the image frame to obtain sampling points of the image frame; projecting the sampling points of the image frame into the ground plane equation corresponding to the image frame to obtain the ground three-dimensional coordinates of the sampling points in the vehicle coordinate system corresponding to the image frame; transforming each sampling point to the world coordinate system based on the ground three-dimensional coordinates of each sampling point to obtain the world coordinates of each sampling point; and obtaining the position of the lane line in the world coordinate system based on the world coordinates of each sampling point. In this embodiment, the position of the lane line in the world coordinate system is determined. Furthermore, the accuracy of 3D positioning based on radar point cloud frames is much higher than that based on image frames. The position of the lane line obtained by combining image frames and radar point cloud frames greatly increases the accuracy of the lane line position compared to obtaining the position of the lane line based solely on image frames.

[0036] The lane line position determination method provided in this disclosure will be described in detail below through specific embodiments.

[0037] The lane line location determination method provided in this disclosure can be implemented through an in-vehicle terminal, a smartphone, or a cloud server connected to the in-vehicle terminal. This lane line location determination method can be applied to fields such as autonomous driving, artificial intelligence, intelligent transportation, and electronic maps.

[0038] See Figure 1 , Figure 1 The lane line position determination method according to an embodiment of this application includes:

[0039] S101, acquire an image frame containing lane lines and a radar point cloud frame corresponding to the image frame.

[0040] In one example, the vehicle is equipped with an image acquisition device and a LiDAR (Light Detection and Ranging) system. The image acquisition device can capture image frames containing lane lines in real time, while the LiDAR can capture radar point cloud frames. Generally, the LiDAR's sampling frequency is higher than that of the image acquisition device. Therefore, after acquiring the image frames and radar point cloud frames, they need to be aligned in time. For any given radar point cloud frame, the image frame whose sampling time is closest to that of the radar point cloud frame is considered its corresponding image frame. This establishes a correspondence between the two, thus obtaining the radar point cloud frames corresponding to each image frame. In one example, one image frame corresponds to multiple radar point cloud frames.

[0041] S102, for each image frame, based on the radar point cloud frame corresponding to that image frame, fit the ground plane equation in the vehicle coordinate system corresponding to that image frame.

[0042] The vehicle coordinate system refers to a three-dimensional coordinate system with the vehicle as the origin. Although the vehicle's position in the world coordinate system may change during its movement, the transformation relationship between the radar coordinate system and the vehicle coordinate system, as well as the transformation relationship between the camera coordinate system and the vehicle coordinate system, will not change because radar, image acquisition equipment, etc. are mounted on the vehicle.

[0043] For each image frame, the ground in that image frame is considered to be a plane. The ground plane equation is fitted to the points representing the ground in each radar point cloud frame corresponding to that image frame. Based on the transformation relationship between the radar coordinate system and the vehicle coordinate system, the ground plane equation in the vehicle coordinate system corresponding to that image frame is obtained.

[0044] S103, for each image frame, sample the lane lines in the image frame to obtain the sampling points of the image frame; project the sampling points of the image frame onto the ground plane equation corresponding to the image frame to obtain the ground three-dimensional coordinates of the sampling points of the image frame in the vehicle coordinate system corresponding to the image frame.

[0045] In real-world scenarios, lane lines are drawn on the ground; therefore, the sampling points within the lane lines must satisfy the constraints of the ground plane equation. Sampling of lane lines in an image frame can be done by sampling at equal intervals according to the actual length of the lane lines, or by sampling at equal pixel intervals according to image pixels, both of which are within the scope of this application. For each image frame, the sampling points of that frame are projected onto the corresponding ground plane equation, thereby obtaining the three-dimensional ground coordinates of the sampling points in the vehicle coordinate system corresponding to that image frame.

[0046] S104. Based on the ground three-dimensional coordinates of each sampling point, transform each sampling point into the world coordinate system to obtain the world coordinates of each sampling point.

[0047] The transformation relationship between the vehicle coordinate system and the world coordinate system is obtained, and the ground 3D coordinates of the sampling points are transformed into the world coordinate system to obtain the world coordinates of the sampling points. In one example, the world coordinate system is the 3D coordinate system of the high-precision map.

[0048] In one possible implementation, for each sampling point, the translation vector and rotation matrix determined when the sampling point was collected are obtained to obtain the translation vector and rotation matrix corresponding to the sampling point; using the translation vector and rotation matrix corresponding to the sampling point, the ground three-dimensional coordinates of the sampling point are transformed into the world coordinate system to obtain the world coordinates of the sampling point.

[0049] The translation vector and rotation matrix determined when acquiring the sampling point represent the correspondence between the vehicle coordinate system and the world coordinate system at that sampling point. Although the vehicle's position in the world coordinate system may change, its position and attitude in the world coordinate system can be obtained through the vehicle's positioning system, i.e., its pose in the world coordinate system. Based on the vehicle's pose in the world coordinate system, the transformation relationship between the vehicle coordinate system and the world coordinate system can be obtained. In one example, for each sampling point, based on the vehicle's pose at the time the sampling point is acquired, the transformation relationship between the vehicle coordinate system and the world coordinate system at that time can be acquired. This transformation relationship can be represented by a translation matrix and a rotation matrix, i.e., the translation vector and rotation matrix corresponding to the sampling point. In this embodiment, by using the translation vector and rotation matrix corresponding to the sampling point, the sampling point can be transformed from the vehicle coordinate system to the world coordinate system, accurately completing the coordinate transformation and thus improving the accuracy of lane line position determination.

[0050] S105: Based on the world coordinates of each sampling point, obtain the position of the lane line in the world coordinate system.

[0051] Based on the world coordinates of each sampling point, the position of the lane line can be obtained in the world coordinate system. In one example, a linear fit of the lane line can be performed on each sampling point in the world coordinate system to obtain the position of the lane line in the world coordinate system.

[0052] In this embodiment, the position of the lane line in the world coordinate system is determined, and the position of the lane line can be determined in real time during vehicle operation, which is applicable to autonomous driving scenarios. In addition, the accuracy of 3D positioning based on radar point cloud frames is much higher than that based on image frames. The position of the lane line obtained by combining image frames and radar point cloud frames greatly increases the accuracy of the lane line position compared to obtaining the position of the lane line based solely on image frames.

[0053] To determine the ground equations more accurately, in one possible implementation, see [link to relevant documentation]. Figure 2 The step of fitting the ground plane equation in the vehicle coordinate system corresponding to each image frame based on the radar point cloud frame corresponding to that image frame includes:

[0054] S201, for each image frame, classify the points in the radar point cloud frame corresponding to that image frame to obtain the ground radar points representing the ground corresponding to that image frame.

[0055] In one example, a pre-trained deep learning model can be used to classify points in a radar point cloud frame to obtain ground radar points representing the ground. The training process of the deep learning model can be found in the model training process section of related technologies. In one example, a sample radar point cloud frame can be input into the deep learning model for prediction to obtain the preset ground radar points in that sample radar point cloud frame. The loss of the deep learning model is calculated based on the ground radar points labeled in the sample radar point cloud frame and the preset ground radar points. The training parameters of the deep learning model are adjusted based on this loss, and sample radar point cloud frames are selected for continued training until the loss converges, resulting in a trained deep learning model.

[0056] S202, using the ground radar points corresponding to the image frame to perform plane fitting, the ground plane equation in the vehicle coordinate system corresponding to the image frame is obtained.

[0057] For each image frame, a plane fitting is performed using the corresponding ground radar points to ensure that all ground radar points for that image frame lie on the same plane as much as possible, thus obtaining the ground plane equation in the vehicle coordinate system corresponding to that image frame. In one example, the ground plane equation can be expressed as ax + by + cz + h = 0, where a, b, and c are the coefficients to be calculated, and a... 2 +b 2 +c 2 =1, where x, y, and z are the three-dimensional coordinates of the ground radar point.

[0058] In this embodiment of the disclosure, classifying the points in the radar point cloud frame corresponding to the image frame can accurately obtain the ground radar points representing the ground; using the ground radar points to fit the ground plane equation can improve the accuracy of the ground plane equation, thereby ultimately improving the accuracy of the lane line position.

[0059] In one possible implementation, see Figure 3 The step of projecting the sampling points of the image frame into the ground plane equation corresponding to the image frame to obtain the ground three-dimensional coordinates of the sampling points of the image frame in the vehicle coordinate system corresponding to the image frame includes:

[0060] S301, Based on the image coordinates of the sampling points in the image frame, determine the horizontal and vertical coordinates of the sampling points in the image frame in the vehicle coordinate system corresponding to the image frame.

[0061] An image frame can accurately represent the x and y coordinates of a sampling point, but it cannot accurately represent the depth information of the sampling point. Therefore, the x and y coordinates of the sampling point in the vehicle coordinate system corresponding to the image frame can be obtained based on the image coordinates of the sampling point in the image frame.

[0062] S302, Substitute the x-coordinate and y-coordinate of the sampling point in the image frame in the vehicle coordinate system corresponding to the image frame into the ground plane equation corresponding to the image frame to obtain the ground three-dimensional coordinates of the sampling point in the image frame in the vehicle coordinate system corresponding to the image frame.

[0063] Lane lines are drawn on the ground. After obtaining the x-coordinate and y-coordinate of the sampling point in the vehicle coordinate system, they can be substituted into the ground plane return of the vehicle coordinate system corresponding to the image frame to obtain the depth coordinate of the sampling point, which is to obtain the ground three-dimensional coordinate of the sampling point of the image frame in the vehicle coordinate system corresponding to the image frame.

[0064] In this embodiment of the disclosure, the depth coordinates of the sampling points are obtained by using the ground plane equation, which makes up for the inability of image frames to accurately obtain depth information, and can increase the accuracy of the obtained ground three-dimensional coordinates of the sampling points, thereby ultimately increasing the accuracy of the lane line position.

[0065] In one possible implementation, see Figure 4 For each sampling point, the translation vector and rotation matrix determined when the sampling point was acquired are obtained to obtain the corresponding translation vector and rotation matrix for that sampling point; using the corresponding translation vector and rotation matrix, the ground 3D coordinates of the sampling point are transformed to the world coordinate system to obtain the world coordinates of the sampling point, including:

[0066] S401, for each image frame, obtain the vehicle's pose in the world coordinate system when the radar point cloud frame corresponding to that image frame was acquired; based on the pose, determine the transformation relationship between the vehicle coordinate system and the world coordinate system corresponding to that image frame.

[0067] In one example, the transformation relationship between the vehicle coordinate system and the world coordinate system corresponding to the image frame can be represented by the first translation vector and the rotation matrix.

[0068] S402, For each sampling point, based on the ground three-dimensional coordinates of the sampling point in the vehicle coordinate system, and with the constraint that the sampling point is in a plane perpendicular to the ground plane equation, the three-dimensional coordinates of the sampling point are calculated and used as the origin three-dimensional coordinates of the sampling point.

[0069] In one example, such as Figure 5 As shown, Figure 5 This is a bird's-eye view of the lane line. The lane line can be represented as a set of many three-dimensional sampling points with two degrees of freedom. Specifically, each three-dimensional sampling point is restricted to move on a plane perpendicular to the equation of the ground plane, so it has only two degrees of freedom. The coordinate variables of the sampling point can be represented as (0, V1, V2), where V1 is the projected coordinate on the Y-axis and V2 is the projected coordinate on the Z-axis.

[0070] S403, for each sampling point, determine the translation vector of the three-dimensional coordinates from the origin of the vehicle coordinate system corresponding to the sampling point to the origin of the sampling point to obtain the target translation vector of the sampling point.

[0071] In one example, for each sampling point, a sampling point three-dimensional coordinate system is established with the origin three-dimensional coordinates of the sampling point as the origin, and the target translation vector from the vehicle coordinate system corresponding to the sampling point to the origin three-dimensional coordinate system of the sampling point is determined. In this case, the X, Y, and Z axes of the origin three-dimensional coordinate system of the sampling point are parallel to the X, Y, and Z axes of the vehicle coordinate system corresponding to the sampling point, respectively.

[0072] S404. Using the translation vector, target translation vector, and rotation matrix corresponding to the sampling point, the origin three-dimensional coordinates of the sampling point are transformed to the world coordinate system to obtain the world coordinates of the sampling point.

[0073] In one example, the translation vector and rotation matrix corresponding to the sampling point are the translation vector and rotation matrix from the vehicle coordinate system to the world coordinate system where the sampling point is located. These can be obtained based on the vehicle's pose. The X, Y, and Z axes in the origin 3D coordinate system of the sampling point are parallel to the X, Y, and Z axes in the vehicle coordinate system corresponding to the sampling point, respectively. Therefore, the coordinate transformation between the origin 3D coordinate system and the vehicle coordinate system can be achieved directly through the target translation vector.

[0074] In one example, the world coordinates of a sampling point can be represented as:

[0075] P = T init +R init (0, V1, V2)

[0076] Among them, T init R is the vector sum of the translation vector corresponding to the sampling point and the target translation vector of that sampling point. init This is the rotation matrix corresponding to the sampling point.

[0077] In this embodiment of the disclosure, the world coordinates of the sampling point in the world coordinate system are obtained by using the constraint that the sampling point is in a plane perpendicular to the equation of the ground plane, which can increase the accuracy of the obtained world coordinates of the sampling point.

[0078] To accurately associate sampling points with lane lines, in one possible implementation, the above method further includes:

[0079] Step 1: Calculate the fitted curve equations for each lane line in the specified image frame.

[0080] The specified image frame can be customized according to the actual situation. In one example, the latest image frame can be selected as the specified image frame. Curve fitting can be performed using the sampling points of each lane line in the specified image frame to obtain the fitted curve equation of that lane line, and finally, the fitted curve equation of each lane line is obtained.

[0081] Step 2: Transform the ground 3D coordinates of each sampling point to the vehicle coordinate system corresponding to the specified image frame to obtain the transformed 3D coordinates of each sampling point;

[0082] Sampling points from multiple image frames before or after a specified image frame can be selected, and the ground 3D coordinates of these sampling points can be transformed to the vehicle coordinate system corresponding to the specified image frame. In one example, if the specified image frame is the latest image frame, sampling points from a preset number of previous image frames can be selected. Using the vehicle's pose when acquiring each image frame, these sampling points can be transformed from the vehicle coordinate system of their respective image frames to the vehicle coordinate system corresponding to the specified image frame. The resulting transformed coordinates are called transformed 3D coordinates.

[0083] Step 3: Determine the lane line to which each sampling point belongs based on the transformed three-dimensional coordinates of each sampling point and the fitted curve equation of each lane line in the specified image frame.

[0084] Based on the transformed three-dimensional coordinates of each sampling point and the fitted curve equation of the lane line, the fitted curve equation of the sampling point can be obtained, and thus the lane line to which the sampling point belongs can be obtained.

[0085] In one possible implementation, determining the lane to which each sampling point belongs based on its transformed 3D coordinates and the fitted curve equations of each lane in the specified image frame includes: for each sampling point, calculating the distance between the sampling point and the fitted curve equations of each lane in the image frame; if the minimum distance is less than a preset distance threshold, the sampling point is determined to belong to the lane corresponding to the minimum distance; if the minimum distance is not less than the preset distance threshold, the sampling point is determined not to belong to any lane in the specified image frame. The preset distance threshold can be customized according to actual conditions, for example, set to 3 cm, 5 cm, or 10 cm. In real-world scenarios, the number of lanes may vary; for example, on a wider road, there may be multiple lanes, while on a narrower road, there may only be one lane; correspondingly, the number of lane lines will also change. For sampling points whose minimum distance is not less than the preset distance threshold, it is considered that the sampling point does not belong to any lane in the specified image frame, thus making it applicable to situations where the number of lane lines varies.

[0086] In this embodiment of the disclosure, the lane line to which the sampling point belongs is associated by fitting the curve equation of the lane line. This can effectively determine the lane line associated with the sampling point, thereby accurately using the world coordinates of the sampling point to associate the lane line and improving the accuracy of the obtained lane line position.

[0087] This disclosure also provides a lane line position determination device, see [link to relevant documentation]. Figure 6 The device includes:

[0088] The relevant data acquisition module 601 is used to acquire image frames containing lane lines and radar point cloud frames corresponding to the image frames;

[0089] The ground plane equation fitting module 602 is used to fit the ground plane equation in the vehicle coordinate system corresponding to each image frame based on the radar point cloud frame corresponding to that image frame.

[0090] The ground three-dimensional coordinate determination module 603 is used to sample the lane lines in each image frame to obtain the sampling points of the image frame; and to project the sampling points of the image frame onto the ground plane equation corresponding to the image frame to obtain the ground three-dimensional coordinates of the sampling points of the image frame in the vehicle coordinate system corresponding to the image frame.

[0091] The world coordinate transformation module 604 is used to transform each sampling point into the world coordinate system based on the ground three-dimensional coordinates of each sampling point to obtain the world coordinates of each sampling point;

[0092] The lane line position determination module 605 is used to obtain the position of the lane line in the world coordinate system based on the world coordinates of each sampling point.

[0093] In one possible implementation, the ground plane equation fitting module is specifically used to: for each image frame, classify the points in the radar point cloud frame corresponding to the image frame to obtain the ground radar points representing the ground corresponding to the image frame; and use the ground radar points corresponding to the image frame to perform plane fitting to obtain the ground plane equation in the vehicle coordinate system corresponding to the image frame.

[0094] In one possible implementation, the ground three-dimensional coordinate determination module is specifically used to: determine the abscissa and ordinate of the sampling point in the image frame in the vehicle coordinate system corresponding to the image frame based on the image coordinates of the sampling point in the image frame; substitute the abscissa and ordinate of the sampling point in the image frame in the vehicle coordinate system corresponding to the image frame into the ground plane equation corresponding to the image frame to obtain the ground three-dimensional coordinates of the sampling point in the image frame in the vehicle coordinate system corresponding to the image frame.

[0095] In one possible implementation, the world coordinate transformation module is specifically used to: for each sampling point, obtain the translation vector and rotation matrix determined when the sampling point was collected, and obtain the translation vector and rotation matrix corresponding to the sampling point; and use the translation vector and rotation matrix corresponding to the sampling point to transform the ground three-dimensional coordinates of the sampling point to the world coordinate system to obtain the world coordinates of the sampling point.

[0096] In one possible implementation, the world coordinate transformation module is specifically used for:

[0097] For each image frame, obtain the vehicle's pose in the world coordinate system when the radar point cloud frame corresponding to that image frame was acquired; based on the pose, determine the transformation relationship between the vehicle coordinate system and the world coordinate system corresponding to that image frame.

[0098] For each sampling point, based on the three-dimensional ground coordinates of the sampling point in the vehicle coordinate system, and with the constraint that the sampling point is in a plane perpendicular to the ground plane equation, the three-dimensional coordinates of the sampling point are calculated and used as the origin three-dimensional coordinates of the sampling point.

[0099] For each sampling point, the target translation vector of the sampling point is obtained by determining the translation vector from the origin of the vehicle coordinate system corresponding to the sampling point to the origin of the sampling point in three-dimensional coordinates.

[0100] Using the translation vector, target translation vector, and rotation matrix corresponding to the sampling point, the origin three-dimensional coordinates of the sampling point are transformed to the world coordinate system to obtain the world coordinates of the sampling point.

[0101] In one possible implementation, see Figure 7 The device further includes:

[0102] The fitting curve equation determination module 701 is used to calculate the fitting curve equation of each lane line in a specified image frame.

[0103] The 3D coordinate transformation determination module 702 is used to transform the ground 3D coordinates of each sampling point to the vehicle coordinate system corresponding to the specified image frame, so as to obtain the transformed 3D coordinates of each sampling point.

[0104] The lane line association module 703 is used to determine the lane line to which each sampling point belongs based on the transformed three-dimensional coordinates of each sampling point and the fitted curve equation of each lane line in the specified image frame.

[0105] In one possible implementation, the lane line association module is specifically used to: for each sampling point, calculate the distance between the sampling point and the fitted curve equation of each lane line in the image frame; if the minimum distance is less than a preset distance threshold, then determine that the sampling point belongs to the lane line corresponding to the minimum distance; if the minimum distance is not less than the preset distance threshold, then determine that the sampling point does not belong to any lane line in the specified image frame.

[0106] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0107] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0108] Among them, electronic devices include:

[0109] At least one processor; and

[0110] A memory communicatively connected to the at least one processor; wherein,

[0111] The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the lane line position determination method described in any one of the present applications.

[0112] A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to perform the lane line position determination method described in any one of the present application.

[0113] Figure 8A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0114] like Figure 8 As shown, device 800 includes a computing unit 801, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 802 or a computer program loaded from storage unit 808 into random access memory (RAM) 803. RAM 803 may also store various programs and data required for the operation of device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via bus 804. Input / output (I / O) interface 805 is also connected to bus 804.

[0115] Multiple components in device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of monitors, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0116] The computing unit 801 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as the lane line position determination method. For example, in some embodiments, the lane line position determination method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the lane line position determination method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the lane line position determination method by any other suitable means (e.g., by means of firmware).

[0117] This disclosure also provides an autonomous vehicle, including any of the electronic devices described in this application. The electronic device is used to execute the lane line position determination method described in any one of these applications, thereby enabling the autonomous vehicle to achieve autonomous driving using the position of lane lines in a world coordinate system.

[0118] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0119] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0120] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0121] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0122] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0123] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0124] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0125] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for determining lane line position, the method comprising: obtaining an image frame containing lane lines and a radar point cloud frame corresponding to the image frame; for each image frame, fitting a ground plane equation in a vehicle coordinate system corresponding to the image frame according to the radar point cloud frame corresponding to the image frame; for each image frame, sampling lane lines in the image frame to obtain sampling points of the image frame; projecting the sampling points of the image frame into the ground plane equation corresponding to the image frame to obtain ground three-dimensional coordinates of the sampling points of the image frame in the vehicle coordinate system corresponding to the image frame; converting the sampling points to world coordinates according to the ground three-dimensional coordinates of the sampling points to obtain world coordinates of the sampling points; obtaining a position of lane lines in the world coordinate system according to the world coordinates of the sampling points; the method further comprising: calculating a fitted curve equation of each lane line in a specified image frame; converting ground three-dimensional coordinates of lane line sampling points in a plurality of image frames adjacent to the specified image frame to the vehicle coordinate system corresponding to the specified image frame to obtain converted three-dimensional coordinates of the sampling points; for each sampling point in the plurality of image frames adjacent to the specified image frame, calculating distances of the sampling point from fitted curve equations of each lane line in the specified image frame based on the converted three-dimensional coordinates of the sampling point, if a minimum distance is less than a preset distance threshold, determining that the sampling point belongs to the lane line corresponding to the minimum distance, and if the minimum distance is not less than the preset distance threshold, determining that the sampling point does not belong to any lane line in the specified image frame. the method further comprising: for each image frame, classifying points in the radar point cloud frame corresponding to the image frame to obtain ground radar points representing the ground corresponding to the image frame; performing plane fitting using the ground radar points corresponding to the image frame to obtain the ground plane equation in the vehicle coordinate system corresponding to the image frame. the method further comprising: determining horizontal and vertical coordinates of the sampling points in the vehicle coordinate system corresponding to the image frame according to image coordinates of the sampling points in the image frame; substituting the horizontal and vertical coordinates of the sampling points in the vehicle coordinate system corresponding to the image frame into the ground plane equation corresponding to the image frame to obtain the ground three-dimensional coordinates of the sampling points in the vehicle coordinate system corresponding to the image frame. the method further comprising: for each sampling point, obtaining a translation vector and a rotation matrix determined when the sampling point is collected to obtain the translation vector and the rotation matrix corresponding to the sampling point; and converting the ground three-dimensional coordinates of the sampling point to the world coordinates of the sampling point using the translation vector and the rotation matrix corresponding to the sampling point. ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ 2. The method of claim 1, wherein, ​ ​ ​ 3. The method of claim 1, wherein, ​ ​ ​ 4. The method of claim 1, wherein, ​ ​ 5. The method of claim 4, wherein, The translation vector and the rotation matrix corresponding to each sampling point are obtained. The ground three-dimensional coordinates of the sampling point are converted into the world coordinate system by using the translation vector and the rotation matrix corresponding to the sampling point to obtain the world coordinates of the sampling point, including: For each image frame, the pose of the vehicle in the world coordinate system when collecting the radar point cloud frame corresponding to the image frame is obtained; and the conversion relationship between the vehicle coordinate system corresponding to the image frame and the world coordinate system is determined according to the pose. For each sampling point, the three-dimensional coordinates of the sampling point are calculated as the original three-dimensional coordinates of the sampling point by taking the ground three-dimensional coordinates of the sampling point in the vehicle coordinate system as the constraint condition and the sampling point in the plane perpendicular to the ground plane equation as the constraint condition. For each sampling point, the translation vector from the original point of the vehicle coordinate system corresponding to the sampling point to the original three-dimensional coordinates of the sampling point is determined to obtain the target translation vector of the sampling point. The original three-dimensional coordinates of the sampling point are converted into the world coordinate system by using the translation vector, the target translation vector and the rotation matrix corresponding to the sampling point to obtain the world coordinates of the sampling point.

6. A lane line position determination apparatus, the apparatus comprising: a related data acquisition module configured to acquire an image frame containing a lane line and a radar point cloud frame corresponding to the image frame; a ground plane equation fitting module configured to, for each image frame, fit a ground plane equation in a vehicle coordinate system corresponding to the image frame according to a radar point cloud frame corresponding to the image frame; a ground three-dimensional coordinate determination module configured to, for each image frame, sample lane lines in the image frame to obtain sampling points of the image frame; project the sampling points of the image frame into a ground plane equation corresponding to the image frame to obtain ground three-dimensional coordinates of the sampling points of the image frame in the vehicle coordinate system corresponding to the image frame; a world coordinate conversion module configured to convert each sampling point into a world coordinate system according to the ground three-dimensional coordinates of the sampling point to obtain the world coordinates of the sampling point; a lane line position determination module configured to obtain the position of the lane line in the world coordinate system according to the world coordinates of each sampling point; the apparatus further comprises: a fitting curve equation determination module configured to calculate fitting curve equations of each lane line in a specified image frame respectively; a converted three-dimensional coordinate determination module configured to convert the ground three-dimensional coordinates of the lane line sampling points in a plurality of image frames adjacent to the specified image frame into the vehicle coordinate system corresponding to the specified image frame to obtain converted three-dimensional coordinates of each sampling point; a lane line association module configured to, for each sampling point in the plurality of image frames adjacent to the specified image frame, calculate the distance of the sampling point from the fitting curve equations of each lane line in the specified image frame based on the converted three-dimensional coordinates of the sampling point, and if the minimum distance is less than a preset distance threshold, determine that the sampling point belongs to the lane line corresponding to the minimum distance; and if the minimum distance is not less than the preset distance threshold, determine that the sampling point does not belong to any lane line in the specified image frame.

7. The apparatus of claim 6, wherein, The ground plane equation fitting module is specifically configured to: for each image frame, classify points in a radar point cloud frame corresponding to the image frame to obtain ground radar points of the image frame; and perform plane fitting on the ground radar points of the image frame to obtain a ground plane equation in a vehicle coordinate system corresponding to the image frame.

8. The apparatus of claim 6, wherein, The ground three-dimensional coordinate determination module is specifically configured to: determine horizontal and vertical coordinates of a sampling point in the vehicle coordinate system corresponding to the image frame according to an image coordinate of the sampling point in the image frame; Substitute the horizontal and vertical coordinates of the sampling point in the vehicle coordinate system corresponding to the image frame into the ground plane equation corresponding to the image frame to obtain a ground three-dimensional coordinate of the sampling point in the vehicle coordinate system corresponding to the image frame.

9. The apparatus of claim 6, wherein, The world coordinate conversion module is specifically configured to: for each sampling point, obtain a translation vector and a rotation matrix determined when the sampling point is collected to obtain a translation vector and a rotation matrix corresponding to the sampling point; and convert the ground three-dimensional coordinate of the sampling point to a world coordinate system to obtain a world coordinate of the sampling point by using the translation vector and the rotation matrix corresponding to the sampling point.

10. The apparatus of claim 9, wherein, The world coordinate conversion module is specifically configured to: For each image frame, obtain a pose of the vehicle in the world coordinate system when a radar point cloud frame corresponding to the image frame is collected; and determine a conversion relationship between the vehicle coordinate system corresponding to the image frame and the world coordinate system according to the pose; For each sampling point, calculate a three-dimensional coordinate of the sampling point as an origin three-dimensional coordinate of the sampling point by taking the sampling point in a plane perpendicular to the ground plane equation as a constraint condition according to the ground three-dimensional coordinate of the sampling point in the vehicle coordinate system; For each sampling point, determine a target translation vector of the sampling point by determining a translation vector from an origin of the vehicle coordinate system corresponding to the sampling point to the origin three-dimensional coordinate of the sampling point; Convert the origin three-dimensional coordinate of the sampling point to the world coordinate system by using the translation vector, the target translation vector, and the rotation matrix corresponding to the sampling point to obtain the world coordinate of the sampling point. 11.An electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-5.

12. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to enable the computer to perform the method of any one of claims 1-5. 13.A computer program product comprising a computer program which, when executed by a processor, implements the method of any one of claims 1-5. 14.An autonomous vehicle comprising the electronic device of claim 11.

Images