Lane line processing method, device, equipment, medium and program product

Abstract

The invention relates to a lane line processing method, device and equipment, a medium and a program product, and the method comprises the steps: firstly, projecting lane line pixels to a vehicle body coordinate system according to a lane line image collected by an image collection device of a vehicle, and determining the first position information of each lane line; then, according to the first position information of each lane line, determining whether the lane line image contains left and right lane lines corresponding to the current lane; then, if the lane line image does not contain the left lane line and the right lane line, pitch angle deviation corresponding to each lane line pixel in the lane line image is determined according to the point cloud data of the road surface in front of the vehicle and the first position information of each lane line; and finally, according to the pitch angle deviation corresponding to each lane line pixel, re-projecting the lane line pixel to the vehicle body coordinate system, and determining second position information of each lane line. According to the invention, the output accuracy and stability of the lane line are improved, so that the safety of vehicle driving is ensured.

Description

Technical Field

[0001] The present invention relates to the field of lane line detection technology, and in particular to a lane line processing method, device, equipment, medium and program product. Background Art

[0002] With the rapid development of autonomous driving technology, lane information has become an important reference in autonomous driving systems. Lanes not only define the driving boundaries of the vehicle, but also provide key constraints for path planning, enabling the system to accurately determine the current lane and make reasonable driving decisions based on road conditions. Therefore, the processing of lane information plays an irreplaceable role in autonomous driving and is the basis for achieving safe and reliable driving.

[0003] In the prior art, the method for processing lane lines is usually to use an image acquisition device to scan the road surface and obtain the pixel information of the lane lines in the image; then, use the camera extrinsic parameters to convert it to the vehicle body coordinate system to obtain the vehicle body pixel coordinates; finally, use the curve fitting method to obtain the lane line information from the vehicle body pixel coordinates.

[0004] However, when processing lane lines in the prior art, projecting lane line pixel information onto vehicle body coordinates may result in distortion and instability, resulting in low accuracy of the output lane line information and making it difficult to ensure vehicle driving safety. Summary of the invention

[0005] One of the purposes of the present invention is to provide a lane line processing method, device, equipment, medium and program product to solve the problem in the prior art that the accuracy of outputting lane line information is low and it is difficult to ensure the safety of vehicle driving.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0007] A lane line processing method, comprising:

[0008] According to the lane line image acquired by the image acquisition device of the vehicle, the lane line pixels are projected into the vehicle body coordinate system to determine the first position information of each lane line;

[0009] Determining, based on the first position information of each lane line, whether the lane line image includes left and right lane lines corresponding to the current lane;

[0010] If the lane line image does not include the left and right lane lines, determining the pitch angle deviation corresponding to each lane line pixel in the lane line image according to the point cloud data of the road surface in front of the vehicle and the first position information of each lane line;

[0011] According to the pitch angle deviation corresponding to each lane line pixel, the lane line pixel is reprojected to the vehicle body coordinate system to determine the second position information of each lane line.

[0012] According to the above-mentioned technical means, the two sensors, image acquisition equipment and vehicle-mounted laser radar, are integrated. The vehicle-mounted laser radar is used to compensate for the situation that the pixel information collected by the image acquisition equipment has pitch angle deviation when the vehicle is on a slope. The vehicle can obtain more accurate and rich lane line information, which not only improves the accuracy of lane line output, but also provides more reliable support for vehicle path planning, lane keeping and other functions, thereby greatly improving the safety and decision-making accuracy of autonomous driving.

[0013] Further, determining the pitch angle deviation corresponding to each lane line pixel in the lane line image according to the point cloud data of the road surface in front of the vehicle and the first position information of each lane line includes:

[0014] Determining height information and pitch angle deviation information of the road surface in front of the vehicle according to the point cloud data of the road surface in front of the vehicle;

[0015] Determine a plurality of position points on the road surface in front of the vehicle along the vehicle's driving direction, and a three-dimensional coordinate of each position point in a vehicle body coordinate system according to the point cloud data of the road surface in front of the vehicle, the height information, and the pitch angle deviation information;

[0016] According to the three-dimensional coordinates of each position point in the vehicle body coordinate system, each position point is reversely projected into the lane line image to determine the pixel coordinates of each position point;

[0017] According to the pixel coordinates of each position point and the pixel coordinates of each lane line pixel, the pitch angle deviation corresponding to each lane line pixel is determined.

[0018] According to the above technical means, the height and pitch angle information of the road ahead are obtained through point cloud data, and this information is used to convert the coordinates in the vehicle coordinate system into the lane line image, and further determine the pitch angle deviation corresponding to each lane line pixel. The output accuracy of the lane line of the vehicle in complex road conditions such as ramps is improved, and the stability and safety of vehicle driving are improved.

[0019] Further, determining the height information and pitch angle deviation information of the road surface in front of the vehicle according to the point cloud data of the road surface in front of the vehicle includes:

[0020] Dividing the road surface in front of the vehicle into a plurality of sub-areas along the driving direction of the vehicle;

[0021] Performing sparse processing on the point cloud data of each sub-area according to a preset longitudinal interval to obtain the sparse point cloud data in each sub-area, wherein the longitudinal direction is the driving direction of the vehicle;

[0022] Determine the height information of the road surface in front of the vehicle according to the sparse point cloud data in each sub-area;

[0023] The pitch angle deviation information is determined according to the height information of the road surface in front of the vehicle.

[0024] According to the above technical means, by dividing the road surface in front of the vehicle into multiple sub-areas and sparsely processing the point cloud data of each area, the data volume can be streamlined and the processing efficiency of the point cloud data can be improved; further calculating the height information and pitch angle deviation of each area can effectively enhance the image's adaptability to changes in road slope, thereby improving the accuracy of outputting lane lines under complex road conditions.

[0025] Further, the step of reprojecting the lane line pixels to the vehicle body coordinate system according to the pitch angle deviation corresponding to each lane line pixel to determine the second position information of each lane line includes:

[0026] For each lane line pixel in each lane line, add the preset pitch angle in the preset rotation matrix to the pitch angle deviation corresponding to the lane line pixel to generate a target rotation matrix;

[0027] According to the target rotation matrix, the preset intrinsic parameter matrix, the preset extrinsic parameter matrix, the preset translation vector and the pixel position of the lane line pixel, the lane line pixel is reprojected to the vehicle body coordinate system to determine the second position information of each lane line.

[0028] According to the above technical means, the pitch angle deviation of the lane line pixel is combined with the preset pitch angle to correct the impact of road undulation or vehicle tilt on the lane line projection. This can improve the projection accuracy of the lane line in the vehicle body coordinate system, making the re-projection of the lane line closer to the actual road conditions.

[0029] Furthermore, the method further comprises:

[0030] If the lane line image includes the left and right lane lines, determining a reference width of the current lane formed by the left and right lane lines according to the first position information of the left and right lane lines;

[0031] According to the reference width and the lane line pixel coordinates corresponding to the left and right lane lines, the left and right lane lines are corrected to obtain third position information of the left and right lane lines in the image acquisition device coordinate system; wherein the width of the current lane represented by the third position information of the left and right lane lines is the reference width;

[0032] According to the third position information of the left and right lane lines, the lane line pixels corresponding to the left and right lane lines are converted to the vehicle body coordinate system to determine the fourth position information of the left and right lane lines.

[0033] According to the above technical means, the lane line position is calibrated multiple times. In particular, when the left and right lane lines are visible at the same time, the correction of the reference width and the lane line pixel coordinates can effectively cope with the influence of complex road conditions (such as slope changes or road curvature), thereby improving the robustness of lane line detection.

[0034] Furthermore, the method further comprises:

[0035] Determining the boundary line point cloud data of the road surface in front of the vehicle from the point cloud data of the road surface in front of the vehicle;

[0036] According to the fourth position information of the left and right lane lines and the boundary line point cloud data, determining whether the average distance between the left and right lane lines and the boundary line is less than a preset distance;

[0037] If the average distance is less than the preset distance, the fourth position information of the left and right lane lines is corrected according to the fourth position information of the left and right lane lines and the boundary line point cloud data to generate corrected fourth position information;

[0038] Among them, the lateral distance between the left and right lane lines indicated by the fourth position information and the vehicle is smaller than the lateral distance between the boundary line and the vehicle, and the lateral direction is a direction parallel to the ground and perpendicular to the driving direction of the vehicle.

[0039] According to the above technical means, the lane line position is compared with the road boundary line point cloud data, and correction is made when the distance is less than the preset value, reducing the possibility of deviation or erroneous adjustment caused by lane line errors and improving driving safety.

[0040] Further, the correcting the left and right lane lines according to the reference width and the lane line pixel coordinates corresponding to the left and right lane lines to obtain third position information of the left and right lane lines in the image acquisition device coordinate system includes:

[0041] For a first pixel point of the left lane line, determine a second pixel point in the right lane line that is closest to the first pixel point according to lane line pixel coordinates of the left and right lane lines;

[0042] According to the reference width, the optical center coordinates and the focal length coordinates of the image acquisition device, the pixel point coordinates of the first pixel point and the pixel point coordinates of the second pixel point are corrected to obtain the third position information of the left and right lane lines.

[0043] According to the above technical means, the pixel coordinates of the first pixel point and the pixel coordinates of the second pixel point are corrected by using the reference width, the optical center coordinates and the focal length coordinates of the image acquisition device, so as to correct the deformation of the lane line in the image caused by projection, make the corrected pixel coordinates more consistent with the actual position of the lane line under the vehicle body coordinates, and improve the output accuracy of the lane line.

[0044] Further, the fourth position information of the left and right lane lines is corrected according to the fourth position information of the left and right lane lines and the boundary line point cloud data to generate the corrected fourth position information, including:

[0045] Determine a target lane line position point in the left and right lane lines according to the fourth position information of the left and right lane lines and the boundary line point cloud data; the lateral distance between the target lane line position point and the vehicle is greater than the lateral distance between the adjacent lane line position point and the vehicle;

[0046] According to the fourth position information of the left and right lane lines, the lane line position points whose ordinates are greater than the target ordinates in the left and right lane lines are deleted to obtain corrected left and right lane lines, where the target ordinate is the ordinate of the target lane line position point;

[0047] According to the boundary line point cloud data, adding the boundary line position points whose ordinates are greater than the target ordinates to the corrected left and right lane lines to generate target left and right lane lines;

[0048] The position information of the target left and right lane lines in the vehicle body coordinate system is determined as the corrected fourth position information.

[0049] According to the above technical means, by using the fourth position information of the left and right lane lines and the boundary line point cloud data, the target lane line position points are accurately located and screened, and the left and right lane line data are corrected and updated based on these position points. This can effectively remove interference information and retain key position points that match the actual lane boundary, thereby improving the stability and accuracy of the lane line output and ensuring the accuracy of the vehicle's driving path.

[0050] Further, the lane line image acquired by the image acquisition device of the vehicle is projected onto the vehicle body coordinate system to determine the first position information of each lane line, including:

[0051] Determine the pixel coordinates of each lane line pixel according to the lane line image;

[0052] According to the preset rotation matrix, the preset intrinsic parameter matrix, the preset extrinsic parameter matrix, the preset translation vector and the first coordinate of each lane line pixel, each lane line pixel is projected into the vehicle body coordinate system to obtain the three-dimensional coordinate of each lane line pixel;

[0053] The first position information of each lane line is determined according to the three-dimensional coordinates of the lane line pixels contained in each lane line.

[0054] Further, determining the first position information of each lane line according to the three-dimensional coordinates of the lane line pixels contained in each lane line includes:

[0055] According to the second coordinates of the lane line pixels contained in each lane line, a cubic curve fitting is performed on each lane line to determine a position fitting curve of each lane line, and the position fitting curve is the first position information.

[0056] According to the above technical means, by performing cubic curve fitting on the second coordinate of the pixel point of each lane line, a position fitting curve is generated as the first position information of the lane line. This cubic curve fitting method can capture the continuous change characteristics of the lane line, improve the accurate description of the lane line position, effectively filter out noise and interference, and enhance the vehicle's perception of lane boundaries in complex road environments.

[0057] Further, determining whether the lane line image includes left and right lane lines corresponding to the current lane according to the first position information of each lane line includes:

[0058] According to the first position information of each lane line, two target lane lines are determined from all lane lines included in the lane line image; wherein the lateral distance between any target lane line and the vehicle is smaller than the lateral distance between other lane lines and the vehicle, and the lateral direction is a direction parallel to the ground and perpendicular to the driving direction of the vehicle;

[0059] Determining a lane width formed by the two target lane lines according to the first position information of the two target lane lines;

[0060] When the lane width is smaller than the preset lane width, and the lateral distance between any target lane line and the vehicle is smaller than the preset lateral distance, the two target lane lines are determined to be the left and right lane lines.

[0061] According to the above technical means, the two target lane lines with the closest lateral distance to the vehicle are accurately identified from all lane lines in the image, and the lane width between them is calculated. When the lane width is less than the preset threshold and the lateral distance between the target lane line and the vehicle is within the preset range, these two lane lines are marked as the left and right lane lines of the vehicle. This can effectively eliminate the interference of non-target lane lines and ensure accurate identification of left and right lane lines.

[0062] A lane line processing device, comprising:

[0063] A projection module, used to project lane line pixels to a vehicle body coordinate system based on a lane line image acquired by an image acquisition device of the vehicle, and determine first position information of each lane line;

[0064] A first determination module, used to determine whether the lane line image includes left and right lane lines corresponding to the current lane according to the first position information of each lane line;

[0065] A second determination module, if the lane line image does not include the left and right lane lines, determines the pitch angle deviation corresponding to each lane line pixel in the lane line image according to the point cloud data of the road surface in front of the vehicle and the first position information of each lane line;

[0066] The third determination module is used to reproject the lane line pixels to the vehicle body coordinate system according to the pitch angle deviation corresponding to each lane line pixel, so as to determine the second position information of each lane line.

[0067] Further, the second determining module is specifically used to:

[0068] Determining height information and pitch angle deviation information of the road surface in front of the vehicle according to the point cloud data of the road surface in front of the vehicle;

[0069] Determine a plurality of position points on the road surface in front of the vehicle along the vehicle's driving direction, and a three-dimensional coordinate of each position point in a vehicle body coordinate system according to the point cloud data of the road surface in front of the vehicle, the height information, and the pitch angle deviation information;

[0070] According to the three-dimensional coordinates of each position point in the vehicle body coordinate system, each position point is reversely projected into the lane line image to determine the pixel coordinates of each position point;

[0071] According to the pixel coordinates of each position point and the pixel coordinates of each lane line pixel, the pitch angle deviation corresponding to each lane line pixel is determined.

[0072] Further, the second determining module is specifically used to:

[0073] Dividing the road surface in front of the vehicle into a plurality of sub-areas along the driving direction of the vehicle;

[0074] Performing sparse processing on the point cloud data of each sub-area according to a preset longitudinal interval to obtain the sparse point cloud data in each sub-area, wherein the longitudinal direction is the driving direction of the vehicle;

[0075] Determine the height information of the road surface in front of the vehicle according to the sparse point cloud data in each sub-area;

[0076] The pitch angle deviation information is determined according to the height information of the road surface in front of the vehicle.

[0077] Furthermore, the lane line processing device further includes a processing module, which is specifically used to:

[0078] For each lane line pixel in each lane line, add the preset pitch angle in the preset rotation matrix to the pitch angle deviation corresponding to the lane line pixel to generate a target rotation matrix;

[0079] According to the target rotation matrix, the preset intrinsic parameter matrix, the preset extrinsic parameter matrix, the preset translation vector and the pixel position of the lane line pixel, the lane line pixel is reprojected to the vehicle body coordinate system to determine the second position information of each lane line.

[0080] Furthermore, the processing module is also used for:

[0081] If the lane line image includes the left and right lane lines, determining a reference width of the current lane formed by the left and right lane lines according to the first position information of the left and right lane lines;

[0082] According to the reference width and the lane line pixel coordinates corresponding to the left and right lane lines, the left and right lane lines are corrected to obtain third position information of the left and right lane lines in the image acquisition device coordinate system; wherein the width of the current lane represented by the third position information of the left and right lane lines is the reference width;

[0083] According to the third position information of the left and right lane lines, the lane line pixels corresponding to the left and right lane lines are converted to the vehicle body coordinate system to determine the fourth position information of the left and right lane lines.

[0084] Furthermore, the processing module is also used for:

[0085] Determining the boundary line point cloud data of the road surface in front of the vehicle from the point cloud data of the road surface in front of the vehicle;

[0086] According to the fourth position information of the left and right lane lines and the boundary line point cloud data, determining whether the average distance between the left and right lane lines and the boundary line is less than a preset distance;

[0087] If the average distance is less than the preset distance, the fourth position information of the left and right lane lines is corrected according to the fourth position information of the left and right lane lines and the boundary line point cloud data to generate corrected fourth position information;

[0088] Among them, the lateral distance between the left and right lane lines indicated by the fourth position information and the vehicle is smaller than the lateral distance between the boundary line and the vehicle, and the lateral direction is a direction parallel to the ground and perpendicular to the driving direction of the vehicle.

[0089] Furthermore, the processing module is specifically used for:

[0090] For a first pixel point of the left lane line, determine a second pixel point in the right lane line that is closest to the first pixel point according to lane line pixel coordinates of the left and right lane lines;

[0091] According to the reference width, the optical center coordinates and the focal length coordinates of the image acquisition device, the pixel point coordinates of the first pixel point and the pixel point coordinates of the second pixel point are corrected to obtain the third position information of the left and right lane lines.

[0092] Furthermore, the processing module is specifically used for:

[0093] Determine a target lane line position point in the left and right lane lines according to the fourth position information of the left and right lane lines and the boundary line point cloud data; the lateral distance between the target lane line position point and the vehicle is greater than the lateral distance between the adjacent lane line position point and the vehicle;

[0094] According to the fourth position information of the left and right lane lines, the lane line position points whose ordinates are greater than the target ordinates in the left and right lane lines are deleted to obtain corrected left and right lane lines, where the target ordinate is the ordinate of the target lane line position point;

[0095] According to the boundary line point cloud data, adding the boundary line position points whose ordinates are greater than the target ordinates to the corrected left and right lane lines to generate target left and right lane lines;

[0096] The position information of the target left and right lane lines in the vehicle body coordinate system is determined as the corrected fourth position information.

[0097] Furthermore, the projection module is specifically used for:

[0098] Determine the pixel coordinates of each lane line pixel according to the lane line image;

[0099] According to the preset rotation matrix, the preset intrinsic parameter matrix, the preset extrinsic parameter matrix, the preset translation vector and the first coordinate of each lane line pixel, each lane line pixel is projected into the vehicle body coordinate system to obtain the three-dimensional coordinate of each lane line pixel;

[0100] The first position information of each lane line is determined according to the three-dimensional coordinates of the lane line pixels contained in each lane line.

[0101] Furthermore, the processing module is specifically used for:

[0102] According to the second coordinates of the lane line pixels contained in each lane line, a cubic curve fitting is performed on each lane line to determine a position fitting curve of each lane line, and the position fitting curve is the first position information.

[0103] Further, the first determining module is specifically configured to:

[0104] According to the first position information of each lane line, two target lane lines are determined from all lane lines included in the lane line image; wherein the lateral distance between any target lane line and the vehicle is smaller than the lateral distance between other lane lines and the vehicle, and the lateral direction is a direction parallel to the ground and perpendicular to the driving direction of the vehicle;

[0105] Determining a lane width formed by the two target lane lines according to the first position information of the two target lane lines;

[0106] When the lane width is smaller than the preset lane width, and the lateral distance between any target lane line and the vehicle is smaller than the preset lateral distance, the two target lane lines are determined to be the left and right lane lines.

[0107] An electronic device includes: a processor, a memory, and computer-executable instructions stored in the memory and executable on the processor, wherein the processor is used to implement the lane line processing method when executing the computer-executable instructions.

[0108] A computer-readable storage medium, characterized in that the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to implement the above-mentioned lane line processing method when executed by a processor.

[0109] A computer program product includes a computer program, wherein the computer program is used to implement the above-mentioned lane line processing method when executed by a processor.

[0110] Beneficial effects of the present invention:

[0111] (1) Since the lane line information obtained by the image acquisition device is easily affected by many factors, resulting in low accuracy of the output lane line, this technical solution will combine the image acquisition device and the on-board laser radar. The on-board laser radar can use the collected point cloud data to determine the height of the road ahead and the pitch angle deviation information of the image acquisition device in complex road sections to compensate for the problem of missing lane line information due to improper image acquisition angle of the image acquisition device, thereby improving the accuracy of lane line output.

[0112] (2) After determining that the left and right lane lines corresponding to the current lane are included in the lane line image, the present invention utilizes a variety of corresponding correction methods, including lane line pixel coordinates corresponding to the reference width and the left and right lane lines, and utilizing boundary line point cloud data pairs. This can effectively eliminate the effects of image acquisition equipment, lane line projection errors, and changes in the road environment on lane line detection, thereby improving the output accuracy of the lane lines and further ensuring the safety of vehicle driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0113] Figure 1 Schematic diagram of the process of the lane line processing method provided by the embodiment of the present invention Figure 1 ;

[0114] Figure 2 Schematic diagram of the process of the lane line processing method provided by the embodiment of the present invention Figure 2 ;

[0115] Figure 3 A schematic diagram of the slope of the pitch angle deviation corresponding to the lane line pixel provided in an embodiment of the present invention;

[0116] Figure 4 Schematic diagram of the process of the lane line processing method provided by the embodiment of the present invention Figure 3 ;

[0117] Figure 5 Schematic diagram of the process of the lane line processing method provided by the embodiment of the present invention Figure 4 ;

[0118] Figure 6 Schematic diagram of the process of the lane line processing method provided by the embodiment of the present invention Figure 5 ;

[0119] Figure 7 Schematic diagram of the process of the lane line processing method provided by the embodiment of the present invention Figure 6 ;

[0120] Figure 8 Schematic diagram of the process of the lane line processing method provided by the embodiment of the present invention Figure 7 ;

[0121] Fig. 9 A schematic diagram of the structure of a lane line processing device provided in an embodiment of the present invention;

[0122] Fig.10 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0123] The following will describe the embodiments of the present invention with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention, not for limiting the scope of protection of the present invention.

[0124] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the present invention are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and corresponding operation entrances shall be provided for users to choose to authorize or refuse.

[0125] Before introducing the present invention, the application background of the present invention is first explained.

[0126] With the rapid development of autonomous driving technology, lane line information has become an important reference in autonomous driving systems. In practical applications, lane lines provide key driving guidance for drivers and autonomous driving systems by drawing different forms of lines on the road. These lines are not only used to divide lanes, but also indicate the driving direction and relative position of vehicles, and provide various traffic information, such as no lane change, overtaking allowed, etc. Lane line information helps maintain the order and safety of road traffic, and at the same time provides clear constraints for vehicle driving under complex road conditions. Therefore, the processing of lane line information plays an irreplaceable role in autonomous driving.

[0127] Accordingly, accurate processing of lane line information is the basis for ensuring safe and reliable driving of vehicles. If lane line processing is deviated or misidentified, the autonomous driving system may make a decision to deviate from the driving path based on the wrong lane information, causing the vehicle to accidentally deviate from the lane, thereby increasing the risk of traffic accidents. Therefore, it is crucial to improve the accuracy and stability of lane line processing and provide real-time reminders to drivers when lane lines are abnormal.

[0128] In the prior art, the method for processing lane lines is usually to use an image acquisition device to scan the road surface and obtain the pixel information of the lane lines in the image; then, use the camera extrinsic parameters to convert it to the vehicle body coordinate system to obtain the vehicle body pixel coordinates; finally, use the curve fitting method to obtain the lane line information from the vehicle body pixel coordinates.

[0129] However, since the data acquired is the lane line image captured by the image acquisition device, this leads to stricter requirements on the installation location, road conditions and weather of the image acquisition device. For example, when going uphill or downhill, the acquisition angle of the image acquisition device changes, resulting in the inability to detect the complete lane line information around the vehicle, which causes large errors in the coordinates projected onto the vehicle body coordinates, making it difficult to ensure the safety of the vehicle's driving and affecting the user experience.

[0130] In summary, the existing technology has the problems of low accuracy and poor user experience when processing lane lines.

[0131] Based on the above technical problems, the present invention provides a lane line processing method. Considering that the lane line information obtained by the image acquisition device is easily affected by many factors, resulting in low accuracy of the output lane line, the on-board laser radar can determine the height of the road ahead through the collected point cloud data, and determine the pitch angle deviation information of the image acquisition device in complex road sections to make up for the problem of missing lane line information due to improper image acquisition angle of the image acquisition device. Therefore, by fusing the lane line information collected by the two sensors, the on-board laser radar and the on-board camera (also called the front-view camera), the accuracy of the output lane line can be greatly improved, and more stable support is provided for the path planning and safe driving of autonomous driving vehicles. Specifically, since the on-board camera has a wide field of view, it can obtain lane line information at a longer distance, provide more data points for the fitted curve, and the amount of calculation is small. Therefore, first, the lane line pixels can be projected to the vehicle body coordinate system according to the lane line image collected by the on-board camera (i.e., the image acquisition device), and the first position information of all lane lines of the current road contained in the lane line image can be determined; then, the main lane line corresponding to the current vehicle is determined using the first position information of each lane line, that is, the left and right lane lines of the lane where the current vehicle is sitting; further, if the main lane line corresponding to the vehicle is not determined by the first position information of each lane line, it means that the current road may have a certain slope, resulting in the on-board camera not collecting complete lane line information, and it is necessary to use the on-board laser radar for auxiliary determination, that is, according to the point cloud data of the road surface in front of the vehicle collected by the on-board laser radar and the first position information of each lane line, determine the pitch angle deviation corresponding to each lane line pixel in the lane line image; finally, according to the pitch angle deviation corresponding to each lane line pixel, the lane line pixel is re-projected to the vehicle body coordinate system to determine the second position information of each lane line. Through this method, the autonomous driving system can obtain more accurate and rich lane line information, which not only improves the accuracy of lane line output, but also provides more reliable support for vehicle path planning, lane keeping and other functions, greatly improving the safety and decision-making accuracy of autonomous driving.

[0132] The technical solution of the present invention is described in detail below through specific embodiments.

[0133] It should be noted that the following specific embodiments may be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0134] Before introducing the specific embodiments, it is necessary to uniformly define the coordinate system involved in the embodiments of the present application: the body coordinate system takes the ground at the center of the rear axle of the vehicle as the origin, the forward direction of the vehicle as the positive direction of the X axis, the Y direction perpendicular to the left side of the vehicle and the upward direction perpendicular to the ground as the positive direction of the Z axis. The camera coordinate system takes the optical center of the camera as the origin, the Z direction parallel to the optical axis and outward, the X direction perpendicular to the optical axis and downward, and the Y direction perpendicular to the optical axis and to the left.

[0135] Figure 1 Schematic diagram of the process of the lane line processing method provided by the embodiment of the present invention Figure 1 .like Figure 1 As shown, the lane line processing method may include the following steps:

[0136] S11. According to the lane line image acquired by the image acquisition device of the vehicle, the lane line pixels are projected into the vehicle body coordinate system to determine the first position information of each lane line.

[0137] The lane line pixels are obtained by semantic segmentation of the lane line image acquired by the vehicle's image acquisition device, including a set of points corresponding to each lane line.

[0138] It can be understood that by combining the lane line information with the vehicle body coordinate system, the position of each lane line in the road can be determined more accurately, providing a data basis for the subsequent determination of the left and right lane lines of the vehicle.

[0139] S12. Determine, based on the first position information of each lane line, whether the lane line image includes left and right lane lines corresponding to the current lane.

[0140] It is understandable that after obtaining the first position information of each lane line, it is also necessary to determine the position of the lane line relative to the vehicle on the road and determine the left and right lane lines corresponding to the vehicle, so as to perform further lane line optimization later.

[0141] In one achievable method, the specific steps of determining whether the lane line image contains the left and right lane lines corresponding to the current lane are:

[0142] First, according to the first position information of each lane line, two target lane lines are determined from all lane lines contained in the lane line image; then, according to the first position information of the two target lane lines, the lane width formed by the two target lane lines is determined; finally, when the lane width is less than the preset lane width and the lateral distance between any target lane line and the vehicle is less than the preset lateral distance, the two target lane lines are determined to be the left and right lane lines.

[0143] Among them, the lateral distance between any target lane line and the vehicle is smaller than the lateral distance between other lane lines and the vehicle, and the lateral direction is a direction parallel to the ground and perpendicular to the driving direction of the vehicle.

[0144] It should be noted that if the lane width formed by the two target lane lines determined based on the first position information of the two target lane lines is greater than the preset lane width, or the lateral distance between any target lane line and the vehicle is greater than the preset lateral distance, then it is determined that the lane line image does not include the left and right lane lines corresponding to the current lane.

[0145] It can be understood that by selecting the target lane line based on the first position information of each lane line and verifying whether their spacing and lateral distance from the vehicle meet the preset conditions, the positions of the left and right lane lines can be accurately identified, and distant or irrelevant lane line information can be effectively filtered out to ensure the accuracy and reliability of lane line recognition.

[0146] S13: If the lane line image does not include left and right lane lines, determine the pitch angle deviation corresponding to each lane line pixel in the lane line image according to the point cloud data of the road surface in front of the vehicle and the first position information of each lane line.

[0147] The pitch angle deviation corresponding to the lane line pixel refers to the deviation in the position of the lane line pixel in the image caused by the pitch angle error of the device angle when the image acquisition device (such as a camera) shoots the lane line.

[0148] It should be noted that when the lane line image determined by the lane line fitting curve does not contain the left and right lane lines, it indicates that the vehicle may be on a certain slope, causing the pitch angle of the vehicle body relative to the horizontal plane to change, and the image collected by the image acquisition device is distorted. Therefore, it is necessary to determine the pitch angle deviation corresponding to each lane line pixel in the lane line image, so as to improve the accuracy of the lane line output and the safety of vehicle driving.

[0149] S14. Reproject the lane line pixels to the vehicle body coordinate system according to the pitch angle deviation corresponding to each lane line pixel, and determine the second position information of each lane line.

[0150] It can be understood that according to the pitch angle deviation corresponding to each lane line pixel, the lane line pixel is re-projected to the vehicle body coordinate system, and then the second position information of each lane line is finally determined by fitting the curve again. This can effectively eliminate the error caused by the angle change of the image acquisition device due to the road slope, make the projection of the lane line in the vehicle body coordinate system more accurate, and can significantly improve the accuracy of the lane line output.

[0151] The lane line processing method provided by the embodiment of the present invention can firstly project the lane line pixels to the vehicle body coordinate system according to the lane line image collected by the vehicle-mounted camera, and determine the first position information of all lane lines of the current road contained in the lane line image; then, use the first position information of each lane line to determine the main lane line corresponding to the current vehicle, that is, the left and right lane lines; further, if the main lane line corresponding to the vehicle is not determined by the first position information of each lane line, it means that the current road may have a certain slope, resulting in the vehicle-mounted camera not collecting complete lane line information, and it is necessary to use the on-board laser radar for auxiliary determination, that is, according to the point cloud data of the road surface in front of the vehicle collected by the on-board laser radar and the first position information of each lane line, determine the pitch angle deviation corresponding to each lane line pixel in the lane line image; finally, according to the pitch angle deviation corresponding to each lane line pixel, re-project the lane line pixel to the vehicle body coordinate system to determine the second position information of each lane line. Through this method, the vehicle can obtain more accurate and rich lane line information, which not only improves the accuracy of lane line output, but also provides more reliable support for vehicle path planning, lane keeping and other functions, greatly improving the safety and decision-making accuracy of autonomous driving.

[0152] In one achievable manner, the specific manner of determining the first position information of each lane line is:

[0153] First, according to the lane line image, the pixel coordinates of each lane line pixel are determined; then, according to the preset rotation matrix, the preset intrinsic parameter matrix, the preset extrinsic parameter matrix, the preset translation vector and the first coordinate of each lane line pixel, each lane line pixel is projected into the vehicle body coordinate system to obtain the three-dimensional coordinates of each lane line pixel; finally, according to the three-dimensional coordinates of the lane line pixels contained in each lane line, the first position information of each lane line is determined.

[0154] It should be understood that the projection method of the lane line pixels to the vehicle body coordinate system is based on the camera calibration internal and external parameters, where the internal parameter matrix is ​​K, and the external parameters include the rotation matrix R and the translation vector T. For example, if the pixel coordinates to be projected are (u, v), the internal parameter matrix K and the rotation matrix R are:

[0155]

[0156] r 11 =cos(pitch)*cos(yaw)

[0157] r 12 =cos(yaw)*sin(roll)*sin(pitch)+cos(roll)*sin(yaw)

[0158] r 13=sin(roll)*sin(yaw)-cos(roll)*cos(yaw)*sin(pitch)

[0159] r 21 =-cos(pitch)*sin(yaw)

[0160] r 22 =-sin(roll)*sin(pitch)*sin(yaw)+cos(roll)*cos(yaw)

[0161] r 23 =cos(roll)*sin(pitch)*sin(yaw)+cos(yaw)*sin(roll)

[0162] r 31 = sin(pitch)

[0163] r 32 =-cos(pitch)*sin(roll)

[0164] r 33 =cos(roll)*cos(pitch)

[0165] T=[t 11 t 12 t 13 ]

[0166] Among them, f u and f v is the focal length of the camera, c u and c v is the optical center of the camera, pitch, roll, and yaw are the pitch angle, roll angle, and yaw angle of the image acquisition device, respectively. (t 11 , t 12 , t 13 ) is the installation position of the image acquisition device relative to the vehicle coordinate origin, all of which are obtained through calibration.

[0167] Correspondingly, the formula for converting pixel coordinates to vehicle body coordinates is:

[0168]

[0169] Among them, (x, y, 1) is the converted vehicle body coordinate; s is the normalization coefficient; (u, v) is the pixel coordinate to be projected.

[0170] Furthermore, the specific process of determining the first position information of each lane line according to the three-dimensional coordinates of the lane line pixels contained in each lane line includes:

[0171] According to the second coordinates of the lane line pixels contained in each lane line, a cubic curve fitting is performed on each lane line to determine the position fitting curve of each lane line, where the position fitting curve is the first position information.

[0172] It should be noted that after determining the vehicle coordinates corresponding to all lane line pixels, the least squares method can be used to perform cubic curve fitting on a set of vehicle coordinates corresponding to each lane line. The equation of each lane line after fitting can be:

[0173] y=C 0 +C 1 x+C 2 x 2 +C 3 x 3

[0174] Among them, (x, y) is the horizontal coordinate and vertical coordinate of the vehicle body of each lane line; C 0 , C 1 , C 2 , C 3 All are fitting coefficients.

[0175] Since the coefficient C in the lane line fitting curve 0 Indicates the lateral distance of the lane line relative to the vehicle, so the coefficient C in the fitting curve determined in S11 can be 0 Arrange them from small to large, and then select C 0 The two lane lines with the smallest absolute value are the target lane lines. Then, the lane width formed by the two target lane lines is determined using the first position information of the two target lane lines. If the obtained lane width is less than the preset lane width, and the lateral distance between any of the two target lane lines and the vehicle is less than the preset lateral distance, the two target lane lines are determined to be the left and right lane lines. The specific lane width calculation formula is:

[0176] w=|C 0l -C 0r |

[0177] Where w is the lane width, C 0l C is the left lane line 0 , C 0r C is the right lane line 0 .

[0178] For example, if the coefficient C corresponding to the selected target lane line 0They are 2.7 and -2.1 respectively, the preset lane width is 5.2, and the preset lateral distance is 3. The selected target lane line corresponds to the target lane of 4.8, which is less than the preset lane width of 5.2. The lateral distance between any of the two target lane lines and the vehicle is less than the preset lateral distance of 3.0, indicating that the selected target lane line is the left or right lane line.

[0179] It should be understood that by choosing the fitting curve coefficient C 0 The two lane lines with the smallest absolute values ​​are taken as the target lane lines, and their spacing and lateral distance from the vehicle are verified to see if they meet the preset conditions. This can accurately identify the positions of the left and right lane lines, and effectively filter out distant or irrelevant lane line information, ensuring the accuracy and reliability of lane line recognition.

[0180] It can be understood that by converting pixel coordinates to the vehicle coordinate system, the distortion and error caused by the camera's perspective are eliminated, so as to obtain the real position of the lane line in the vehicle coordinate system. Then, the cubic curve fitting method (i.e., the least squares method) can effectively suppress the influence of noise and have a certain anti-interference ability for possible image acquisition errors. Even if there are a small amount of errors or occlusions in the image acquisition, the fitting curve can still better reflect the overall shape of the lane line and provide more accurate position lane line information for the vehicle.

[0181] Figure 2 Schematic diagram of the process of the lane line processing method provided by the embodiment of the present invention Figure 2 .like Figure 2 As shown, this embodiment, based on the above embodiment, describes in detail the specific process of determining the pitch angle deviation corresponding to each lane line pixel in the lane line image, including the following steps:

[0182] S21. Determine height information and pitch angle deviation information of the road surface in front of the vehicle based on the point cloud data of the road surface in front of the vehicle.

[0183] It should be understood that when the vehicle is on a certain slope, the vehicle's image acquisition equipment will produce a certain pitch angle error, and it is necessary to use the point cloud data of the road surface collected by the lidar for further correction to further improve the position information of the lane line.

[0184] In one feasible method, since the LiDAR road point cloud data is dense near the end and sparse far the end, in order to reduce the amount of calculation, the road point cloud needs to be sparsely processed, specifically including:

[0185] First, the road surface in front of the vehicle is divided into multiple sub-areas along the vehicle's driving direction; then, the point cloud data of each sub-area is sparsely processed according to the preset longitudinal interval to obtain the sparse point cloud data in each sub-area, with the longitudinal direction being the vehicle's driving direction; then, the height information of the road surface in front of the vehicle is determined based on the sparse point cloud data in each sub-area; finally, the pitch angle deviation information is determined based on the height information of the road surface in front of the vehicle.

[0186] Among them, sparse processing refers to extracting efficient and sparse feature data from the data to reduce the amount of data calculation and improve data processing efficiency.

[0187] Taking the process of sparse processing of LiDAR road surface point cloud data as an example, assuming that the range of 5 meters to 40 meters in front of the vehicle and -6 meters to 6 meters horizontally is taken as the effective area, it is first necessary to filter out all other road surface point cloud data outside the effective area in the LiDAR road surface point cloud data; then, the effective area is divided into multiple elongated horizontal sub-areas, each sub-area is 0.2 meters wide, and a total of 60 horizontal sub-areas are divided, and the road surface point cloud in each sub-area is sparse, starting from 5 meters in the longitudinal direction, taking a point every 1 meter until the farthest end of the sub-area, that is, each sub-area includes 35 point cloud data, and the road surface effective area includes 60*35 point cloud data.

[0188] It should be understood that after obtaining the sparse point cloud data in each sub-area, it is also necessary to determine the height information of the road surface in front of the vehicle to help identify the slope of the road surface in front. The specific calculation method can be:

[0189] For each sub-area, at intervals of 1 meter, first calculate the average height of each distance point in the longitudinal direction. If the current longitudinal distance to be calculated is y m , then take the vertical distance in each sub-area as y m The vertical heights of the points are summed and averaged. The specific calculation formula can be:

[0190]

[0191] Where N is the number of sub-regions, h avg_m The vertical distance is y m The average vertical height, z nm The vertical distance y in the nth sub-area m The height of the point.

[0192] Afterwards, the pitch angle deviation information is determined using the calculated height information of the road surface in front of the vehicle according to the pitch angle deviation formula. The specific pitch angle deviation formula can be:

[0193]

[0194] Among them, dpitch m The vertical distance is y m Pitch angle deviation information, y m is the vertical distance.

[0195] At this point, the pitch angle deviation of the point cloud data at every 1 meter in the longitudinal direction in each sub-area is calculated, and the calculation result is saved so that the lane line pixels can be further processed using the pitch angle deviation information later.

[0196] It is understandable that by sparsely processing the road point cloud data of the lidar and calculating the pitch angle deviation, it is helpful to improve the accuracy of the lane line perception output and ensure the safety and stability of vehicle driving.

[0197] S22. Determine a plurality of position points on the road surface in front of the vehicle along the vehicle's driving direction, and the three-dimensional coordinates of each position point in the vehicle body coordinate system, based on the point cloud data, height information, and pitch angle deviation information of the road surface in front of the vehicle.

[0198] Continuing with the above example, if the segment longitudinal distance to be calculated is y m , and the average height of each distance point in the longitudinal distance is h avg-m , then the three-dimensional coordinates of this position point in the vehicle coordinate system are (0, y m ,h avg-m ).

[0199] It should be understood that based on the sparse point cloud data and corresponding height information in each sub-area obtained in S21, multiple position points along the vehicle's driving direction on the road ahead of the vehicle and the three-dimensional coordinates of each position point in the vehicle body coordinate system can be obtained.

[0200] S23. According to the three-dimensional coordinates of each position point in the vehicle body coordinate system, each position point is reversely projected into the lane line image to determine the pixel coordinates of each position point.

[0201] Among them, inverse projection refers to the process of converting vehicle body coordinates into pixel coordinates, that is, the process of converting three-dimensional information into two-dimensional information.

[0202] Specifically, the specific method of inverse projection can be:

[0203]

[0204] Among them, u m and v m is the pixel coordinate after reverse projection, and the other parameter values ​​are in Figure 1 The detailed description is given in the embodiments, and this embodiment will not be repeated here.

[0205] It is understandable that when the vehicle is on a slope, due to the deviation of the pitch angle, it is necessary to convert the vehicle body coordinates into pixel coordinates and then correct the pixel coordinates using the pitch angle deviation to avoid the problem of decreased accuracy of lane line output due to image deformation.

[0206] S24. Determine the pitch angle deviation corresponding to each lane line pixel according to the pixel coordinates of each position point and the pixel coordinates of each lane line pixel.

[0207] For example, if the pixel coordinates of a position point after reverse projection are (u m , v m ), the pixel coordinates of the lane line pixel are (p, q), if The corresponding pitch angle deviation is dpitch m ,like The corresponding pitch angle deviation is dpitch m+1 ,like The corresponding pitch angle error is dpitch m .

[0208] It should be noted that when determining the pitch angle deviation (dpitch m ), it is also necessary to reproject the pixel coordinates of the lane line pixels to determine the second position information of each lane line. The specific steps include:

[0209] Firstly, for each lane line pixel in each lane line, the preset pitch angle in the preset rotation matrix is ​​added to the pitch angle deviation corresponding to the lane line pixel to generate a target rotation matrix; then, according to the target rotation matrix, the preset intrinsic parameter matrix, the preset extrinsic parameter matrix, the preset translation vector and the pixel position of the lane line pixel, the lane line pixel is reprojected to the vehicle body coordinate system to determine the second position information of each lane line.

[0210] It should be understood that the pitch angle error (dpitch m ) updates the camera's pitch, and then uses the updated camera pitch new ) Figure 1 The rotation matrix R in the embodiment is obtained by updating the rotation matrix R new Finally, using Figure 1 The pixel coordinates in the embodiment are converted into the vehicle body coordinate formula to complete the re-projection correction of the lane line pixel coordinates. The specific camera pitch angle update formula is:

[0211] pitch new =pitch+dpitch m

[0212] Among them, pitch new is the updated camera pitch angle, pitch is the original camera pitch angle, dpitch m is the pitch angle deviation corresponding to the lane line pixel.

[0213] It should be noted that the specific process of reprojecting the lane line pixels to the vehicle body coordinate system according to the target rotation matrix, the preset intrinsic matrix, the preset extrinsic matrix, the preset translation vector and the pixel position of the lane line pixels is the same as Figure 1 The projection conversion process in the embodiment is similar and will not be described in detail in this embodiment.

[0214] It can be understood that the pitch angle deviation corresponding to each lane line pixel is determined based on the pixel coordinates of each position point and the pixel coordinates of each lane line pixel. Then, the pitch angle of the camera is recalculated using the pitch angle deviation, and the pixel coordinates of the lane line pixels are reprojected to obtain the reprojection correction of the pixel coordinates of the lane line pixels. This can effectively eliminate the projection error caused by changes in vehicle posture, so as to quickly adapt to changes in the road surface and reduce the error in lane line output.

[0215] Figure 3 A schematic diagram of the slope of the pitch angle deviation corresponding to the lane line pixel provided in an embodiment of the present invention. Figure 3 As shown, the coordinate system of the image acquisition device takes the optical center of the device as the origin, and the optical axis direction is parallel to the optical center and outward. The positions of the vertical coordinates of the pixel coordinates of different lane line pixels are different, and the corresponding pitch angle deviations are also different. The specific method for determining the pitch angle deviation is described in detail in the embodiment of S24, and this embodiment will not be repeated here.

[0216] Figure 4 Schematic diagram of the process of the lane line processing method provided by the embodiment of the present invention Figure 3 .like Figure 4 As shown, in this embodiment Figure 1 Based on the embodiment, the specific process of the method for processing the lane line image containing left and right lane lines is described in detail, including the following steps:

[0217] S41. If the lane line image includes left and right lane lines, determine a reference width of a current lane formed by the left and right lane lines according to first position information of the left and right lane lines.

[0218] Among them, the reference width refers to the lane width that the vehicle needs to maintain under normal driving conditions.

[0219] For example, if the lane width is assumed to be constant, the position 10 meters in front of the vehicle is taken as the reference, and the fitting curves corresponding to the first position information of the left and right lane lines are yl ,y r , then the reference width of the current lane formed by the left and right lane lines is:

[0220] y l =C 0l +C 1l x+C 2l x 2 +C 3l x 3

[0221] y r =C 0r +C 1r x+C 2r x 2 +C 3r x 3

[0222] W=|C 0l -C 0r +10(C 1l -C 1r )+100(C 2l -C 2r )+1000(C 3l -C 3r )|

[0223] Among them, y l is the fitting curve corresponding to the first position information of the left lane line; r is the fitting curve corresponding to the first position information of the right lane line; W is the reference width.

[0224] It can be understood that when the lane line image contains left and right lane lines, the reference width of the current lane determined by the first position information of the left and right lane lines can be used for subsequent correction processing of the lane line pixel coordinates, thereby reducing the output error of the lane line. In addition, in different road environments, the reference width can be used as a basis for dynamic correction, thereby increasing the adaptability of the lane line processing method.

[0225] S42. Correct the left and right lane lines according to the reference width and the lane line pixel coordinates corresponding to the left and right lane lines, and obtain third position information of the left and right lane lines in the coordinate system of the image acquisition device.

[0226] In one achievable method, the specific process of correcting the left and right lane lines according to the reference width and the lane line pixel coordinates corresponding to the left and right lane lines is as follows:

[0227] First, for the first pixel point of the left lane line, the second pixel point in the right lane line that is closest to the first pixel point is determined according to the lane line pixel coordinates of the left and right lane lines; then, according to the reference width, the optical center coordinates and the focal length coordinates of the image acquisition device, the pixel coordinates of the first pixel point and the pixel coordinates of the second pixel point are corrected to obtain the third position information of the left and right lane lines.

[0228] Among them, the width of the current lane represented by the third position information of the left and right lane lines is the reference width.

[0229] For example, if the pixel coordinates of the first pixel in the left lane are (p l ,q l ), the pixel coordinate set of all the second pixel points of the right lane line is {(p r1 ,q r1 ), (p r2 ,q r2 )…(p rn ,q rn )}, then calculate the pixel distance d between the pixel coordinates of the first pixel and the pixel coordinates of each second pixel of the right lane line n , and finally select the pixel distance d n The pixel coordinates of the second pixel of the right lane line corresponding to the minimum value (p rn ,q rn ) as the corresponding point of the pixel coordinates of the first pixel, and stored as a point pair {(p l ,q l ), (p r ,q r )}, the specific pixel distance calculation formula can be:

[0230]

[0231] Among them, d n is the pixel distance between the pixel coordinates of the first pixel and the pixel coordinates of each second pixel of the right lane line; (p l ,q l ) The pixel coordinates of the first pixel in the left lane line; (p r ,q r ) is the pixel coordinate of the first pixel in the right lane line.

[0232] Furthermore, the pixel coordinates of the first pixel and the second pixel in the image acquisition device coordinate system are recalculated according to the reference width, the optical center coordinates and the focal length coordinates of the image acquisition device, and the image acquisition device coordinates are converted to the vehicle body coordinate system to obtain the third position information of the left and right lane lines, and finally the pixel coordinates of the first pixel and the second pixel are corrected. Specifically, if the pixel coordinates of the first pixel are (p l ,q l ), the pixel coordinates of the second pixel are (p r ,q r ), the correction process using the reference width W, optical center coordinates and focal length coordinates is:

[0233]

[0234] Among them, cam x is the position of the horizontal coordinate of the pixel point of the first pixel point after correction in the coordinate system of the image acquisition device; cam y is the position of the vertical coordinate of the first pixel after correction in the coordinate system of the image acquisition device; u is the focal length of the camera; c u The optical center of the camera.

[0235] It should be understood that when using the reference width to correct the left and right lane lines, it is first necessary to determine the shortest pixel distance point pair of the left and right lane lines, that is, the first pixel point of the left lane line and the second pixel point of the right lane line that is closest to the first pixel point; then, using the reference width, the optical center coordinates and the focal length coordinates of the image acquisition device, the pixel point coordinates of the first pixel point and the pixel point coordinates of the second pixel point are corrected, which can correct the deformation of the lane line in the image caused by projection, so that the corrected pixel coordinates are more consistent with the actual position of the lane line under the vehicle body coordinates, thereby improving the output accuracy of the lane line.

[0236] S43. According to the third position information of the left and right lane lines, the lane line pixels corresponding to the left and right lane lines are converted to the vehicle body coordinate system to determine the fourth position information of the left and right lane lines.

[0237] It should be noted that after obtaining the third position information of the left and right lane lines, the lane line pixels corresponding to the left and right lane lines need to be converted to the vehicle body coordinate system. The specific conversion formula is:

[0238]

[0239] Among them, cam x is the position of the horizontal coordinate of the pixel point of the first pixel point after correction in the coordinate system of the image acquisition device; cam yvcs is the position of the vertical coordinate of the first pixel after correction in the coordinate system of the image acquisition device; x vcs is the position of the horizontal coordinate of the first pixel after correction in the vehicle coordinate system; y is the vertical coordinate of the first pixel after correction in the vehicle coordinate system; the other parameter values ​​are in Figure 1 The detailed description is given in the embodiments, and this embodiment will not be repeated here.

[0240] Optionally, after determining the fourth position information of the left and right lane lines, in order to ensure the accuracy of the output lane lines, the fourth position information needs to be further corrected using the boundary line point cloud data. The specific process includes:

[0241] First, the boundary line point cloud data of the road surface in front of the vehicle is determined from the point cloud data of the road surface in front of the vehicle; then, based on the fourth position information of the left and right lane lines and the boundary line point cloud data, it is determined whether the average distance between the left and right lane lines and the boundary line is less than a preset distance; finally, if the average distance is less than the preset distance, the fourth position information of the left and right lane lines is corrected based on the fourth position information of the left and right lane lines and the boundary line point cloud data to generate corrected fourth position information.

[0242] Among them, the lateral distance between the left and right lane lines and the vehicle indicated by the fourth position information is smaller than the lateral distance between the boundary line and the vehicle, and the lateral direction is a direction parallel to the ground and perpendicular to the vehicle's driving direction.

[0243] It should be noted that since the road point cloud data collected by the laser radar is three-dimensional point cloud data of the vehicle's surrounding environment, it includes the road boundary line point cloud data and lane line point cloud data, and a set of coordinate point data in the road boundary line point cloud data is represented as a boundary. For a set of coordinate point data corresponding to a boundary, they are matched with the determined left and right lane lines respectively, and the average distance between the left and right lane lines and the boundary lines is calculated. If the average distance is less than the preset distance, it indicates that the average distance between the lane line and the boundary line remains within a stable small range, indicating that there is no significant offset between the two, that is, the direction and position relationship are basically consistent. Therefore, it can be judged that the left and right lane lines are in a parallel state with the boundary line, so that the fourth position information of the left and right lane lines can be directly corrected according to the fourth position information of the left and right lane lines and the boundary line point cloud data to generate the corrected fourth position information. The specific calculation formula for the average distance between the lane line and the boundary line can be:

[0244]

[0245] Among them, dis avg is the average distance between the lane line and the boundary line; N lidaris the number of valid points in the boundary point cloud data; (x lidar ,y lidar ) is the point coordinate in the boundary line point cloud data; C is the fitting coefficient.

[0246] Furthermore, according to the fourth position information of the left and right lane lines and the boundary line point cloud data, the specific method of correcting the fourth position information of the left and right lane lines includes:

[0247] First, according to the fourth position information of the left and right lane lines and the boundary line point cloud data, the target lane line position points in the left and right lane lines are determined; then, according to the fourth position information of the left and right lane lines, the lane line position points in the left and right lane lines whose ordinates are greater than the target ordinates are deleted to obtain the corrected left and right lane lines, and the target ordinate is the ordinate of the target lane line position point; then, according to the boundary line point cloud data, the boundary line position points in the boundary line whose ordinates are greater than the target ordinate are added to the corrected left and right lane lines to generate the target left and right lane lines; finally, the position information of the target left and right lane lines in the vehicle body coordinate system is determined as the corrected fourth position information.

[0248] Among them, the lateral distance between the target lane line position point and the vehicle is greater than the lateral distance between the adjacent lane line position point and the vehicle.

[0249] It should be noted that the method for determining the position of the target lane line can be as follows: first, according to the fourth position information of the left and right lane lines, a point to be checked is selected, for example (vcs x , vcs y ), then match the boundary point coordinates closest to the check point in the boundary line point cloud data, for example (lidia x ,lidia y ); then, if the selected point to be checked belongs to the left lane line, determine whether it satisfies lidia y <vcs y ; Correspondingly, if the selected point to be checked belongs to the right lane line, then determine whether lidia is satisfied y >vcs y Until all the positions in the left and right lanes are traversed, the first point to be checked that meets the conditions is found as the target lane position, which can be recorded as (vcs 0 , vcs 0); further, according to the fourth position information of the left and right lane lines, the lane line position points whose ordinates are greater than the target ordinates are deleted to obtain the corrected left and right lane lines. In order to make the lane line information more complete, it is also necessary to add the deleted lane line position points, that is, according to the boundary line point cloud data, the boundary line position points whose ordinates are greater than the target ordinates are added to the corrected left and right lane lines to generate the target left and right lane lines.

[0250] It should be noted that if the boundary line position point whose ordinate is greater than the target ordinate is directly added to the corrected left and right lane lines, the overall continuity of the lane lines may be destroyed. Therefore, it is necessary to add translation. Through translation adjustment, the position of the laser boundary point is more in line with the curve direction of the original lane line, ensuring that the added laser boundary point can be seamlessly integrated into the lane line data. The translation is calculated as the average horizontal distance between the boundary line and the target ordinate within a longitudinal range of 20 meters.

[0251] Optionally, if the target lane line position is not found after traversing all position points in the left and right lane lines, it indicates that the current left and right lane lines have no conflict with the boundary, and the current fourth position information does not need to be updated.

[0252] It can be understood that the point cloud data of the boundary line is integrated into the corrected lane line, providing more detailed lane boundary information and improving driving safety; and by screening the target lane line position points and deleting unnecessary lane line information points, the interference of erroneous or noisy data can be effectively reduced, thereby further improving the accuracy of lane line output.

[0253] Figure 5 Schematic diagram of the process of the lane line processing method provided by the embodiment of the present invention Figure 4 ;like Figure 5 As shown, the method includes:

[0254] S51. According to the lane line image acquired by the image acquisition device of the vehicle, the lane line pixels are projected into the vehicle body coordinate system to determine the first position information of each lane line.

[0255] S52. According to the first position information of each lane line, determine whether the lane line image contains the left and right lane lines corresponding to the current lane. If not, execute S53; if so, execute S55.

[0256] S53: Determine a pitch angle deviation corresponding to each lane line pixel in the lane line image according to the point cloud data of the road surface in front of the vehicle and the first position information of each lane line.

[0257] S54. Reproject the lane line pixels to the vehicle body coordinate system according to the pitch angle deviation corresponding to each lane line pixel, and determine the second position information of each lane line.

[0258] S55. Determine a reference width of the current lane formed by the left and right lane lines based on the first position information of the left and right lane lines, and correct the left and right lane lines based on the reference width and the lane line pixel coordinates corresponding to the left and right lane lines to obtain third position information of the left and right lane lines in the coordinate system of the image acquisition device.

[0259] S56. According to the third position information of the left and right lane lines, the lane line pixels corresponding to the left and right lane lines are converted to the vehicle body coordinate system to determine the fourth position information of the left and right lane lines.

[0260] It should be noted that the specific implementation process is illustrated by way of example in the above embodiments, and the embodiments of the present application will not be elaborated in detail here.

[0261] Figure 6 Schematic diagram of the process of the lane line processing method provided by the embodiment of the present invention Figure 5 ;like Figure 6 As shown, the method includes:

[0262] S61. According to the point cloud data of the road surface in front of the vehicle, the road surface in front of the vehicle is divided into a plurality of sub-areas along the driving direction of the vehicle, and the point cloud data of each sub-area is sparsely processed according to a preset longitudinal interval to obtain the sparse point cloud data in each sub-area.

[0263] S62: Determine height information of the road surface in front of the vehicle according to the sparse point cloud data in each sub-area.

[0264] S63: Determine pitch angle deviation information according to height information of the road surface in front of the vehicle.

[0265] S64. Determine a plurality of position points on the road surface in front of the vehicle along the vehicle's driving direction, and the three-dimensional coordinates of each position point in the vehicle body coordinate system, based on the point cloud data, height information, and pitch angle deviation information of the road surface in front of the vehicle.

[0266] S65. According to the three-dimensional coordinates of each position point in the vehicle body coordinate system, each position point is reversely projected into the lane line image to determine the pixel coordinates of each position point.

[0267] S66. Determine the pitch angle deviation corresponding to each lane line pixel according to the pixel coordinates of each position point and the pixel coordinates of each lane line pixel.

[0268] It should be noted that the specific implementation process is illustrated by way of example in the above embodiments, and the embodiments of the present application will not be elaborated in detail here.

[0269] Figure 7 Schematic diagram of the process of the lane line processing method provided by the embodiment of the present invention Figure 6 ;like Figure 7 As shown, the method includes:

[0270] S71. If the lane line image includes left and right lane lines, determine a reference width of a current lane formed by the left and right lane lines according to first position information of the left and right lane lines.

[0271] S72. For a first pixel point of the left lane line, determine a second pixel point in the right lane line that is closest to the first pixel point according to lane line pixel coordinates of the left and right lane lines.

[0272] S73. Correct the pixel coordinates of the first pixel and the pixel coordinates of the second pixel according to the reference width, the optical center coordinates and the focal length coordinates of the image acquisition device to obtain third position information of the left and right lane lines.

[0273] S74. According to the third position information of the left and right lane lines, the lane line pixels corresponding to the left and right lane lines are converted to the vehicle body coordinate system to determine the fourth position information of the left and right lane lines.

[0274] It should be noted that the specific implementation process is illustrated by way of example in the above embodiments, and the embodiments of the present application will not be elaborated in detail here.

[0275] Figure 8 Schematic diagram of the process of the lane line processing method provided by the embodiment of the present invention Figure 7 ;like Figure 8 As shown, the method includes:

[0276] S81. Determine boundary point cloud data of the road surface in front of the vehicle from the point cloud data of the road surface in front of the vehicle.

[0277] S82. According to the fourth position information of the left and right lane lines and the boundary line point cloud data, determine whether the average distance between the left and right lane lines and any boundary line is less than a preset distance. If not, execute S83; if yes, execute S84.

[0278] S83, obtaining other boundary line point cloud data in the boundary line point cloud data of the road surface in front of the vehicle, and making another judgment.

[0279] S84. Determine the target lane line position point in the left and right lane lines according to the fourth position information of the left and right lane lines and the boundary line point cloud data, and delete the lane line position points in the left and right lane lines whose ordinates are greater than the target ordinates according to the fourth position information of the left and right lane lines to obtain corrected left and right lane lines.

[0280] S85. According to the boundary line point cloud data, the boundary line position points whose ordinates are greater than the target ordinates are added to the corrected left and right lane lines to generate the target left and right lane lines.

[0281] S86. Determine the position information of the target left and right lane lines in the vehicle body coordinate system as the corrected fourth position information.

[0282] It should be noted that the specific implementation process is illustrated by way of example in the above embodiments, and the embodiments of the present application will not be elaborated in detail here.

[0283] Fig. 9 A schematic diagram of the structure of a lane line processing device provided in an embodiment of the present invention; Fig. 9 As shown, the lane line processing device 9 includes:

[0284] A projection module 91 is used to project lane line pixels to a vehicle body coordinate system according to a lane line image acquired by an image acquisition device of the vehicle, and determine first position information of each lane line;

[0285] A first determination module 92, configured to determine whether the lane line image includes left and right lane lines corresponding to the current lane according to the first position information of each lane line;

[0286] The second determination module 93 determines the pitch angle deviation corresponding to each lane line pixel in the lane line image according to the point cloud data of the road surface in front of the vehicle and the first position information of each lane line if the lane line image does not include the left and right lane lines;

[0287] The third determination module 94 is used to reproject the lane line pixels to the vehicle body coordinate system according to the pitch angle deviation corresponding to each lane line pixel, and determine the second position information of each lane line.

[0288] Further, the second determining module 93 is specifically configured to:

[0289] Determine the height information and pitch angle deviation information of the road surface in front of the vehicle based on the point cloud data of the road surface in front of the vehicle;

[0290] According to the point cloud data, height information and pitch angle deviation information of the road surface in front of the vehicle, a plurality of position points are determined along the vehicle driving direction on the road surface in front of the vehicle, and the three-dimensional coordinates of each position point in the vehicle body coordinate system;

[0291] According to the three-dimensional coordinates of each position point in the vehicle body coordinate system, each position point is reversely projected into the lane line image to determine the pixel coordinates of each position point;

[0292] According to the pixel coordinates of each position point and the pixel coordinates of each lane line pixel, the pitch angle deviation corresponding to each lane line pixel is determined.

[0293] Further, the second determining module 93 is specifically configured to:

[0294] Divide the road surface in front of the vehicle into multiple sub-areas along the vehicle's driving direction;

[0295] Sparse processing is performed on the point cloud data of each sub-area according to a preset longitudinal interval to obtain the sparse point cloud data in each sub-area, where the longitudinal direction is the driving direction of the vehicle;

[0296] Determine the height information of the road surface in front of the vehicle based on the sparse point cloud data in each sub-area;

[0297] The pitch angle deviation information is determined based on the height information of the road surface in front of the vehicle.

[0298] Furthermore, the lane processing device 9 further includes a processing module 95, which is specifically used for:

[0299] For each lane line pixel in each lane line, add the preset pitch angle in the preset rotation matrix to the pitch angle deviation corresponding to the lane line pixel to generate a target rotation matrix;

[0300] According to the target rotation matrix, the preset intrinsic parameter matrix, the preset extrinsic parameter matrix, the preset translation vector and the pixel position of the lane line pixel, the lane line pixel is reprojected to the vehicle body coordinate system to determine the second position information of each lane line.

[0301] Further, the processing module 95 is further used for:

[0302] If the lane line image includes left and right lane lines, determining a reference width of a current lane formed by the left and right lane lines according to first position information of the left and right lane lines;

[0303] According to the reference width and the lane line pixel coordinates corresponding to the left and right lane lines, the left and right lane lines are corrected to obtain the third position information of the left and right lane lines in the image acquisition device coordinate system; wherein the width of the current lane represented by the third position information of the left and right lane lines is the reference width;

[0304] According to the third position information of the left and right lane lines, the lane line pixels corresponding to the left and right lane lines are converted to the vehicle body coordinate system to determine the fourth position information of the left and right lane lines.

[0305] Further, the processing module 95 is further used for:

[0306] Determine the boundary line point cloud data of the road surface in front of the vehicle from the point cloud data of the road surface in front of the vehicle;

[0307] According to the fourth position information of the left and right lane lines and the boundary line point cloud data, it is determined whether the average distance between the left and right lane lines and the boundary line is less than a preset distance;

[0308] If the average distance is less than the preset distance, the fourth position information of the left and right lane lines is corrected according to the fourth position information of the left and right lane lines and the boundary line point cloud data to generate corrected fourth position information;

[0309] Among them, the lateral distance between the left and right lane lines and the vehicle indicated by the fourth position information is smaller than the lateral distance between the boundary line and the vehicle, and the lateral direction is a direction parallel to the ground and perpendicular to the vehicle's driving direction.

[0310] Further, the processing module 95 is specifically used for:

[0311] For a first pixel point of the left lane line, determine a second pixel point in the right lane line that is closest to the first pixel point according to lane line pixel coordinates of the left and right lane lines;

[0312] According to the reference width, the optical center coordinates and the focal length coordinates of the image acquisition device, the pixel point coordinates of the first pixel point and the pixel point coordinates of the second pixel point are corrected to obtain the third position information of the left and right lane lines.

[0313] Further, the processing module 95 is specifically used for:

[0314] Determine the target lane line position point in the left and right lane lines according to the fourth position information of the left and right lane lines and the boundary line point cloud data; the lateral distance between the target lane line position point and the vehicle is greater than the lateral distance between the adjacent lane line position point and the vehicle;

[0315] According to the fourth position information of the left and right lane lines, the lane line position points whose ordinates are greater than the target ordinates in the left and right lane lines are deleted to obtain corrected left and right lane lines, where the target ordinate is the ordinate of the target lane line position point;

[0316] According to the boundary line point cloud data, the boundary line position points whose ordinates are greater than the target ordinates are added to the corrected left and right lane lines to generate the target left and right lane lines;

[0317] The position information of the left and right lane lines of the target in the vehicle body coordinate system is determined as the corrected fourth position information.

[0318] Furthermore, the projection module 91 is specifically used for:

[0319] According to the lane line image, determine the pixel coordinates of each lane line pixel;

[0320] According to the preset rotation matrix, the preset intrinsic parameter matrix, the preset extrinsic parameter matrix, the preset translation vector and the first coordinate of each lane line pixel, each lane line pixel is projected into the vehicle body coordinate system to obtain the three-dimensional coordinates of each lane line pixel;

[0321] The first position information of each lane line is determined according to the three-dimensional coordinates of the lane line pixels contained in each lane line.

[0322] Further, the processing module 95 is specifically used for:

[0323] According to the second coordinates of the lane line pixels contained in each lane line, a cubic curve fitting is performed on each lane line to determine the position fitting curve of each lane line, where the position fitting curve is the first position information.

[0324] Further, the first determining module 92 is specifically configured to:

[0325] According to the first position information of each lane line, two target lane lines are determined from all lane lines included in the lane line image; wherein the lateral distance between any target lane line and the vehicle is smaller than the lateral distance between any other lane line and the vehicle, and the lateral direction is a direction parallel to the ground and perpendicular to the driving direction of the vehicle;

[0326] Determine a lane width formed by the two target lane lines according to the first position information of the two target lane lines;

[0327] When the lane width is less than the preset lane width, and the lateral distance between any target lane line and the vehicle is less than the preset lateral distance, the two target lane lines are determined to be the left and right lane lines.

[0328] The lane line processing method device provided in an embodiment of the present invention can be used to execute the lane line processing method in any of the above embodiments. Its implementation principle and technical effects are similar and will not be repeated here.

[0329] It should be noted that it should be understood that the division of the various modules of the above device is only a division of logical functions. In actual implementation, all or part of them can be integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software called by processing elements. They can also be implemented in the form of hardware. Some modules can also be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. In addition, all or part of these modules can be integrated together or implemented independently. The processing element here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in the processor element or instructions in the form of software.

[0330] Fig.10 Schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. Fig.10As shown, the electronic device 10 may include: a processor 101, a memory 102, and computer execution instructions stored in the memory 102 and executable on the processor 101. When the processor 101 executes the computer execution instructions, the lane line processing method provided by any of the aforementioned embodiments is implemented. Optionally, the above-mentioned components of the electronic device 110 may be connected via a system bus.

[0331] The memory 102 may be a separate storage unit or a storage unit integrated in a processor. The number of processors may be one or more.

[0332] Optionally, the electronic device 10 may further include a communication interface for interacting with other devices.

[0333] It should be understood that the processor 101 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc. The steps of the method disclosed in the present invention may be directly embodied as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0334] The system bus may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The system bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage.

[0335] All or part of the steps of the above-mentioned method embodiments can be completed by hardware related to program instructions. The above-mentioned program can be stored in a readable memory. When the program is executed, the steps of the above-mentioned method embodiments are executed. The above-mentioned memory (storage medium) includes: read-only memory (ROM), RAM, flash memory, hard disk, solid state drive, magnetic tape, floppy disk, optical disc and any combination thereof.

[0336] The electronic device provided in an embodiment of the present invention can be used to execute the lane line processing method provided in any of the above method embodiments. Its implementation principle and technical effects are similar and will not be repeated here.

[0337] An embodiment of the present invention provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed on a computer, the computer executes the above-mentioned lane line processing method.

[0338] The computer-readable storage medium mentioned above can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory, electrically erasable programmable read-only memory, erasable programmable read-only memory, programmable read-only memory, read-only memory, magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0339] Optionally, a readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0340] An embodiment of the present invention also provides a computer program product, which includes a computer program. The computer program is stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium. When at least one processor executes the computer program, the above-mentioned lane line processing method can be implemented.

[0341] It should be understood that the present invention is not limited to the exact construction that has been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A lane line processing method, characterized in that: include: According to the lane line image acquired by the image acquisition device of the vehicle, the lane line pixels are projected into the vehicle body coordinate system to determine the first position information of each lane line; Determining, based on the first position information of each lane line, whether the lane line image includes left and right lane lines corresponding to the current lane; If the lane line image does not include the left and right lane lines, determining the pitch angle deviation corresponding to each lane line pixel in the lane line image according to the point cloud data of the road surface in front of the vehicle and the first position information of each lane line; According to the pitch angle deviation corresponding to each lane line pixel, the lane line pixel is reprojected to the vehicle body coordinate system to determine the second position information of each lane line.

2. The method according to claim 1, characterized in that The step of determining the pitch angle deviation corresponding to each lane line pixel in the lane line image according to the point cloud data of the road surface in front of the vehicle and the first position information of each lane line includes: Determining height information and pitch angle deviation information of the road surface in front of the vehicle according to the point cloud data of the road surface in front of the vehicle; Determine a plurality of position points on the road surface in front of the vehicle along the vehicle's driving direction, and a three-dimensional coordinate of each position point in a vehicle body coordinate system according to the point cloud data of the road surface in front of the vehicle, the height information, and the pitch angle deviation information; According to the three-dimensional coordinates of each position point in the vehicle body coordinate system, each position point is reversely projected into the lane line image to determine the pixel coordinates of each position point; According to the pixel coordinates of each position point and the pixel coordinates of each lane line pixel, the pitch angle deviation corresponding to each lane line pixel is determined.

3. The method according to claim 2, characterized in that Determining the height information and pitch angle deviation information of the road surface in front of the vehicle according to the point cloud data of the road surface in front of the vehicle includes: Dividing the road surface in front of the vehicle into a plurality of sub-areas along the driving direction of the vehicle; Performing sparse processing on the point cloud data of each sub-area according to a preset longitudinal interval to obtain the sparse point cloud data in each sub-area, wherein the longitudinal direction is the driving direction of the vehicle; Determine the height information of the road surface in front of the vehicle according to the sparse point cloud data in each sub-area; The pitch angle deviation information is determined according to the height information of the road surface in front of the vehicle.

4. The method according to any one of claims 1 to 3, characterized in that: The step of reprojecting the lane line pixels to the vehicle body coordinate system according to the pitch angle deviation corresponding to each lane line pixel to determine the second position information of each lane line includes: For each lane line pixel in each lane line, add the preset pitch angle in the preset rotation matrix to the pitch angle deviation corresponding to the lane line pixel to generate a target rotation matrix; According to the target rotation matrix, the preset intrinsic parameter matrix, the preset extrinsic parameter matrix, the preset translation vector and the pixel position of the lane line pixel, the lane line pixel is reprojected to the vehicle body coordinate system to determine the second position information of each lane line.

5. The method according to claim 2 or 3, characterized in that: The method further comprises: If the lane line image includes the left and right lane lines, determining a reference width of the current lane formed by the left and right lane lines according to the first position information of the left and right lane lines; According to the reference width and the lane line pixel coordinates corresponding to the left and right lane lines, the left and right lane lines are corrected to obtain third position information of the left and right lane lines in the image acquisition device coordinate system; wherein the width of the current lane represented by the third position information of the left and right lane lines is the reference width; According to the third position information of the left and right lane lines, the lane line pixels corresponding to the left and right lane lines are converted to the vehicle body coordinate system to determine the fourth position information of the left and right lane lines.

6. The method according to claim 5, characterized in that The method further comprises: Determining the boundary line point cloud data of the road surface in front of the vehicle from the point cloud data of the road surface in front of the vehicle; Determining, based on the fourth position information of the left and right lane lines and the boundary line point cloud data, whether an average distance between the left and right lane lines and the boundary line is less than a preset distance; If the average distance is less than the preset distance, the fourth position information of the left and right lane lines is corrected according to the fourth position information of the left and right lane lines and the boundary line point cloud data to generate corrected fourth position information; Among them, the lateral distance between the left and right lane lines indicated by the fourth position information and the vehicle is smaller than the lateral distance between the boundary line and the vehicle, and the lateral direction is a direction parallel to the ground and perpendicular to the driving direction of the vehicle.

7. The method according to claim 5, characterized in that The step of correcting the left and right lane lines according to the reference width and the lane line pixel coordinates corresponding to the left and right lane lines to obtain third position information of the left and right lane lines in the image acquisition device coordinate system includes: For a first pixel point of the left lane line, determine a second pixel point in the right lane line that is closest to the first pixel point according to lane line pixel coordinates of the left and right lane lines; According to the reference width, the optical center coordinates and the focal length coordinates of the image acquisition device, the pixel point coordinates of the first pixel point and the pixel point coordinates of the second pixel point are corrected to obtain the third position information of the left and right lane lines.

8. The method according to claim 6, characterized in that The method of correcting the fourth position information of the left and right lane lines according to the fourth position information of the left and right lane lines and the boundary line point cloud data to generate corrected fourth position information includes: Determine a target lane line position point in the left and right lane lines according to the fourth position information of the left and right lane lines and the boundary line point cloud data; the lateral distance between the target lane line position point and the vehicle is greater than the lateral distance between the adjacent lane line position point and the vehicle; According to the fourth position information of the left and right lane lines, the lane line position points whose ordinates are greater than the target ordinates in the left and right lane lines are deleted to obtain corrected left and right lane lines, where the target ordinate is the ordinate of the target lane line position point; According to the boundary line point cloud data, adding the boundary line position points whose ordinates are greater than the target ordinates to the corrected left and right lane lines to generate target left and right lane lines; The position information of the target left and right lane lines in the vehicle body coordinate system is determined as the corrected fourth position information.

9. The method according to claim 4, characterized in that The method of projecting lane line pixels onto a vehicle body coordinate system based on a lane line image acquired by an image acquisition device of the vehicle to determine first position information of each lane line includes: Determine the pixel coordinates of each lane line pixel according to the lane line image; According to the preset rotation matrix, the preset intrinsic parameter matrix, the preset extrinsic parameter matrix, the preset translation vector and the first coordinate of each lane line pixel, each lane line pixel is projected into the vehicle body coordinate system to obtain the three-dimensional coordinate of each lane line pixel; The first position information of each lane line is determined according to the three-dimensional coordinates of the lane line pixels contained in each lane line.

10. The method according to claim 9, characterized in that The determining the first position information of each lane line according to the three-dimensional coordinates of the lane line pixels included in each lane line includes: According to the second coordinates of the lane line pixels contained in each lane line, a cubic curve fitting is performed on each lane line to determine a position fitting curve of each lane line, and the position fitting curve is the first position information.

11. The method according to claim 2 or 3, characterized in that: The determining, based on the first position information of each lane line, whether the lane line image includes left and right lane lines corresponding to the current lane includes: According to the first position information of each lane line, two target lane lines are determined from all lane lines included in the lane line image; wherein the lateral distance between any target lane line and the vehicle is smaller than the lateral distance between other lane lines and the vehicle, and the lateral direction is a direction parallel to the ground and perpendicular to the driving direction of the vehicle; Determining a lane width formed by the two target lane lines according to the first position information of the two target lane lines; When the lane width is smaller than the preset lane width, and the lateral distance between any target lane line and the vehicle is smaller than the preset lateral distance, the two target lane lines are determined to be the left and right lane lines.

12. A lane line processing device, characterized in that: include: A projection module, used to project lane line pixels to a vehicle body coordinate system based on a lane line image acquired by an image acquisition device of the vehicle, and determine first position information of each lane line; A first determination module, used to determine whether the lane line image includes left and right lane lines corresponding to the current lane according to the first position information of each lane line; A second determination module, if the lane line image does not include the left and right lane lines, determines the pitch angle deviation corresponding to each lane line pixel in the lane line image according to the point cloud data of the road surface in front of the vehicle and the first position information of each lane line; The third determination module is used to reproject the lane line pixels to the vehicle body coordinate system according to the pitch angle deviation corresponding to each lane line pixel, so as to determine the second position information of each lane line.

13. An electronic device, comprising: A processor, a memory, and a computer-executable instruction stored in the memory and executable on the processor, wherein the processor is used to implement the lane line processing method as described in any one of claims 1 to 11 when executing the computer-executable instruction.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the lane line processing method according to any one of claims 1 to 11.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, it is used to implement the lane line processing method as described in any one of claims 1 to 11.

Images