Roadside line fitting and tracking method, device and equipment and storage medium

Abstract

The application particularly relates to a road edge line fitting and tracking method and device, equipment and a storage medium, the method comprising the following steps: scanning a region of interest (ROI) in a drivable area of a vehicle to obtain road edge line segmentation data points; performing interpolation processing on the road edge line segmentation data points to obtain a plurality of road edge line points; projecting the plurality of road edge line points from image coordinates to bird's-eye view coordinates based on a preset external parameter calibration result; performing road edge line fitting on the bird's-eye view coordinates to obtain a plurality of fitting parameters, traversing images in a preset order to obtain a vanishing point and a resolution of the image, calculating a real-time pitch angle and a real-time yaw angle of the vehicle, fitting new road edge line points, back-projecting the new road edge line points to the image coordinates, predicting a road edge line state according to the image coordinates obtained by back-projection, and obtaining a road edge fitting line to complete tracking of the road edge line. Thus, the problem of inaccurate line coordinate information caused by road surface changes and the like is eliminated.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of vehicles, in particular to a road curb line fitting and tracking method and device, equipment and a storage medium. BACKGROUND

[0002] In the process of driving, in order to better understand and predict the surrounding environment information, curve fitting of lane lines is necessary. Curve fitting of the collected lane line data can well estimate the parameter information of the lane line, such as offset, inclination angle, and curvature radius, so as to predict the trend of the lane line and provide help for the driver or the automatic control system of the vehicle.

[0003] In related technologies, one is to propose an adaptive fitting algorithm, which only fits the lane line, and the other is to propose a lane line tracking method, which comprehensively considers the lane line.

[0004] However, the first one only fits the lane line and does not consider the tracking problem, which cannot eliminate the inaccuracy of line coordinate information caused by road surface changes and the like, and the second one does not update the pitch angle and yaw angle information of the ego vehicle, which leads to inaccurate lane line coordinate information fitted by the ego vehicle on uphill and downhill road sections. SUMMARY

[0005] The present application provides a road curb line fitting and tracking method, device, equipment and storage medium to solve the problem of inaccurate line coordinate information caused by road surface changes and the like.

[0006] The first aspect of the present application provides a road curb line fitting and tracking method, comprising the following steps: scanning a region of interest (ROI) region in the drivable area of a vehicle to obtain curb line segmentation data points; performing interpolation processing on the curb line segmentation data points to obtain a plurality of curb line points; projecting the plurality of curb line points from image coordinates to bird's eye view coordinates based on a preset external parameter calibration result; fitting the bird's eye view coordinates to obtain a plurality of fitting parameters, and traversing the image according to a preset order to obtain a vanishing coordinate point; calculating a real-time pitch angle and a real-time yaw angle of the vehicle according to the vanishing coordinate point and the resolution of the image, and fitting new curb line points according to the real-time pitch angle and the real-time yaw angle; and projecting the new curb line points to the image coordinates, and predicting the state of the curb line according to the image coordinates obtained by the inverse projection to obtain a curb fitting line, so as to complete the tracking of the curb line.

[0007] According to the technical means, the problems of being affected by light intensity, road water and other factors can be solved, the complexity operation of classifying each pixel point is avoided, the drivable area segmentation is realized quickly, the position of the road edge can be effectively detected, the curvature and trajectory of the road edge can be predicted in real time, the pitch and yaw information of the ego vehicle is updated in real time, the lane line coordinate information fitted by the ego vehicle on the uphill and downhill road sections is more accurate, and the situation that the remote road edge deviates from the ground is avoided.

[0008] Further, in some embodiments, before the interpolation processing is performed on the road edge segmentation data points, the method further includes: determining whether the number of the road edge segmentation data points is even and greater than a preset number; if the number of the road edge segmentation data points is even and greater than the preset number, performing interpolation processing on the road edge segmentation data points by using a cubic Bezier curve; otherwise, not performing interpolation processing.

[0009] According to the technical means, the complexity operation of classifying each pixel point is avoided, and the drivable area segmentation is realized quickly.

[0010] Further, in some embodiments, the cubic Bezier curve is:

[0011]

[0012] wherein n=3, t∈[0, 1], P i is the i-th control point.

[0013] According to the technical means, the coordinates of a part of points are supplemented by using the interpolation algorithm, and are used for subsequent point fitting.

[0014] Further, in some embodiments, the calculating the real-time pitch and real-time yaw of the vehicle according to the vanishing point and the resolution of the image includes: calculating the real-time pitch and real-time yaw of the vehicle according to the vanishing point and the resolution of the image based on a preset angle calculation formula, wherein the preset angle calculation formula is:

[0015] pitch = vanish_point.y - image.rows / 2;

[0016] yaw = vanish_point.x - image.cols / 2;

[0017] wherein pitch is the pitch, yaw is the yaw, vanish_point.y is the y-direction vanishing point, vanish_point.x is the x-direction vanishing point, image.rows is the number of rows of the image, and image.cols is the number of columns of the image.

[0018] According to the technical means, the vanishing point, the yaw angle and the pitch angle can be updated in real time, and the driving state of the ego vehicle and the real road surface condition can be accurately predicted.

[0019] Further, in some embodiments, the aerial view coordinates are fitted with a road edge line, including: constructing a first matrix and a second matrix; fitting the aerial view coordinates with a cubic polynomial based on the first matrix and the second matrix to obtain the plurality of fitting parameters.

[0020] According to the technical means, the optimal solution of the fitting and tracking weight can be calculated to obtain the final road edge line result, and the problem of inaccurate line coordinate information caused by road surface changes and the like can be eliminated.

[0021] Further, in some embodiments, the cubic polynomial is:

[0022] x = c0 + c1y + c2y 2 + c3y 3 ;

[0023] wherein c0 is a lateral offset of a proximal end of the road edge line relative to a center of the vehicle body, c1 is a direction of the road edge line, c2 is a curvature of the road edge line, and c3 is a rate of change of the curvature.

[0024] According to the technical means, the coordinates of a part of points can be supplemented by using an interpolation algorithm, for subsequent fitting of points.

[0025] The second aspect embodiment of the present application provides a road edge line fitting and tracking device, including: a scanning module configured to scan a region of interest (ROI) in a drivable area of a vehicle to obtain road edge line segmentation data points; a processing module configured to perform interpolation processing on the road edge line segmentation data points to obtain a plurality of road edge line points; a projection module configured to project the plurality of road edge line points from image coordinates to aerial view coordinates based on a preset extrinsic calibration result; a fitting module configured to fit the aerial view coordinates with a road edge line to obtain a plurality of fitting parameters, and the plurality of fitting parameters are traversed in a preset order to obtain vanishing coordinate points; a calculation module configured to calculate real-time pitch angle and real-time yaw angle of the vehicle according to the vanishing coordinate points and a resolution of the image, and fit new road edge line points according to the real-time pitch angle and the real-time yaw angle; and a tracking module configured to project the new road edge line points to the image coordinates, and predict a road edge line state according to the image coordinates obtained by the projection to obtain a road edge fitting line, so as to complete tracking of the road edge line.

[0026] Further, in some embodiments, the processing module is further configured to: determine whether the number of road edge line segmentation data points is even and greater than a preset number; if the number of road edge line segmentation data points is even and greater than the preset number, perform interpolation processing on the road edge line segmentation data points using a cubic Bezier curve; otherwise, do not perform interpolation processing.

[0027] Further, in some embodiments, the cubic Bezier curve is:

[0028]

[0029] wherein n = 3, t ∈ [0, 1], P i is the i-th control point.

[0030] Further, in some embodiments, the calculating the real-time pitch angle and the real-time yaw angle of the vehicle according to the vanishing point and the resolution of the image comprises: calculating the real-time pitch angle and the real-time yaw angle of the vehicle according to the vanishing point and the resolution of the image based on a preset angle calculation formula, wherein the preset angle calculation formula is:

[0031] pitch = vanish_point.y - image.rows / 2;

[0032] yaw = vanish_point.x - image.cols / 2;

[0033] wherein pitch is the pitch angle, yaw is the yaw angle, vanish_point.y is the y-direction vanishing point, vanish_point.x is the x-direction vanishing point, image.rows is the number of image rows, and image.cols is the number of image columns.

[0034] Further, in some embodiments, the fitting the aerial view coordinates to a road edge line comprises: constructing a first matrix and a second matrix; and fitting the aerial view coordinates to a road edge line using a cubic polynomial based on the first matrix and the second matrix to obtain the plurality of fitting parameters.

[0035] Further, in some embodiments, the cubic polynomial is:

[0036] x = c0 + c1y + c2y 2 + c3y 3 ;

[0037] wherein c0 is the lateral offset of the proximal end of the road edge line relative to the center of the vehicle body, c1 is the direction of the road edge line, c2 is the curvature of the road edge line, and c3 is the rate of change of curvature.

[0038] A third aspect of this application provides an apparatus comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the roadside fitting and tracking method as described in the above embodiments.

[0039] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement the roadside fitting and tracking method for roads as described in the above embodiments.

[0040] Therefore, by using a drivable area segmentation method based on object detection, several object detection boxes are used to outline the approximate drivable area. Then, combined with subsequent filtering, distortion correction, and angle filtering algorithms, the coordinates of this coarse drivable area are refined to obtain the final drivable area. This solves the complexity of classifying each pixel and achieves drivable area segmentation more quickly. It can not only effectively detect the position of the roadside, but also predict the curvature and trajectory of the roadside in real time. It effectively overcomes the influence of factors such as light intensity and road water accumulation on roadside detection. The real-time updated vehicle pitch and yaw angle information makes the lane line coordinate information fitted by the vehicle on uphill and downhill road sections more accurate, and will not cause the fitted far-end roadside to deviate from the ground.

[0041] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0042] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0043] Figure 1 This is a flowchart of a roadside fitting and tracking method according to an embodiment of this application;

[0044] Figure 2 This is a schematic diagram illustrating the principle of a cubic Bézier curve according to an embodiment of this application;

[0045] Figure 3 This is a flowchart of a roadside fitting and tracking method according to an embodiment of this application;

[0046] Figure 4 This is a block diagram of a roadside fitting and tracking device provided according to an embodiment of this application.

[0047] Figure 5 This is a schematic diagram of an electronic device provided according to an embodiment of this application.

[0048] Among them, 10 is the roadside fitting and tracking device, 100 is the scanning module, 200 is the processing module, 300 is the projection module, 400 is the calculation module and 500 is the tracking module. Detailed Implementation

[0049] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0050] The following describes a method, apparatus, device, and storage medium for roadside fitting and tracking according to embodiments of this application, with reference to the accompanying drawings. Addressing the problem of inaccurate line coordinate information caused by road surface changes and other factors mentioned in the background art, this application provides a method for roadside fitting and tracking. In this method, a drivable area segmentation method based on target detection is used to outline a rough drivable area using several target detection boxes. Then, a subsequent filtering algorithm, distortion correction algorithm, and angle filtering algorithm are used to refine the coordinates of this coarse drivable area, resulting in the final drivable area. This solves the complexity of classifying each pixel, achieves faster drivable area segmentation, effectively detects the position of the roadside, and predicts the curvature and trajectory of the roadside in real time. It effectively overcomes the influence of factors such as light intensity and road water accumulation on roadside detection. The real-time updated vehicle pitch and yaw angle information makes the lane line coordinate information fitted by the vehicle on uphill and downhill sections more accurate, preventing the fitted distant roadside from detaching from the ground.

[0051] Specifically, Figure 1 This is a flowchart illustrating a roadside fitting and tracking method provided in an embodiment of this application.

[0052] like Figure 1 As shown, the roadside fitting and tracking method for this road includes the following steps:

[0053] In step S101, the region of interest (ROI) within the drivable area of ​​the vehicle is scanned to obtain roadside segmentation data points.

[0054] Specifically, the drivable area results output by the perception algorithm are first scanned line by line from bottom to top. Since part of the image captured by the vehicle's front-view camera is the hood area, the scanned ROI area is set as follows to eliminate redundant traversal:

[0055] Scan start point = image rows – 90;

[0056] Scan endpoint = image rows / 2 – 50;

[0057] During the scanning process, it is necessary to distinguish between static and dynamic boundary points. Dynamic boundary points are filtered out directly; static boundary points are further processed using the cubic Bézier curve algorithm to perform nearest-neighbor interpolation, providing sufficient point coordinates for subsequent roadside fitting.

[0058] In step S102, the roadside segmentation data points are interpolated to obtain multiple roadside points.

[0059] Furthermore, in some embodiments, before interpolating the roadside segmentation data points, the method further includes: determining whether the number of roadside segmentation data points is even and greater than a preset number; if the number of roadside segmentation data points is even and greater than the preset number, then interpolating the roadside segmentation data points using a cubic Bézier curve; otherwise, no interpolation operation is performed.

[0060] The preset number can be 4.

[0061] Specifically, cubic Bézier curves are used to interpolate the data points along the roadside, primarily to provide a preliminary description of the roadside's trend and curvature. This method is only invoked when the number of points along the roadside is even and greater than or equal to 4.

[0062] Furthermore, in some embodiments, the principle of cubic Bézier curves is as follows:

[0063]

[0064] Where n = 3, t ∈ [0, 1], P i For the i-th control point, such as Figure 2 As shown, the Bézier curve only passes through the first and last control points and not the middle control point. Based on the proportionality of the sides of similar triangles, we can deduce: CE / BC = EF / EM = GK / EK = HG / FG = t, n = 4, i = 0, 1, 2, 3. Substituting these values ​​into the formula, we obtain the coordinates of the difference point: point = P0(1 - 3t + 3t). 2 -t 3 )+P1(3t-6t 2 +3t 3 )+P2(3t 2 -3t 3 )+P3t 3 When t=0, point=P0, and when t=1, point=P3. This relationship is a constant.

[0065] In step S103, based on the preset extrinsic calibration results, multiple roadside points are projected from image coordinates to bird's-eye view coordinates.

[0066] Among them, the preset external parameter calibration result can be the calibration result of the external parameter table of the forward-view camera relative to the vehicle.

[0067] Specifically, based on the extrinsic calibration results of the forward-view camera relative to the vehicle, the homography matrix of multiple roadside points is calculated, projecting these points from image coordinates to vehicle body coordinates, thus realizing the projection from the image coordinate system to the vehicle body bird's-eye view coordinate system. The vehicle body coordinate system has its origin at the rear axle center, with the front of the vehicle as the positive X-axis and the left side as the positive Y-axis, and moves with the vehicle.

[0068] In step S104, roadside lines are fitted to the bird's-eye view coordinates to obtain multiple fitting parameters. The multiple fitting parameters are then traversed through the image in a preset order to obtain the vanishing coordinate points.

[0069] The preset order can be from bottom to top.

[0070] Furthermore, in some embodiments, the roadside line is fitted using the following cubic polynomial equation:

[0071] x = c0 + c1y + c2y 2 +c3y 3 ;

[0072] Where c0 is the lateral offset of the near end of the roadside line relative to the center of the vehicle body, c1 is the orientation of the roadside line, c2 is the curvature of the roadside line, and c3 is the rate of change of curvature.

[0073] Furthermore, in some embodiments, roadside fitting is performed on the bird's-eye view coordinates, including: constructing a first matrix and a second matrix; and using a cubic polynomial to fit the roadside coordinates on the bird's-eye view based on the first matrix and the second matrix to obtain multiple fitting parameters.

[0074] Construct matrices X and Y, where X is a 4x4 matrix and Y is a 4x1 matrix. Obtain matrix A by using AX = Y.

[0075]

[0076] Y = [P0.x + P1.x + PN-1x]

[0077] P0.y*P0.x+P1.y*P1.x+...+PN-1y*PN-1.x

[0078] (P0.y)^2*P0.x+(P1y)^2*P1.x+..+(PN-1.y)^2*PN-1.x

[0079] (P0.y)^3*P0.x+(P1.y)^3*P1.x+..+(PN-1.y)^3*PN-1.x];

[0080] Where P0...PN-1 are the coordinates (x, y) of the points output along the road, and there are a total of N points.

[0081] From the above formulas, the fitting parameters c0 = A(0,0), c1 = A(1,0), c2 = A(2,0), and c3 = A(3,0) can be obtained. Using the fitting parameters c0, c1, c2, and c3, the (x,y) coordinates of the two paths can be obtained by traversing the image from bottom to top. For the same y-value, the absolute values ​​of the differences between the x-coordinates of the two lines are calculated and accumulated. If the maximum value of this accumulated sum is within a set threshold range, the (x,y) coordinates reached by this maximum value are used as the vanishing point coordinates (vanish_point(x,y)) of the current frame.

[0082] In step S105, the real-time pitch angle and real-time yaw angle of the vehicle are calculated based on the vanishing coordinate points and the resolution of the image, and new roadside points are fitted based on the real-time pitch angle and real-time yaw angle.

[0083] Furthermore, in some embodiments, calculating the vehicle's real-time pitch angle and real-time yaw angle based on the vanishing coordinates and the image resolution includes: calculating the vehicle's real-time pitch angle and real-time yaw angle based on a preset angle calculation formula, wherein the preset angle calculation formula is:

[0084] pitch = vanish_point.y - image.rows / 2;

[0085] yaw = vanish_point.x - image.cols / 2;

[0086] Where pitch is the pitch angle, yaw is the yaw angle, vanish_point.y is the vanishing point in the y direction, vanish_point.x is the vanishing point in the x direction, image.rows is the number of image rows, and image.cols is the number of image columns.

[0087] It should be noted that because the two paths are fitted and updated in real time, that is, the fitting parameters c0c1c2c3 are updated in real time, the vanishing point is also updated accordingly. The pitch and yaw are calculated from the vanishing point, so they are also updated accordingly.

[0088] In step S106, the new roadside points are back-projected onto the image coordinates, and the roadside state is predicted based on the back-projected image coordinates to obtain the roadside fitting line, thereby completing the tracking of the roadside.

[0089] Specifically, the fitted roadside points are used to establish a one-to-one temporal correspondence using prediction and matching algorithms, ensuring that the fitted roadside points match those obtained from the mathematical prediction model. The left and right attributes of the roadside are distinguished according to the principle that c0 of the left roadside line is less than c0 of the right roadside line. Then, an extended Kalman filter (EKF) is used to track the fitted roadside lines, with vehicle body information as the control variable and the current frame's roadside pixel coordinates as the observation. The Kalman filter algorithm is used to predict the roadside state, and combined with the Hungarian algorithm to complete the matching of the corresponding roadside lines, thus completing the tracking.

[0090] To enable those skilled in the art to further understand the roadside fitting and tracking method of this application, a detailed description is provided below with reference to specific embodiments.

[0091] like Figure 3 As shown, Figure 3 A flowchart of a roadside fitting and tracking method according to an embodiment of this application.

[0092] S301, Input of roadside data for drivable areas.

[0093] S302, three-stage Bezier preprocessing.

[0094] S303, Image coordinates are projected onto bird's-eye view coordinates.

[0095] S304, using a cubic polynomial for roadside line fitting.

[0096] S305, updated vanishing point, pitch, and yaw.

[0097] S306, Fitting and Tracking.

[0098] S307, backprojected to image coordinates.

[0099] S308, draw the roadside fitting line in the original video frame.

[0100] The roadside fitting and tracking method proposed in this application uses a target detection-based drivable area segmentation method to outline the approximate drivable area with several target detection boxes. Then, it combines subsequent filtering, distortion correction, and angle filtering algorithms to refine the coordinates of the coarse drivable area, thus obtaining the final drivable area. This solves the complexity of classifying each pixel and achieves drivable area segmentation more quickly. It can not only effectively detect the position of the roadside but also predict the curvature and trajectory of the roadside in real time. It effectively overcomes the influence of factors such as light intensity and road water accumulation on roadside detection. The real-time updated vehicle pitch and yaw angle information makes the lane line coordinate information fitted by the vehicle on uphill and downhill sections more accurate and prevents the fitted far-end roadside from deviating from the ground.

[0101] Next, referring to the accompanying drawings, a roadside fitting and tracking device according to an embodiment of this application is described.

[0102] Figure 4 This is a block diagram of a roadside fitting and tracking device according to an embodiment of this application.

[0103] like Figure 4 As shown, the roadside fitting and tracking device 10 includes: a scanning module 100, a processing module 200, a projection module 300, a fitting module 400, a calculation module 500, and a tracking module 600.

[0104] The system comprises the following modules: a scanning module 100, which scans the Region of Interest (ROI) within the drivable area of ​​the vehicle to obtain roadside segmentation data points; a processing module 200, which interpolates the roadside segmentation data points to obtain multiple roadside points; a projection module 300, which projects the multiple roadside points from image coordinates to bird's-eye view coordinates based on preset extrinsic parameter calibration results; a fitting module 400, which fits the roadside points to the bird's-eye view coordinates to obtain multiple fitting parameters, and then iterates through the image according to a preset order to obtain vanishing coordinate points; a calculation module 500, which calculates the vehicle's real-time pitch and yaw angles based on the vanishing coordinate points and the image resolution, and fits new roadside points based on the real-time pitch and yaw angles; and a tracking module 600, which back-projects the new roadside points onto the image coordinates and predicts the roadside state based on the back-projected image coordinates to obtain a roadside fitting line, thereby completing the tracking of the roadside.

[0105] Furthermore, in some embodiments, the processing module 200 is also used to: determine whether the number of roadside segmentation data points is even and greater than a preset number; if the number of roadside segmentation data points is even and greater than the preset number, then interpolate the roadside segmentation data points using a cubic Bézier curve; otherwise, no interpolation operation is performed.

[0106] Furthermore, in some embodiments, the cubic Bézier curve is:

[0107]

[0108] Where n = 3, t ∈ [0, 1], P i Let i be the i-th control point.

[0109] Furthermore, in some embodiments, the calculation module 500 is specifically used to: calculate the real-time pitch angle and real-time yaw angle of the vehicle based on a preset angle calculation formula, according to the vanishing coordinate point and the image resolution, wherein the preset angle calculation formula is:

[0110] pitch = vanish_point.y - image.rows / 2;

[0111] yaw = vanish_point.x - image.cols / 2;

[0112] Where pitch is the pitch angle, yaw is the yaw angle, vanish_point.y is the vanishing point in the y direction, vanish_point.x is the vanishing point in the x direction, image.rows is the number of image rows, and image.cols is the number of image columns.

[0113] Furthermore, in some embodiments, the fitting module 400 is specifically used to: construct a first matrix and a second matrix; and based on the first matrix and the second matrix, use a cubic polynomial to fit the roadside lines to the bird's-eye view coordinates to obtain multiple fitting parameters.

[0114] Furthermore, in some embodiments, the cubic polynomial is:

[0115] x = c0 + c1y + c2y 2 +c3y 3 ;

[0116] Where c0 is the lateral offset of the near end of the roadside line relative to the center of the vehicle body, c1 is the orientation of the roadside line, c2 is the curvature of the roadside line, and c3 is the rate of change of curvature.

[0117] It should be noted that the explanation of the roadside fitting and tracking method embodiment described above also applies to the roadside fitting and tracking device of this embodiment, and will not be repeated here.

[0118] The roadside fitting and tracking device proposed in this application uses a target detection-based drivable area segmentation method to outline the approximate drivable area with several target detection boxes. Then, it combines subsequent filtering, distortion correction, and angle filtering algorithms to refine the coordinates of the coarse drivable area, thus obtaining the final drivable area. This solves the complexity of classifying each pixel and achieves drivable area segmentation more quickly. It can not only effectively detect the position of the roadside but also predict the curvature and trajectory of the roadside in real time. It effectively overcomes the influence of factors such as light intensity and road water accumulation on roadside detection. The real-time updated vehicle pitch and yaw angle information makes the lane line coordinate information fitted by the vehicle on uphill and downhill sections more accurate and prevents the fitted far-end roadside from deviating from the ground.

[0119] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include:

[0120] The memory 501, the processor 502, and the computer program stored on the memory 501 and capable of running on the processor 502.

[0121] When the processor 502 executes the program, it implements the roadside fitting and tracking method provided in the above embodiments.

[0122] Furthermore, electronic devices also include:

[0123] Communication interface 503 is used for communication between memory 501 and processor 502.

[0124] The memory 501 is used to store computer programs that can run on the processor 502.

[0125] The memory 501 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0126] If the memory 501, processor 502, and communication interface 503 are implemented independently, then the communication interface 503, memory 501, and processor 502 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation,Figure 5 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0127] Optionally, in a specific implementation, if the memory 501, processor 502, and communication interface 503 are integrated on a single chip, then the memory 501, processor 502, and communication interface 503 can communicate with each other through an internal interface.

[0128] Processor 502 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0129] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the roadside fitting and tracking method described above.

[0130] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0131] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0132] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0133] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0134] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0135] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0136] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0137] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A method for fitting and tracking the roadside line of a road, characterized in that, Includes the following steps: Scan the region of interest (ROI) within the drivable area of ​​the vehicle to obtain roadside segmentation data points; Interpolation processing is performed on the roadside segmentation data points to obtain multiple roadside points; Based on the preset extrinsic calibration results, the multiple roadside points are projected from image coordinates to bird's-eye view coordinates; The coordinates of the bird's-eye view are fitted with roadside lines to obtain multiple fitting parameters. The multiple fitting parameters are then traversed in a preset order to obtain the vanishing coordinate points. The real-time pitch angle and real-time yaw angle of the vehicle are calculated based on the vanishing coordinates and the resolution of the image, and a new roadside point is fitted based on the real-time pitch angle and the real-time yaw angle. as well as The new roadside points are back-projected onto the image coordinates, and the roadside state is predicted based on the back-projected image coordinates to obtain the roadside fitting line, thereby completing the tracking of the roadside. Before interpolating the roadside segmentation data points, the process also includes: Determine whether the number of roadside segmentation data points is even and greater than a preset number; If the number of roadside segmentation data points is even and greater than the preset number, then the roadside segmentation data points are interpolated using a cubic Bézier curve; otherwise, no interpolation is performed.

2. The method according to claim 1, characterized in that, The cubic Bézier curve is: ； Where n=3, t∈[0,1], For the first One control point.

3. The method according to claim 1, characterized in that, The calculation of the vehicle's real-time pitch angle and real-time yaw angle based on the vanishing coordinates and the image resolution includes: Based on a preset angle calculation formula, the real-time pitch angle and real-time yaw angle of the vehicle are calculated according to the vanishing coordinate point and the resolution of the image. The preset angle calculation formula is as follows: pitch = vanish_point.y - image.rows / 2; yaw = vanish_point.x - image.cols / 2; Where pitch is the pitch angle, yaw is the yaw angle, vanish_point.y is the vanishing point in the y direction, vanish_point.x is the vanishing point in the x direction, image.rows is the number of image rows, and image.cols is the number of image columns.

4. The method according to claim 1, characterized in that, Fitting the roadside line to the coordinates of the bird's-eye view includes: Construct the first matrix and the second matrix; Based on the first matrix and the second matrix, a cubic polynomial is used to fit the roadside lines to the coordinates of the bird's-eye view to obtain the multiple fitting parameters.

5. The method according to claim 4, characterized in that, The cubic polynomial is: ； Where c0 is the lateral offset of the near end of the roadside line relative to the center of the vehicle body, c1 is the orientation of the roadside line, c2 is the curvature of the roadside line, and c3 is the rate of change of curvature.

6. A roadside fitting and tracking device, used to implement the roadside fitting and tracking method as described in any one of claims 1-5, characterized in that, include: The scanning module is used to scan the region of interest (ROI) within the drivable area of ​​the vehicle to obtain roadside segmentation data points. The processing module is used to perform interpolation processing on the roadside segmentation data points to obtain multiple roadside points; The projection module is used to project the multiple roadside points from image coordinates to bird's-eye view coordinates based on the preset extrinsic calibration results. The fitting module is used to fit the coordinates of the bird's-eye view to the roadside line, obtain multiple fitting parameters, and traverse the image according to the multiple fitting parameters in a preset order to obtain the vanishing coordinate points. The calculation module is used to calculate the real-time pitch angle and real-time yaw angle of the vehicle based on the vanishing coordinate point and the resolution of the image, and to fit a new roadside point based on the real-time pitch angle and the real-time yaw angle. as well as The tracking module backprojects the new roadside points onto the image coordinates and predicts the roadside state based on the backprojected image coordinates to obtain a roadside fitting line, thereby completing the tracking of the roadside.

7. The apparatus according to claim 6, characterized in that, The processing module is further configured to: Determine whether the number of roadside segmentation data points is even and greater than a preset number; If the number of roadside segmentation data points is even and greater than the preset number, then the roadside segmentation data points are interpolated using the cubic Bézier curve. Otherwise, no interpolation operation will be performed.

8. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the program to implement the roadside fitting and tracking method for a road as described in any one of claims 1-5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the roadside fitting and tracking method as described in any one of claims 1-5.

Images