Roadside fitting prediction method, device, system, equipment and storage medium

Abstract

The present application provides a roadside fitting prediction method, apparatus, system, equipment and storage medium. The method obtains the image positions of multiple roadside points in the current drivable area, determines the vehicle body position of the roadside points according to the image positions and vehicle body image mapping parameters, obtains the vehicle body position of each roadside point, obtains two initial fitting roadside lines based on the vehicle body position fitting of each roadside point, determines the vanishing point position in the drivable area at the current moment, determines the vehicle body pitch angle and vehicle body yaw angle according to the vanishing point position and the image resolution of the current road image, uses the vehicle body position of each roadside point as an observation quantity, uses the vanishing point position, vehicle body pitch angle and vehicle body yaw angle as control variables, tracks and predicts the two initial fitting roadside lines through an extended Kalman filter model, and obtains two predicted fitting roadside lines. The method improves the accuracy of roadside coordinate information on uphill and downhill sections, and more accurately predicts the driving status of the vehicle and the actual road conditions.

Description

Technical Field

[0001] The present application relates to the field of vehicle technology, and in particular to a roadside fitting prediction method, device, system, equipment and storage medium. Background Art

[0002] Lane curve fitting is necessary to better understand and predict the surrounding environment during driving. Curve fitting of collected lane data can effectively estimate lane parameter information, thereby predicting the lane direction and providing assistance to the driver or the vehicle's automatic control system.

[0003] Conventional solutions only fit lane lines on a two-dimensional plane using multiple lane line points. This can lead to significant errors on urban roads with frequent uphill and downhill sections. The resulting lane line coordinates are inaccurate, making it difficult to accurately predict the vehicle's driving status and the actual road conditions. Furthermore, for special conditions like flooded roads, simply fitting lane lines on a two-dimensional plane results in poor lane line fitting accuracy, resulting in a poor user experience. Summary of the Invention

[0004] In view of the shortcomings of the existing technology mentioned above, the present invention provides a roadside line fitting prediction method, device, system, equipment and storage medium to solve the technical problem in the related technology that the lane line coordinate information fitted in uphill and downhill sections is inaccurate and cannot accurately predict the driving status of the vehicle and the actual road conditions.

[0005] The embodiment of the present invention provides a roadside line fitting prediction method, which includes:

[0006] Obtaining image positions of a plurality of roadside points in a current drivable area, wherein the current drivable area is determined based on a current road image;

[0007] determining the vehicle body position of the roadside point according to the image position and a vehicle body image mapping parameter, thereby obtaining the vehicle body position of each roadside point; wherein the vehicle body image mapping parameter is used to represent a position mapping relationship between the image position of the image and the vehicle body position;

[0008] Two initial fitted roadside lines are obtained based on the vehicle body position fitting at each of the roadside line points, and the vanishing point position in the drivable area at the current moment is determined;

[0009] determining a vehicle body pitch angle and a vehicle body yaw angle according to the vanishing point position and the image resolution of the current road image;

[0010] The vehicle body position at each roadside point is used as an observation quantity, and the vanishing point position, the vehicle body pitch angle, and the vehicle body yaw angle are used as control variables. The two initial fitted roadside lines are predicted using an extended Kalman filter model to obtain two predicted fitted roadside lines.

[0011] In one embodiment of the present invention, before obtaining the image positions of multiple roadside points in the current drivable area, the roadside fitting prediction method includes:

[0012] Inputting the current road image into a preset drivable area detection model to determine a current drivable area in the current road image, and determining image labels of a plurality of image points in the current drivable area, wherein the image labels include static boundary labels and dynamic boundary labels;

[0013] interpolating the image points of the image labels including the static boundary labels using a third-order Bezier curve function to obtain interpolation points;

[0014] The interpolation point and the image point are determined as roadside points.

[0015] In one embodiment of the present invention, before interpolating the image points whose image labels include static boundary labels using a third-order Bezier curve function, the roadside line fitting prediction method includes:

[0016] The region of interest in the current drivable area is scanned row by row from bottom to top, the image points whose image labels are dynamic boundary labels are filtered out, and the image points whose image labels include static boundary labels are retained.

[0017] In one embodiment of the present invention, the method for determining the region of interest includes:

[0018] A=rows–a, B=rows / b1–b2;

[0019] Wherein, A is the scanning start pixel row of the region of interest, B is the scanning end pixel row of the region of interest, rows is the total number of image pixel rows of the current road image, a is the preset first threshold, b1 is the preset second threshold, b2 is the preset third threshold, and 0<A<B<rows.

[0020] In one embodiment of the present invention, interpolating the image points of the image labels including the static boundary labels using a third-order Bezier curve function includes:

[0021] Obtaining the number of image points of the image points whose image tags include static boundary tags;

[0022] If the number of the image points is greater than 4 and is an even number, the image points of the image labels including the static boundary labels are interpolated using a third-order Bezier curve function.

[0023] In one embodiment of the present invention, obtaining an initial fitted roadside line based on the vehicle body position fitting of each roadside point includes:

[0024] constructing a first matrix and a second matrix according to the vehicle body position information of each of the roadside points located on one side of the vehicle body, and determining a fitting parameter matrix based on the first matrix and the second matrix to obtain fitting parameters;

[0025] An initial fitting roadside line is generated according to the fitting parameters.

[0026] In one embodiment of the present invention, the first matrix includes:

[0027] X=[N P0.y+P1.y+..+PN-1.y (P0.y)^2+(P1.y)^2+..+(PN-1.y)^2 (P0.y)^3+(P1.y)^3+..+(PN-1.y)^3 P0.y+P1.y+..+PN-1.y (P0.y)^2+(P1.y)^2+..+(PN-1.y)^2(P0.y)^3+(P1.y)^3+..+(PN-1.y)^3 (P0.y)^4+(P1.y)^4+..+(PN-1.y)^4 (P0.y)^2+(P1.y)^2+..+(PN-1.y)^2 (P0.y)^3+(P1.y)^3+..+(PN-1.y)^3 (P0.y)^4+(P1.y)^4+..+(PN-1.y)^4 (P0.y)^5+(P1.y)^5+..+(PN-1.y)^5 (P0.y)^3+(P1.y)^3+..+(PN-1.y)^3(P0.y)^4+(P1.y)^4+..+(PN-1.y)^4 (P0.y)^5+(P1.y)^5+..+(PN-1.y)^5 (P0.y)^6+(P1.y)^6+..+(PN-1.y)^6]

[0028] Wherein, X is the first matrix, N is the number of roadside points, P0.y is the vehicle body side coordinate value of the P0 roadside point, P1.y is the vehicle body side coordinate value of the P1 roadside point, and PN-1.y is the vehicle body side coordinate value of the PN-1 roadside point.

[0029] In one embodiment of the present invention, the second matrix includes:

[0030] Y=[P0.x+P1.x+...+PN-1.x P0.y*P0.x+P1.y*P1.x+...+PN-1.y*PN-1.x(P0.y)^2*P0.x+(P1.y)^2*P1.x+.. +(PN-1.y)^2*PN-1.x(P0.y)^3*P0.x+(P1.y)^3*P1.x+..+(PN-1.y)^3*PN-1.x]

[0031] Among them, Y is the second matrix, P0.x is the front coordinate value of the vehicle body of the point along the road P0, P1.x is the front coordinate value of the vehicle body of the point along the road P1, PN-1.x is the front coordinate value of the vehicle body of the point along the road PN-1, P0.y is the side coordinate value of the vehicle body of the point along the road P0, P1.y is the side coordinate value of the vehicle body of the point along the road P1, and PN-1.y is the side coordinate value of the vehicle body of the point along the road PN-1.

[0032] In one embodiment of the present invention, determining a fitting parameter matrix based on the first matrix and the second matrix includes:

[0033] AX=Y,

[0034] Where X is the first matrix, Y is the second matrix, and A is the fitting parameter matrix.

[0035] In one embodiment of the present invention, the fitting parameters include a lateral offset parameter of the proximal end of the roadside relative to the center of the vehicle body, a route orientation parameter, a roadside curvature parameter, and a curvature change rate parameter. The initial fitting roadside line includes:

[0036] x=c0+c1y+c2y 2 +c3y 3 ,

[0037] Where c0 is the lateral offset parameter of the near end of the curb relative to the center of the vehicle body, c1 is the route orientation parameter, c2 is the curb curvature parameter, c3 is the curvature change rate parameter, x is the front coordinate of the vehicle body in the vehicle body coordinate system, and y is the side coordinate of the vehicle body in the vehicle body coordinate system.

[0038] In one embodiment of the present invention, the method for determining the fitting parameters includes:

[0039] c0=A(0,0);

[0040] c1=A(1,0);

[0041] c2=A(2,0);

[0042] c3=A(3,0);

[0043] Among them, c0 is the lateral offset parameter of the near end of the curb relative to the center of the vehicle body, c1 is the route orientation parameter, c2 is the curb curvature parameter, c3 is the curvature change rate parameter, and A is the fitting parameter matrix.

[0044] In one embodiment of the present invention, determining the position of the vanishing point in the drivable area at the current moment includes:

[0045] Accumulating the distance between the two initially fitted roadside lines to obtain a cumulative value until the cumulative value reaches a preset range, the distance being determined based on the vehicle body front coordinates of a roadside point pair on the two initially fitted roadside lines, the roadside point pair being two roadside points having the same vehicle body side coordinates;

[0046] A target vehicle body position corresponding to the accumulated value is determined as a vanishing point position, wherein the target vehicle body position is a vehicle body position corresponding to a minimum distance among the accumulated distances.

[0047] In one embodiment of the present invention, determining the vehicle body pitch angle and the vehicle body yaw angle according to the vanishing point position and the image resolution of the current road image includes:

[0048] pitch=vanish_point.y-rows / 2;

[0049] yaw=vanish_point.x-cols / 2;

[0050] Wherein, pitch is the pitch angle, yaw is the yaw angle, vanish_point.y is the side coordinate of the vanishing point, vanish_point.x is the front coordinate of the vanishing point, rows is the total number of image pixel rows of the current road image, cols is the total number of image pixel columns of the current road image, and the image resolution includes the total number of image pixel rows and the total number of image pixel columns.

[0051] In one embodiment of the present invention, after obtaining two predicted fitting roadside lines, the roadside line fitting prediction method further includes:

[0052] Matching the initial fitting roadside line and the predicted fitting roadside line through a Hungarian algorithm model;

[0053] If the matching is successful, the roadside tracking of the current road image is completed.

[0054] In one embodiment of the present invention, after obtaining two predicted fitting roadside lines, the roadside line fitting prediction method further includes:

[0055] The predicted fitting roadside lines are displayed on the current road image based on the vehicle body image mapping parameters.

[0056] An embodiment of the present invention further provides a roadside line fitting prediction device, the roadside line fitting prediction device comprising:

[0057] An acquisition module, configured to acquire image positions of a plurality of roadside points in a current drivable area, wherein the current drivable area is determined based on a current road image;

[0058] a position conversion module, configured to determine the vehicle body position of the roadside point based on the image position and vehicle body image mapping parameters, thereby obtaining the vehicle body position of each roadside point;

[0059] An initial fitting module, configured to obtain two initial fitting roadside lines based on the vehicle body position fitting of each of the roadside line points;

[0060] a vanishing point determination module, configured to determine a vanishing point position in the drivable area at the current moment;

[0061] a vehicle body pitch angle and vehicle body yaw angle determination module, configured to determine the vehicle body pitch angle and vehicle body yaw angle according to the vanishing point position and the image resolution of the current road image;

[0062] The prediction fitting module is used to use the vehicle body position of each roadside point as an observation quantity, the vanishing point position, the vehicle body pitch angle and the vehicle body yaw angle as control variables, and predict the two initial fitting roadside lines through an extended Kalman filter model to obtain two predicted fitting roadside lines.

[0063] An embodiment of the present invention provides an electronic device, comprising:

[0064] one or more processors;

[0065] A storage device is used to store one or more programs, and when the one or more programs are executed by the one or more processors, the electronic device implements the roadside line fitting prediction method as described in any of the above embodiments.

[0066] In a ninth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a computer processor, the computer executes the roadside line fitting prediction method as described in any one of the above embodiments.

[0067] Beneficial effects of the present invention: The roadside fitting prediction method, apparatus, system, device, and storage medium of the present invention obtain image positions of multiple roadside points in a current drivable area, determine the vehicle body position at the roadside points based on the image positions and vehicle body image mapping parameters, obtain the vehicle body position at each roadside point, obtain two initial fitted roadside lines based on the vehicle body position fitting of each roadside point, determine the vanishing point position in the current drivable area, determine the vehicle body pitch angle and vehicle body yaw angle based on the vanishing point position and the image resolution of the current road image, use the vehicle body position at each roadside point as an observation, use the vanishing point position, vehicle body pitch angle, and vehicle body yaw angle as control variables, track and predict the two initial fitted roadside lines using an extended Kalman filter model, and obtain two predicted fitted roadside lines. By determining the vehicle body pitch angle and vehicle body yaw angle, the accuracy of roadside coordinate information fitted for uphill and downhill sections is improved. The method is particularly suitable for accurate roadside fitting of urban roads with many uphill and downhill sections and sections with water accumulation. The method can accurately predict the driving state of the vehicle and the actual road surface conditions, resulting in more accurate roadside fitting, which is conducive to improving user experience.

[0068] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] The accompanying drawings are incorporated into and constitute a part of the specification, illustrating embodiments consistent with the present application and, together with the specification, serving to explain the principles of the present application. It is obvious that the drawings described below are merely some embodiments of the present application, and a person of ordinary skill in the art can derive other drawings based on these drawings without inventive effort. In the drawings:

[0070] Figure 1 1 is a schematic diagram of an implementation environment of a roadside line fitting prediction method according to an exemplary embodiment of the present application;

[0071] Figure 2 1 is a flow chart of a roadside fitting prediction method according to an exemplary embodiment of the present application;

[0072] Figure 3 is a schematic diagram of a vehicle body coordinate system shown in an exemplary embodiment of the present application;

[0073] Figure 4 This is a specific flow chart of a roadside line fitting prediction method shown in an exemplary embodiment of the present application;

[0074] Figure 5 is a block diagram of a roadside line fitting prediction device shown in an exemplary embodiment of the present application;

[0075] Figure 6 A schematic diagram of the structure of a computer system suitable for implementing an electronic device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0076] The following describes the embodiments of the present invention with reference to the accompanying drawings and preferred embodiments. Those skilled in the art will readily appreciate the other advantages and benefits of the present invention from the disclosure herein. The present invention may also be implemented or applied through various other specific embodiments, and the various details in this specification may be modified or altered based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are intended only to illustrate the present invention and are not intended to limit the scope of protection of the present invention.

[0077] It should be noted that the illustrations provided in the following embodiments are merely schematic illustrations of the basic concept of the present invention. Therefore, the illustrations only show components related to the present invention and are not drawn according to the number, shape, and size of components in actual implementation. In actual implementation, the type, quantity, and proportion of each component may be changed arbitrarily, and the component layout may also be more complex.

[0078] In the following description, numerous details are discussed to provide a more thorough explanation of the embodiments of the present invention. However, it will be apparent to those skilled in the art that the embodiments of the present invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring the embodiments of the present invention.

[0079] Figure 1 FIG. 1 is a schematic diagram of an implementation environment for the roadside fitting prediction system service activation management shown in an exemplary embodiment of the present application. Figure 1 As shown, the roadside fitting prediction system includes a client 102 and a server 101, wherein the server 101 may include an independently operated server, a distributed server, or a server cluster composed of multiple servers. The server 101 may include a network communication unit, a processor, a memory, etc. The server 101 can be used to execute the roadside fitting prediction method provided in this embodiment. The client 102 may include physical devices of the type of smart phones, desktop computers, tablet computers, laptops, digital assistants, smart wearable devices, vehicle-mounted terminals, etc., and may also include software running in the physical devices, such as web pages provided to users by some service providers, or applications provided to users by these service providers. The client 102 can also be used to execute the roadside fitting prediction method provided in this embodiment, or to execute the roadside fitting prediction method provided in this embodiment through the interaction between the server and the client. This is not limited here.

[0080] Lane curve fitting is necessary to better understand and predict the surrounding environment during driving. Curve fitting of collected lane data can effectively estimate lane parameter information, thereby predicting the lane direction and providing assistance to the driver or the vehicle's automatic control system.

[0081] In the related technical solutions, the lane line fitting is only performed without considering the tracking problem. It cannot eliminate the inaccurate line coordinate information caused by the influence of road surface changes. In addition, the lane line coordinate information fitted on uphill and downhill sections is inaccurate, which makes it impossible to accurately predict the driving status of the vehicle and the actual road conditions.

[0082] To solve these problems, the embodiments of the present application respectively propose a roadside line fitting prediction method, a roadside line fitting prediction device, an electronic device, a computer-readable storage medium and a computer program product. These embodiments will be described in detail below.

[0083] See also Figure 2 , Figure 2 This is a flowchart of a roadside line fitting prediction method shown in an exemplary embodiment of the present application. This method can be applied to Figure 1 The implementation environment shown is specifically implemented by the server 101 and / or the client 102 in the implementation environment. It should be understood that the method can also be applied to other exemplary implementation environments and specifically implemented by devices in other implementation environments. This embodiment does not limit the implementation environment to which the method is applicable.

[0084] like Figure 2 As shown, in an exemplary embodiment, the roadside line fitting prediction method includes at least steps S201 to S205, which are described in detail as follows:

[0085] Step S201: Acquire image positions of multiple roadside points in the current drivable area.

[0086] The current drivable area is determined based on the current road image. The current road image can be captured by an image acquisition device such as a vehicle camera. The current road image can be a video frame in a driving video. By processing multiple video frames in the driving video using the method provided in this embodiment, a road driving image with the predicted and fitted roadside line can be obtained, facilitating driving decisions for the driver, such as reversing, turning, or moving forward.

[0087] The image position in this embodiment is the relative position of objects such as roadside points in the current road image, that is, the image position of each roadside point is determined by the image coordinate system of the current road image. The vehicle body position in subsequent embodiments is the relative position of objects such as roadside points in the vehicle body coordinate system, that is, the vehicle body position of each roadside point is determined by the real-world coordinate system of the vehicle body. The image acquisition device of the current road image can be used to calibrate the external parameters relative to the vehicle body to calculate the homography matrix of each roadside point projected from the image coordinate system to the vehicle body coordinate system, thereby realizing the projection of the image coordinate system to the vehicle body bird's-eye view coordinate system (vehicle body coordinate system). The vehicle body coordinate system can be found in Figure 3 , Figure 3 It is a schematic diagram of a vehicle body coordinate system shown in an exemplary embodiment of the present application, which takes the center of the rear axle of the vehicle as the origin O, the front of the vehicle as X positive, and the left side of the vehicle as Y positive, and moves with the vehicle (body).

[0088] The curb marks the boundaries of the current vehicle's (the host vehicle's) drivable area. When both lanes are occupied, meaning only the host vehicle's lane is passable, the lane markings may be obstructed by other vehicles or obstacles. Therefore, the curb marks can also represent the boundaries of the current drivable area, i.e., the boundaries formed by obstacles surrounding the host vehicle. There are curb marks on both the left and right sides of the host vehicle. A curb mark can also represent lane markings or a line connecting raised curbs on the road.

[0089] In one embodiment, the current road image can be collected by the front-view camera of the vehicle, and the drivable area is obtained by detecting a preset drivable area detection model. Each image point in the detection result of the preset drivable area has an image label, and the image label includes a static label and a dynamic label. The static boundary label is used to characterize that the image point is an image point of a fixed object such as a curb or road, and the dynamic label is an image point of a movable object such as a vehicle.

[0090] In one embodiment, before obtaining the image positions of multiple roadside points in the current drivable area, the roadside fitting prediction method includes:

[0091] Inputting the current road image into a preset drivable area detection model to determine the current drivable area in the current road image, and determining image labels for multiple image points in the current drivable area, wherein the image labels include static boundary labels and dynamic boundary labels. The preset drivable area detection model can adopt a model known to those skilled in the art. Through the preset drivable area detection model, the drivable area of ​​the vehicle can be detected and the boundary of the drivable area can be determined. Image labels are annotated for image points on the boundary of the drivable area to obtain image labels for each image point;

[0092] Interpolating image points of image labels including static boundary labels using a third-order Bezier curve function to obtain interpolation points;

[0093] The interpolation points and image points are determined as roadside points.

[0094] In one embodiment, before interpolating image points of image labels including static boundary labels using a third-order Bezier curve function, the roadside fitting prediction method includes:

[0095] The area of ​​interest in the current drivable area is scanned line by line from bottom to top, image points with dynamic boundary labels are filtered out, and image points with static boundary labels are retained.

[0096] In one embodiment, the region of interest is determined by:

[0097] A=rows–a, B=rows / b1–b2;

[0098] Wherein, A is the scanning start pixel row of the region of interest, B is the scanning end pixel row of the region of interest, rows is the total number of image pixel rows of the current road image, a is the preset first threshold, b1 is the preset second threshold, b2 is the preset third threshold, and 0<A<B<rows.

[0099] In one embodiment, interpolating image points of image labels including static boundary labels using a third-order Bezier curve function includes:

[0100] Obtain the number of image points of image labels including image points of static boundary labels;

[0101] If the number of image points is greater than 4 and is an even number, the image points of the image labels including the static boundary labels are interpolated using a third-order Bezier curve function.

[0102] For example, the drivable area result information output by the perception algorithm can be used to scan the image points (pixels) in the drivable area from the bottom upward (scanning from the road to the sky). Because the image captured by the front camera of the ego vehicle (the vehicle) partly covers the hood area, to eliminate redundant traversal, the scanning ROI area (region of interest) is set as:

[0103] Scan starting point = image rows – 90,

[0104] Scan end point = image rows / 2–50;

[0105] Here, image rows is the total number of rows of image pixels in the current road image. Those skilled in the art can adjust “90, 50, 2” in the formula as needed and are not limited here.

[0106] During scanning, it is necessary to distinguish between static and dynamic boundary points. Dynamic boundary points are directly filtered; static boundary points are further processed using the cubic Bezier curve algorithm for nearest neighbor interpolation to provide sufficient point coordinates for subsequent roadside fitting.

[0107] The roadside segmentation data points (image labels including those with static boundary labels) output by the preset drivable area detection model are interpolated using a cubic Bezier curve, primarily to provide a preliminary description of the roadside's trajectory and curvature. It should be noted that the following method is only called when the number of roadside points (image labels including those with static boundary labels) is an even number and greater than or equal to 4. The principle behind the cubic Bezier curve is as follows:

[0108]

[0109] Where n = 3, t∈[0,1], i = 0, 1, 2, 3, B(t) is the coordinate of the interpolation point, t i is the coordinate of the image point, P i is the coordinate of the control point, when i=1, t i =t 1 =0, i=N when t i =t N =1, when i is between 1 and N, t i -t i-1 =△t 1 .

[0110] In one embodiment, formula (1) can be transformed into:

[0111] B(t)=P0*(1-3t+3t^2-t^3)+P1*(3t-6t^2+3t^3)+P2*(3t^2-3t^3)+P3*t^3Formula (2);

[0112] Among them, B(t) is the coordinate of the interpolation point, P0 is the coordinate of the first control point, P1 is the coordinate of the second control point, P3 is the coordinate of the third control point, P4 is the coordinate of the fourth control point, t∈[0,1], n=4, i=0,1,2,3, respectively substitute into the formula to obtain the coordinates of the interpolation point. When t=0, point=P0, when t=1, point=P3. This relationship is a constant.

[0113] This method can be used to determine the image positions of roadside lines and roadside points on them. When the perception algorithm outputs a relatively small number of roadside point coordinates (including image points with static boundary labels), an interpolation algorithm (third-order Bezier curve function) can be used to supplement some of the point coordinates (interpolation points) for subsequent line fitting, improving fitting accuracy.

[0114] Step S202 : determining the vehicle body position of each roadside point according to the image position and the vehicle body image mapping parameter, and obtaining the vehicle body position of each roadside point.

[0115] The vehicle body image mapping parameter is used to represent the position mapping relationship between the image position of the image and the vehicle body position of the vehicle body. The vehicle body image mapping parameter may be the following homography matrix.

[0116] In one embodiment, based on the external parameter calibration results of the image acquisition device such as the front-view camera that captures the current road image relative to the vehicle (host vehicle) (for example, the positional relationship between the image acquisition device and the center of the rear axle of the host vehicle), the homography matrix of these points projected from the image coordinate system to the vehicle body coordinate system is calculated to achieve the projection of the image coordinate system to the vehicle body bird's-eye view coordinate system. The vehicle body coordinate system takes the center of the rear axle of the vehicle as the origin, the front of the vehicle as the positive X point, and the left side of the vehicle as the positive Y point. As the host vehicle moves, the coordinate system can be seen in the following figure: Figure 3 .

[0117] A homography matrix is ​​used to describe the perspective transformation between a real-world plane and its corresponding image, as well as to transform an image from one view to another through perspective transformation. In computer vision, a plane homography is defined as the projective mapping of one plane onto another. A homography matrix can be understood as representing the mapping matrix between the vehicle coordinate system and the image coordinate system, or the mapping matrix between a plane defined by the vehicle coordinate system and a plane defined by the image coordinate system.

[0118] Step S203 , obtaining two initial fitted roadside lines based on the vehicle body position fitting of each roadside point, and determining the position of the vanishing point in the drivable area at the current moment.

[0119] In one embodiment, the vanishing point may be a vehicle body position at a roadside point.

[0120] In one embodiment, obtaining an initial fitted roadside line based on the vehicle body position fitting of each roadside point includes:

[0121] Constructing a first matrix and a second matrix according to the vehicle body position information of each point along the road located on one side of the vehicle body, and determining a fitting parameter matrix based on the first matrix and the second matrix to obtain fitting parameters;

[0122] Generate an initial fitting roadside line according to the fitting parameters.

[0123] The first matrix includes:

[0124] X=[N P0.y+P1.y+..+PN-1.y (P0.y)^2+(P1.y)^2+..+(PN-1.y)^2 (P0.y)^3+(P1.y)^3+..+(PN-1.y)^3 P0.y+P1.y+..+PN-1.y (P0.y)^2+(P1.y)^2+..+(PN-1.y)^2(P0.y)^3+(P1.y)^3+..+(PN-1.y)^3 (P0.y)^4+(P1.y)^4+..+(PN-1.y)^4 (P0.y)^2+(P1.y)^2+..+(PN-1.y)^2 (P0.y)^3+(P1.y)^3+..+(PN-1.y)^3 (P0.y)^4+(P1.y)^4+..+(PN-1.y)^4 (P0.y)^5+(P1.y)^5+..+(PN-1.y)^5 (P0.y)^3+(P1.y)^3+..+(PN-1.y)^3(P0.y)^4+(P1.y)^4+..+(PN-1.y)^4 (P0.y)^5+(P1.y)^5+..+(PN-1.y)^5 (P0.y)^6+(P1.y)^6+..+(PN-1.y)^6] Formula (3);

[0125] Where X is the first matrix, N is the number of roadside points, P0.y is the lateral coordinate value of the P0 roadside point, P1.y is the lateral coordinate value of the P1 roadside point, and PN-1.y is the lateral coordinate value of the PN-1 roadside point. N = (P0.y)^0 + (P1.y)^0 + ... + (PN-1.y)^0.

[0126] In one embodiment, the first matrix can also be expressed as:

[0127] X=[(P0.y )^ 0+(P1.y)^0+..+(PN-1.y)^0 |P0.y+P1.y+..+PN-1.y (P0.y )^ 2+(P1.y)^2+.. +( PN-1.y)^2 (P0.y)^3+(P1.y)^3+.. +( PN-1.y)^3 P0.y+P1.y+ .. +PN-1.y(P0.y )^ 2+(P1.y)^2+.. +( PN-1.y)^2 (P0.y)^3+(P1.y)^3+.. +( PN-1.y)^3 (P0.y)^4+(P1.y)^4+..+( PN-1.y)^4 (P0.y)^2 + (P1.y)^2+..+(PN-1.y)^2 (P0.y)^3+(P1.y)^3+.. + ( PN-1.y)^3 (P0.y)^4+(P1.y)^4+.. +( PN-1.y)^4 (P0.y)^5+(P1.y)^5+.. +( PN-1.y)^5(P0.y)^3 +( P1.y)^3+..+(PN-1.y)^3 (P0.y)^4+(P1.y)^4+.. +( PN-1.y)^4 (P0.y)^5+(P1.y)^5+.. +( PN-1.y)^5 (P0.y)^6+(P1.y)^6+.. +( PN-1.y)^6] Formula (4);

[0128] Where X is the first matrix, P0.y is the lateral coordinate value of the vehicle body at point P0, P1.y is the lateral coordinate value of the vehicle body at point P1, and PN-1.y is the lateral coordinate value of the vehicle body at point PN-1. "..." represents the values ​​of the (N-2) points between P1 and PN-1, all of which use the same calculation method as P1.

[0129] In one embodiment, the first matrix can also be expressed as:

[0130]

[0131] Where X is the first matrix, Pi.y is the lateral coordinate value of the vehicle body along the Pi road, and i∈[0, N-1].

[0132] The second matrix includes:

[0133] Y=[P0.x+P1.x+...+PN-1.x P0.y*P0.x+P1.y*P1.x+...+PN-1.y*PN-1.x(P0.y)^2*P0.x+(P1.y)^2*P1.x+.. +(PN-1.y)^2*PN-1.x(P0.y)^3*P0.x+(P1.y)^3*P1.x+..+(PN-1.y)^3*PN-1.x] Formula (6);

[0134] Where Y is the second matrix, P0.x is the front coordinate value of the P0 roadside point, P1.x is the front coordinate value of the P1 roadside point, PN-1.x is the front coordinate value of the PN-1 roadside point, P0.y is the side coordinate value of the P0 roadside point, P1.y is the side coordinate value of the P1 roadside point, and PN-1.y is the side coordinate value of the PN-1 roadside point. P0...PN-1 are the point coordinates (x, y) output along the roadside, for a total of N points.

[0135] It should be noted that the initial fitted roadside line is the fitted roadside line of which book of the vehicle body, and the above-mentioned roadside points are the points on which side of the vehicle body.

[0136] In one embodiment, determining a fitting parameter matrix based on the first matrix and the second matrix includes:

[0137] AX=Y formula (7);

[0138] Where X is the first matrix, Y is the second matrix, and A is the fitting parameter matrix.

[0139] As shown in the above formula (3), the first matrix is ​​a 4*4 order matrix, and as shown in the above formula (6), the second matrix is ​​a 4*1 order matrix. The fitting parameter matrix A can be obtained by formula (7). The fitting parameters are determined based on at least a portion of the matrix elements in the fitting parameter matrix, where there are 4 fitting parameters.

[0140] In one embodiment, the fitting parameters include a lateral offset parameter of the proximal end of the roadside relative to the center of the vehicle body, a route orientation parameter, a roadside curvature parameter, and a curvature change rate parameter. The initial fitting of the roadside includes:

[0141] x=c0+c1y+c2y 2 +c3y 3 Formula (8);

[0142] Where c0 is the lateral offset parameter of the near end of the curb relative to the center of the vehicle body, c1 is the route orientation parameter, c2 is the curb curvature parameter, c3 is the curvature change rate parameter, x is the front coordinate of the vehicle body in the vehicle body coordinate system, and y is the side coordinate of the vehicle body in the vehicle body coordinate system.

[0143] For example, the fitting parameters are determined by:

[0144] c0=A(0,0) Formula (9);

[0145] c1=A(1,0) Formula (10);

[0146] c2=A(2,0) Formula (11);

[0147] c3=A(3,0) Formula (12);

[0148] Among them, c0 is the lateral offset parameter of the near end of the curb relative to the center of the vehicle body, c1 is the route orientation parameter, c2 is the curb curvature parameter, c3 is the curvature change rate parameter, and A is the fitting parameter matrix.

[0149] It should be noted that continuing Figure 3 Taking the coordinate system of as an example, c0 of the left roadside is less than c0 of the right roadside. Based on c0, the left and right attributes of the roadside can be distinguished.

[0150] In one embodiment, determining the location of the vanishing point in the drivable area at the current moment includes:

[0151] Accumulating the distance between the two initially fitted roadside lines to obtain a cumulative value until the cumulative value reaches a preset range. The distance is determined based on the vehicle body front coordinates of a roadside point pair on the two initially fitted roadside lines. The roadside point pair is two roadside points with the same vehicle body side coordinates. The preset range can be a preset threshold. Once the preset threshold is reached, the coordinates of the roadside point accumulated in the previous accumulation are used as the vanishing point coordinates. The preset range can also be a numerical interval. The coordinates of the roadside point corresponding to the last accumulation that reaches the numerical interval and does not exceed the numerical interval are used as the vanishing point coordinates.

[0152] The target vehicle body position corresponding to the accumulated value is determined as the vanishing point position. The target vehicle body position is the vehicle body position corresponding to the minimum distance in the accumulated distances.

[0153] The two initial fitting roadside equations on both sides of the vehicle body can be obtained by the above formula (3) to formula (12). Based on the initial fitting roadside equations, the vehicle body positions of the roadside points obtained in step S202 are filtered to obtain multiple roadside points located on the initial fitting roadside equations. Figure 3 The absolute values ​​of the differences in the Y values ​​of the points along the road (taking the coordinate system shown as an example) are accumulated. If the maximum value of the accumulated sum is within the set threshold range, the (x, y) coordinate traversed by the maximum value is used as the vanishing point coordinate vanish_point(x, y) of the current frame.

[0154] It should be noted that in the process of accumulating the distance between the two initial fitting paths to obtain the cumulative value, Figure 3 Taking the embodiment shown as an example, the distance determined from the roadside point with the smallest X value is accumulated toward the roadside point with the larger X value.

[0155] Step S204 : determining the vehicle body pitch angle and vehicle body yaw angle according to the vanishing point position and the image resolution of the current road image.

[0156] In one embodiment, determining the vehicle body pitch angle and vehicle body yaw angle according to the vanishing point position and the image resolution of the current road image includes:

[0157] pitch=vanish_point.y-rows / 2 formula (13);

[0158] yaw = vanish_point.x-cols / 2 Formula (14);

[0159] Where pitch is the pitch angle, yaw is the yaw angle, vanish_point.y is the side coordinate of the vanishing point, vanish_point.x is the front coordinate of the vanishing point, rows is the total number of image pixel rows in the current road image, cols is the total number of image pixel columns in the current road image, and the image resolution includes the total number of image pixel rows and the total number of image pixel columns.

[0160] It should be noted that, since the above-mentioned two initial fitting roadside lines are determined based on the image positions of multiple roadside points in the current drivable area determined by the current road image, the two initial fitting roadside lines in the roadside fitting prediction method provided in this embodiment can be updated in real time as needed (real-time update of fitting parameters), and the vanishing point is also updated accordingly, and the vehicle body pitch angle and vehicle body yaw angle are further updated accordingly. Therefore, the method provided in this embodiment can be applied to scenarios with higher real-time requirements.

[0161] In step S205, the vehicle body position at each roadside point is used as an observation quantity, and the vanishing point position, the vehicle body pitch angle, and the vehicle body yaw angle are used as control variables. The two initial fitted roadside lines are predicted using an extended Kalman filter model to obtain two predicted fitted roadside lines.

[0162] Among them, Kalman filter (KF), Kalman filter is an algorithm that uses the linear system state equation to optimally estimate the system state through the system input and output observation data. Since the observation data includes the influence of noise and interference in the system, the optimal estimation can also be regarded as a filtering process. The extended Kalman filter EKF (extended Kalman filter model) is a filtering method that can be used for nonlinear systems based on KF. EKF mainly linearizes the nonlinear system and then performs Kalman filtering. In an embodiment of the present application, EKF is used to filter the initial fitted roadside line in the vehicle body coordinate system to obtain a predicted fitted roadside line.

[0163] In one embodiment, after obtaining two predicted fitting roadside lines, the roadside line fitting prediction method further includes:

[0164] The Hungarian algorithm model is used to match the initial fitting roadside and the predicted fitting roadside;

[0165] If the matching is successful, the roadside tracking of the current road image is completed.

[0166] The predicted roadside lines and the true roadside lines are matched through the Hungarian fitting loss. The Hungarian algorithm effectively solves the matching problem. Then, the matching results are used to optimize the path-related regression loss to achieve roadside tracking.

[0167] In one embodiment, after obtaining two predicted fitting roadside lines, the roadside line fitting prediction method further includes:

[0168] Based on the vehicle body image mapping parameters, each predicted fitting roadside line is displayed on the current road image.

[0169] In this way, the predicted fitted roadside line can be displayed on the current road image for people such as drivers to understand and use for lane navigation, reversing images, etc.

[0170] The roadside fitting prediction method provided in this embodiment obtains the image positions of multiple roadside points in the current drivable area, determines the vehicle body position at the roadside points based on the image positions and vehicle body image mapping parameters, and obtains the vehicle body position of each roadside point. Two initial fitted roadside lines are obtained based on the vehicle body position fitting of each roadside point. The vanishing point position in the current drivable area is determined, and the vehicle body pitch angle and yaw angle are determined based on the vanishing point position and the image resolution of the current road image. The vehicle body position at each roadside point is used as an observation, and the vanishing point position, vehicle body pitch angle, and vehicle body yaw angle are used as control variables. The two initial fitted roadside lines are tracked and predicted using an extended Kalman filter model to obtain two predicted fitted roadside lines. By determining the vehicle body pitch angle and vehicle body yaw angle, the accuracy of the roadside coordinate information fitted on uphill and downhill sections is improved, and the driving state of the vehicle and the actual road conditions can be accurately predicted. In particular, the method provided in this embodiment can be applied to the accurate fitting of roadside lines on urban roads with many ups and downs and on roads with water accumulation. It can accurately predict the driving status of the vehicle and the actual road conditions. The roadside line fitting is more accurate, which is conducive to improving the user experience.

[0171] When the method of this embodiment is applied to continuous current road images, such as multiple frames (continuous or discontinuous) of driving video captured by a driving recorder, the real-time updating of the vanishing point, yaw angle, and pitch angle allows accurate prediction of the vehicle's driving state and the actual road surface conditions. Furthermore, by calculating the optimal solution for fitting and tracking weights, the final roadside line result is obtained, eliminating inaccurate line coordinate information caused by road surface changes and other factors.

[0172] The method provided in this embodiment, using a least-squares curve fitting method based on a cubic polynomial, not only effectively detects the location of the roadside but also predicts its curvature and trajectory in real time. Interpolation algorithms supplement the coordinates of some points for subsequent roadside fitting, effectively overcoming the impact of factors such as light intensity and road water on roadside detection. Real-time updates of the vehicle's pitch and yaw angles ensure more accurate lane coordinates on uphill and downhill sections, preventing the fitted far-end roadside line from deviating from the ground.

[0173] See also Figure 4 , Figure 4 This is a specific flow chart of a roadside line fitting prediction method shown in an exemplary embodiment of the present application. This method can be applied to Figure 4 The implementation environment shown is specifically implemented by the server 101 and / or the client 102 in the implementation environment. It should be understood that the method can also be applied to other exemplary implementation environments and specifically implemented by devices in other implementation environments. This embodiment does not limit the implementation environment to which the method is applicable.

[0174] like Figure 4 As shown, in an exemplary embodiment, the specific roadside line fitting prediction method includes at least steps S401 to S408, which are described in detail as follows:

[0175] Step S401: inputting data along the roads in the drivable area.

[0176] The perception algorithm (preset drivable area detection model) can detect the current road image captured by the vehicle's front-view camera. The output drivable area result information is scanned line by line from the bottom to the top. Because part of the image captured by the vehicle's front-view camera is the hood area, to eliminate redundant traversal, the scanning ROI area (region of interest) is set as:

[0177] Scan starting point = image rows – 90,

[0178] Scan end point = image rows / 2–50,

[0179] During scanning, it is necessary to distinguish between static and dynamic boundary points. If the boundary point is dynamic, it is directly filtered; if it is static, it is further processed in step S402, using the cubic Bezier curve algorithm to perform nearest neighbor interpolation on the static boundary point to provide sufficient point coordinates for subsequent roadside fitting.

[0180] Step S402: cubic Bezier preprocessing.

[0181] The model outputs roadside segmentation data points (static boundary points, image labels including image points with static boundary labels) and uses cubic Bezier curves for interpolation. This is primarily to provide a preliminary description of the roadside trend and curvature. This method is only called when the number of roadside points is even and greater than or equal to 4. The principle of the cubic Bezier curve can be found in formula (1) and will not be elaborated here.

[0182] Step S403: projecting the image coordinates to the bird's-eye view coordinates.

[0183] The difference points and the original image labels, including those for static boundary labels, are used as roadside points. Based on the extrinsic calibration results of the front-view camera relative to the vehicle, such as the positional relationship between the camera and the rear axle center, the homography matrix for projecting these roadside points from image coordinates to vehicle body coordinates is calculated. This allows for the projection of the image coordinate system onto the vehicle body's bird's-eye view coordinate system. The vehicle body coordinate system has the rear axle center as its origin, with the front of the vehicle as positive X and the left side of the vehicle as positive Y, and moves with the vehicle.

[0184] Step S404: fitting the roadside line with a cubic polynomial.

[0185] The curb is fitted using a cubic polynomial equation as shown in formula (8), where C0 is the lateral offset of the curb's proximal end relative to the center of the vehicle body. C1 is the curb's orientation. C2 is the curb's curvature. C3 is the curvature change rate.

[0186] Construct matrices X (such as formulas (3), (4) and (5)) and Y (such as formula (6)), where X is a 4*4 matrix and Y is a 4*1 matrix. By AX=Y, we can get the matrix A.

[0187] The fitting parameters c0 = A(0,0); c1 = A(1,0); c2 = A(2,0); c3 = A(3,0) can be obtained from the above formula.

[0188] Step S405: Update the vanishing point, pitch, and yaw.

[0189] By fitting the parameters c0, c1, c2, and c3, and traversing the current road image from bottom to top, we can get the (x, y) coordinates of the two roadside lines. Figure 3 Taking the coordinate system shown in the figure as an example, the absolute value of the difference between the y values ​​of two lines with the same x value is calculated and accumulated. If the maximum value of the accumulated sum is within the set threshold range, the (x, y) coordinate traversed by the maximum value is used as the vanishing point coordinate vanish_point(x, y) of the current frame.

[0190] The pitch angle and yaw angle of the vehicle can be calculated based on the vanishing point coordinates and image resolution. The specific calculation formula is as follows:

[0191] pitch = vanish_point.y-image.rows / 2.

[0192] yaw=vanish_point.x-image.cols / 2.

[0193] Because the two road lines are updated in real time, that is, the fitting parameters c0, c1, c2, and c3 are updated in real time, the vanishing points are also updated accordingly. Pitch and yaw are calculated from the vanishing points and are therefore also updated accordingly.

[0194] Step S406: fitting and tracking.

[0195] The fitted roadside points are assigned a one-to-one temporal correspondence using prediction and matching algorithms, ensuring that the fitted roadside points are consistent with those obtained from the mathematical prediction model. The left and right attributes of the roadside are distinguished based on the principle that c0 for the left roadside is less than c0 for the right roadside. The fitted roadside is then tracked using an extended Kalman filter (EKF), with the vehicle body information as the control variable and the observed variables being the pixel coordinates of the roadside in the current frame. The Kalman filter algorithm is used to predict the roadside state, and the Hungarian algorithm is used to match the corresponding roadside, completing the tracking process.

[0196] Step S407: back-project to image coordinates.

[0197] Since the predicted fitting roadside line obtained in the above manner is a curve in the vehicle body coordinate system, the predicted fitting roadside line can be converted into image coordinates in the image coordinate system according to the homography matrix obtained in the above steps.

[0198] Step S408: Draw a roadside fitting line in the original video frame.

[0199] Based on the image coordinates obtained in step S407 , a predicted fitting roadside line (roadside fitting line) is drawn in the current road image (original video frame).

[0200] The method provided in the above embodiment can, when the perception algorithm outputs relatively few roadside point coordinates, supplement a portion of the point coordinates through an interpolation algorithm for subsequent line fitting; calculate the optimal solution of the fitting and tracking weights to obtain the final roadside result, eliminating the problem of inaccurate line coordinate information caused by changes in the road surface, etc.; update the vanishing point, yaw angle, and pitch angle in real time to accurately predict the driving status of the vehicle and the actual road conditions.

[0201] Figure 5 This is a block diagram of a roadside line fitting prediction device shown in an exemplary embodiment of the present application. The device can be applied to Figure 1The device may also be applicable to other exemplary implementation environments and specifically configured in other devices. This embodiment does not limit the implementation environment to which the device is applicable.

[0202] like Figure 5 As shown, the exemplary roadside fitting prediction device 500 includes:

[0203] An acquisition module 501 is used to acquire image positions of multiple roadside points in a current drivable area, where the current drivable area is determined based on a current road image;

[0204] A position conversion module 502 is configured to determine the vehicle position of each roadside point based on the image position and the vehicle image mapping parameter, thereby obtaining the vehicle position of each roadside point. The vehicle image mapping parameter is configured to represent a position mapping relationship between the image position of the image and the vehicle position of the vehicle.

[0205] An initial fitting module 503 is used to obtain two initial fitting roadside lines based on the vehicle body position fitting of each roadside point;

[0206] a vanishing point determination module 504 for determining the location of a vanishing point in the drivable area at the current moment;

[0207] A vehicle body pitch angle and vehicle body yaw angle determination module 505 is used to determine the vehicle body pitch angle and vehicle body yaw angle according to the vanishing point position and the image resolution of the current road image;

[0208] The prediction fitting module 506 is used to use the vehicle body position at each roadside point as an observation quantity, the vanishing point position, the vehicle body pitch angle and the vehicle body yaw angle as control variables, and predict the two initial fitting roadside lines through the extended Kalman filter model to obtain two predicted fitting roadside lines.

[0209] It should be noted that the roadside fitting prediction device provided in the above embodiment is different from the above embodiment. Figure 2 The provided roadside fitting prediction method is based on the same concept. The specific manner in which each module and unit performs its operations has been described in detail in the method embodiments and will not be repeated here. In actual applications, the roadside fitting prediction device provided in the above embodiment can distribute the above functions to different functional modules as needed, that is, divide the internal structure of the device into different functional modules to complete all or part of the functions described above, and this is not limited here.

[0210] An embodiment of the present application also provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the electronic device implements the roadside line fitting prediction method provided in the above-mentioned embodiments.

[0211] Figure 6 The following is a schematic diagram showing the structure of a computer system suitable for implementing an electronic device according to an embodiment of the present application. Figure 6 The computer system 600 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0212] like Figure 6 As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 602 or the program loaded from the storage part 608 into the random access memory (RAM) 603, such as executing the method described in the above embodiment. Various programs and data required for system operation are also stored in the RAM 603. The CPU 601, ROM 602 and RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0213] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, and the like; an output section 607 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 608 including a hard disk; and a communication section 609 including a network interface card such as a LAN (Local Area Network) card or a modem. The communication section 609 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 605 as needed. Removable media 611, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 610 as needed, so that computer programs read therefrom can be installed into the storage section 608 as needed.

[0214] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 609, and / or installed from a removable medium 611. When the computer program is executed by the central processing unit (CPU) 601, the various functions defined in the system of the present application are executed.

[0215] It should be noted that the computer-readable medium shown in the embodiments of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, which carries a computer-readable computer program. This propagated data signal can take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. A computer program embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0216] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. Among them, each box in the flowchart or block diagram can represent a module, program segment, or part of the code, and the above-mentioned module, program segment, or part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0217] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. In some cases, the names of these units do not constitute limitations on the units themselves.

[0218] Another aspect of the present application provides a computer-readable storage medium having a computer program stored thereon. When executed by a computer processor, the computer program causes the computer to perform the aforementioned roadside line fitting prediction method. The computer-readable storage medium may be included in the electronic device described in the above embodiments, or may exist independently and not be incorporated into the electronic device.

[0219] Another aspect of the present application provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the roadside line fitting prediction method provided in each of the above embodiments.

[0220] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the present invention. Anyone skilled in the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, any equivalent modifications or alterations made by one of ordinary skill in the art without departing from the spirit and technical principles disclosed herein are intended to be covered by the claims of the present invention.

Claims

1. A roadside fitting prediction method, characterized in that: The roadside fitting prediction method includes: Obtaining image positions of a plurality of roadside points in a current drivable area, wherein the current drivable area is determined based on a current road image; determining the vehicle body position of the roadside point according to the image position and a vehicle body image mapping parameter, thereby obtaining the vehicle body position of each roadside point; wherein the vehicle body image mapping parameter is used to represent a position mapping relationship between the image position of the image and the vehicle body position; Two initial fitted roadside lines are obtained based on the vehicle body position fitting of each of the roadside line points, and a vanishing point position in the drivable area at the current moment is determined, wherein determining the vanishing point position in the drivable area at the current moment comprises: accumulating distances between the two initial fitted roadside lines to obtain a cumulative value until the cumulative value reaches a preset range, the distance being determined based on vehicle body front coordinates of a pair of roadside line points on the two initial fitted roadside lines, the pair of roadside line points being two roadside line points having the same vehicle body side coordinates; and determining a target vehicle body position corresponding to the cumulative value as the vanishing point position, the target vehicle body position being the vehicle body position corresponding to the minimum distance among the accumulated distances. determining a vehicle body pitch angle and a vehicle body yaw angle according to the vanishing point position and the image resolution of the current road image; The vehicle body position at each roadside point is used as an observation quantity, and the vanishing point position, the vehicle body pitch angle, and the vehicle body yaw angle are used as control variables. The two initial fitted roadside lines are predicted using an extended Kalman filter model to obtain two predicted fitted roadside lines.

2. The roadside fitting prediction method according to claim 1, wherein: Before obtaining the image positions of multiple roadside points in the current drivable area, the roadside fitting prediction method includes: Inputting the current road image into a preset drivable area detection model to determine a current drivable area in the current road image, and determining image labels of a plurality of image points in the current drivable area, wherein the image labels include static boundary labels and dynamic boundary labels; interpolating the image points of the image labels including the static boundary labels using a third-order Bezier curve function to obtain interpolation points; The interpolation point and the image point are determined as roadside points.

3. The roadside fitting prediction method according to claim 2, wherein: Before interpolating the image points of the image labels including the static boundary labels using a third-order Bezier curve function, the roadside fitting prediction method includes: The region of interest in the current drivable area is scanned row by row from bottom to top, the image points whose image labels are dynamic boundary labels are filtered out, and the image points whose image labels include static boundary labels are retained.

4. The roadside fitting prediction method according to claim 3, wherein: The method for determining the region of interest includes: A=rows–a, B=rows / b1–b2; Wherein, A is the scanning start pixel row of the region of interest, B is the scanning end pixel row of the region of interest, rows is the total number of image pixel rows of the current road image, a is the preset first threshold, b1 is the preset second threshold, b2 is the preset third threshold, and 0<A<B<rows.

5. The roadside fitting prediction method according to any one of claims 2 to 4, characterized in that: Performing interpolation processing on the image points including the static boundary labels using a third-order Bezier curve function includes: Obtaining the number of image points of the image points whose image tags include static boundary tags; If the number of the image points is greater than 4 and is an even number, the image points of the image labels including the static boundary labels are interpolated using a third-order Bezier curve function.

6. The roadside fitting prediction method according to claim 1, wherein: An initial fitting roadside line is obtained based on the vehicle position fitting of each point along the road, including: constructing a first matrix and a second matrix according to the vehicle body position information of each of the roadside points located on one side of the vehicle body, and determining a fitting parameter matrix based on the first matrix and the second matrix to obtain fitting parameters; An initial fitting roadside line is generated according to the fitting parameters.

7. The roadside fitting prediction method according to claim 6, wherein: The first matrix includes: Among them, X is the first matrix, Pi.y is the side coordinate value of the vehicle body along the Pi road, and i∈[0, N-1].

8. The roadside fitting prediction method according to claim 6, wherein: The second matrix includes: Among them, Y is the second matrix, Pi.x is the front coordinate value of the vehicle body at the point along the Pi road, Pi.y is the side coordinate value of the vehicle body at the point along the Pi road, and i∈[0, N-1].

9. The roadside fitting prediction method according to claim 6, wherein: Determining a fitting parameter matrix based on the first matrix and the second matrix includes: AX=Y, Where X is the first matrix, Y is the second matrix, and A is the fitting parameter matrix.

10. The roadside fitting prediction method according to any one of claims 6 to 9, characterized in that: The fitting parameters include a lateral offset parameter of the proximal end of the roadside relative to the center of the vehicle body, a route orientation parameter, a roadside curvature parameter, and a curvature change rate parameter. The initial fitting roadside line includes: x=c0+c1y+c2y 2 +c3y 3 , Where c0 is the lateral offset parameter of the near end of the curb relative to the center of the vehicle body, c1 is the route orientation parameter, c2 is the curb curvature parameter, c3 is the curvature change rate parameter, x is the front coordinate of the vehicle body in the vehicle body coordinate system, and y is the side coordinate of the vehicle body in the vehicle body coordinate system.

11. The roadside fitting prediction method according to claim 10, wherein: The determination method of the fitting parameters includes: c0=A(0,0); c1=A(1,0); c2=A(2,0); c3=A(3,0); Among them, c0 is the lateral offset parameter of the near end of the curb relative to the center of the vehicle body, c1 is the route orientation parameter, c2 is the curb curvature parameter, c3 is the curvature change rate parameter, and A is the fitting parameter matrix.

12. The roadside fitting prediction method according to any one of claims 1 to 4, characterized in that: Determining the vehicle body pitch angle and vehicle body yaw angle according to the vanishing point position and the image resolution of the current road image includes: pitch=vanish_point.y-rows / 2; yaw=vanish_point.x-cols / 2; Wherein, pitch is the pitch angle, yaw is the yaw angle, vanish_point.y is the side coordinate of the vanishing point, vanish_point.x is the front coordinate of the vanishing point, rows is the total number of image pixel rows of the current road image, cols is the total number of image pixel columns of the current road image, and the image resolution includes the total number of image pixel rows and the total number of image pixel columns.

13. The roadside fitting prediction method according to any one of claims 1 to 4, characterized in that: After obtaining two predicted fitting roadside lines, the roadside line fitting prediction method further includes: Matching the initial fitting roadside line and the predicted fitting roadside line through a Hungarian algorithm model; If the matching is successful, the roadside tracking of the current road image is completed.

14. The roadside fitting prediction method according to any one of claims 1 to 4, characterized in that: After obtaining two predicted fitting roadside lines, the roadside line fitting prediction method further includes: The predicted fitting roadside lines are displayed on the current road image based on the vehicle body image mapping parameters.

15. A roadside fitting prediction device, characterized in that: The roadside fitting prediction device comprises: An acquisition module, configured to acquire image positions of a plurality of roadside points in a current drivable area, wherein the current drivable area is determined based on a current road image; a position conversion module, configured to determine the vehicle body position of the roadside point based on the image position and vehicle body image mapping parameters, thereby obtaining the vehicle body position of each roadside point; An initial fitting module, configured to obtain two initial fitting roadside lines based on the vehicle body position fitting of each of the roadside line points; a vanishing point determination module, configured to determine a vanishing point position within a current drivable area, comprising: accumulating the distances between the two initially fitted roadside lines to obtain a cumulative value until the cumulative value reaches a preset range, the distance being determined based on the vehicle body front coordinates of a pair of roadside points on the two initially fitted roadside lines, the pair of roadside points being two roadside points having identical vehicle body side coordinates; and determining a target vehicle body position corresponding to the cumulative value as the vanishing point position, the target vehicle body position being the vehicle body position corresponding to the minimum distance among the accumulated distances; a vehicle body pitch angle and vehicle body yaw angle determination module, configured to determine the vehicle body pitch angle and vehicle body yaw angle according to the vanishing point position and the image resolution of the current road image; The prediction fitting module is used to use the vehicle body position of each roadside point as an observation quantity, the vanishing point position, the vehicle body pitch angle and the vehicle body yaw angle as control variables, and predict the two initial fitting roadside lines through an extended Kalman filter model to obtain two predicted fitting roadside lines.

16. An electronic device, characterized in that: The electronic device comprises: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the electronic device to implement the roadside line fitting prediction method as described in any one of claims 1 to 14.

17. A computer-readable storage medium, characterized in that A computer program is stored thereon, and when the computer program is executed by a processor of a computer, the computer is caused to execute the roadside line fitting prediction method according to any one of claims 1 to 14.

Images