Method and device for generating trajectory of vehicle-mounted panoramic image

Abstract

The invention discloses a trajectory generation method and device for a vehicle-mounted panoramic image. The method comprises the steps of predicting a first trajectory of a preset wheel in an Ackerman steering coordinate system through a preset single vehicle model according to a first deflection angle of a steering wheel of a current vehicle; acquiring a second trajectory of the first trajectory in a bird's-eye view corresponding to the vehicle according to the first trajectory; and projecting the second trajectory to a single view plane through a reverse projection algorithm to obtain a third trajectory in the single view. According to the invention, through the single vehicle model and the Ackerman steering coordinate system, the real-time performance and precision of generating the second trajectory in the aerial view are improved, the second trajectory in the aerial view is further projected into the single view through the reverse projection algorithm, and the precision of the third trajectory in the panoramic image is improved.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving, and in particular to a method and device for generating a trajectory line of a vehicle-mounted panoramic image. Background Art

[0002] Existing self-driving vehicles are often equipped with a panoramic imaging system, which uses the on-board camera to collect images around the vehicle and displays the vehicle's expected driving trajectory in real time based on the vehicle's steering angle and driving path. This function can help the driver predict the vehicle's movement trajectory, assist in parking and avoid obstacles.

[0003] When predicting real-time trajectory lines, existing technologies often require real-time calculations based on the vehicle's steering angle and wheel position. When generating real-time trajectory lines, existing technologies dynamically adjust existing trajectory lines based on the existing trajectory lines, the current vehicle's steering, and the homography matrix between each single view. The adjustment process depends on the coordinate transformation of the homography matrix. Under complex perspective distortion, such as the nonlinear distortion of a fisheye lens, the correction effect of the homography matrix is ​​limited, resulting in a difference between the displayed trajectory line and the actual trajectory.

[0004] Therefore, how to improve the trajectory accuracy while ensuring real-time calculation speed is a technical problem that needs to be solved at present. Summary of the invention

[0005] The present application provides a method and device for generating a trajectory line of a vehicle-mounted panoramic image, so as to solve the technical problem of how to improve the trajectory line accuracy while ensuring the real-time calculation speed.

[0006] In order to solve the above technical problems, in a first aspect, an embodiment of the present application provides a method for generating a trajectory line of a vehicle-mounted panoramic image, comprising:

[0007] According to a first deflection angle of the steering wheel of the current vehicle, a first trajectory line of a preset wheel in an Ackerman steering coordinate system is predicted by using a preset single-vehicle model;

[0008] According to the first trajectory line, obtain a second trajectory line of the first trajectory line in the bird's-eye view corresponding to the vehicle;

[0009] The second trajectory line is projected onto the single view plane through a reverse projection algorithm to obtain a third trajectory line in the single view.

[0010] Compared with the prior art, the embodiments of the present application have the following beneficial effects: since the bicycle model assumes that the front and rear axles of the vehicle are each represented by a wheel, the computational complexity of the vehicle motion is simplified and the real-time performance of predicting the vehicle's motion trajectory is improved; further combined with the Ackerman steering coordinate system, the difference in the front and rear wheel trajectories of the vehicle when turning is accurately described, and the accuracy of the second trajectory line in the bird's-eye view is improved by mutual conversion between the Ackerman coordinate system and the plane coordinate system of the bird's-eye view; finally, through the reverse projection algorithm, the second trajectory line under the bird's-eye view perspective is projected to a single-view plane under a nonlinear distortion perspective, which effectively solves the coordinate mapping problem under complex perspective distortion. At the same time, since the second trajectory line under the bird's-eye view has higher accuracy, the accuracy of the third trajectory line in the single view can be further improved.

[0011] In some embodiments of the first aspect of the present application, predicting a first trajectory of a preset wheel in an Ackerman steering coordinate system by using a preset single-vehicle model according to a first deflection angle of a steering wheel of the current vehicle includes:

[0012] According to the first deflection angle, a second deflection angle of the front of the vehicle within a plurality of preset time intervals is calculated by using the preset single-vehicle model;

[0013] Calculating a displacement matrix corresponding to the vehicle in the Ackerman steering coordinate system according to the second deflection angle;

[0014] The first trajectory line is obtained according to each of the displacement matrices and the current trajectory point of the preset wheel.

[0015] Compared with the prior art, the above embodiment has the following beneficial effects: the second deflection angle corresponding to the front of the current vehicle at several time points in the future is calculated in real time according to the first deflection angle of the steering wheel through the single-vehicle model; the displacement matrix corresponding to each time point in the Ackerman steering coordinate system is updated in real time according to the obtained deflection angle, thereby reducing the amount of data calculation, improving the real-time performance of the subsequent dynamic update of the third trajectory line, and improving the accuracy of the trajectory line.

[0016] In some embodiments of the first aspect of the present application, the calculating, according to the first deflection angle, a second deflection angle of the front of the vehicle within a plurality of preset time intervals by using the preset bicycle model includes:

[0017] According to the first deflection angle and the steering ratio between the steering wheel and the front wheels of the vehicle, a third deflection angle of the front wheels is obtained:

[0018] Obtaining a rotation radius of the front wheel according to the third deflection angle and the wheelbase of the vehicle;

[0019] According to the rotation radius and the current speed of the vehicle, in combination with the preset bicycle model, a second deflection angle corresponding to a plurality of preset time intervals is obtained.

[0020] Compared with the prior art, the above embodiment has the following beneficial effects: the third deflection angle of the front wheel can be obtained through the steering ratio and the first deflection angle, and the rotation radius of the front wheel can be further obtained based on the third deflection angle. Under the premise of simplifying the bicycle model, the second deflection angle of the front of the vehicle at a certain time interval and with a fixed front wheel rotation radius and vehicle speed can be obtained. By changing the length of the time interval, the second deflection angle at different preset time intervals can be obtained, which improves data calculation efficiency and thus improves the real-time performance of the subsequent dynamic update of the third trajectory line.

[0021] In some embodiments of the first aspect of the present application, calculating, according to the second deflection angle, a displacement matrix corresponding to the vehicle in the Ackerman steering coordinate system includes:

[0022] The displacement matrix includes a rotation matrix and a translation matrix;

[0023] The rotation matrix is ​​obtained by the following formula:

[0024]

[0025] The translation matrix is ​​obtained by the following formula:

[0026] Kt=P+ΔP-Kr·P

[0027] ΔP=[R sinθ,RR cosθ]

[0028] Among them, Kr is the rotation matrix; θ is the second deflection angle; Kt is the translation matrix; P is the origin coordinate; R is the rotation radius of the front wheel of the vehicle; ΔP is the translation parameter.

[0029] In some embodiments of the first aspect of the present application, acquiring the first trajectory line according to the displacement matrix and the current trajectory point of the preset wheel includes:

[0030] The calculation formula of the first trajectory line is specifically:

[0031] P t =P0·Kr+Kt

[0032] Among them, P0 is the current trajectory point; P t is the trajectory point where the wheel is preset when the preset time interval is t in the first trajectory line; Kr is the rotation matrix; Kt is the translation matrix.

[0033] In some embodiments of the first aspect of the present application, projecting the second trajectory line to a single view plane by a reverse projection algorithm to obtain a third trajectory line in the single view includes:

[0034] In combination with the offset of the camera relative to the ground, the second trajectory line is projected into a three-dimensional ground coordinate system to obtain a fifth trajectory line;

[0035] Projecting the fifth trajectory line into the camera coordinate system to obtain a sixth trajectory line;

[0036] According to the intrinsic parameters of the camera corresponding to the camera coordinate system, the sixth trajectory line is projected into the single view corresponding to the camera to obtain the third trajectory line.

[0037] Compared with the prior art, the above embodiments have the following beneficial effects: the present application takes into account the influence of the camera position offset on the accuracy of the reverse projection, adopts the second trajectory line with higher accuracy generated under the bird's-eye view, and projects the second trajectory line to the three-dimensional ground coordinates in combination with the offset of the camera relative to the ground, corrects the error caused by the camera position offset, and then projects the three-dimensional ground coordinates into the camera coordinate system, so that the camera coordinate system and the single view plane can be converted into each other in combination with the camera's internal parameters, thereby improving the accuracy of the third trajectory line.

[0038] In some embodiments of the first aspect of the present application, projecting the sixth trajectory line into the single view corresponding to the camera according to the intrinsic parameters of the camera coordinate system to obtain the third trajectory line includes:

[0039] Traversing each first track point in the sixth track line, and calculating the radial distance of the first track point in the camera coordinate system;

[0040] Correcting a fourth deflection angle of each of the first trajectory points relative to the optical axis of the camera according to the internal parameter;

[0041] The third trajectory line is acquired according to the internal parameter, the fourth deflection angle, and the radial distance in combination with each of the first trajectory points.

[0042] Compared with the prior art, the above embodiment has the following beneficial effects: since the radial distortion coefficient is included in the camera's intrinsic parameters, the present application takes into account the influence of nonlinear distortion on the accuracy of reverse projection, and corrects the fourth deflection angle corresponding to the first trajectory point through the camera's intrinsic parameters. Furthermore, the third trajectory line is calculated based on the first trajectory point corrected for the camera's position offset, the fourth deflection angle after distortion correction, and the camera's intrinsic parameters, thereby improving the accuracy of the third trajectory line finally obtained.

[0043] In some embodiments of the first aspect of the present application, acquiring the third trajectory line according to the intrinsic parameter, the fourth deflection angle, and the radial distance in combination with each of the first trajectory points includes:

[0044] The second trajectory point corresponding to the first trajectory point in the third trajectory line is calculated by the following formula:

[0045]

[0046] Where, (w,m) is the coordinate of the second trajectory point; (f x ,f y ) is the focal length in the internal reference; (X c ,Y c ) are the x-axis and y-axis coordinates of the first trajectory point in the camera coordinate system; d is the fourth deflection angle after correction; (c x ,c y ) is the principal point in the internal parameter; r is the radial distance of the first trajectory point in the camera coordinate system.

[0047] In some embodiments of the first aspect of the present application, the step of correcting the fourth deflection angle of each of the first trajectory points relative to the optical axis of the camera according to the internal parameter includes:

[0048] Calculating the fourth deflection angle according to the radial distance and the coordinate value of the optical axis of the first trajectory point in the camera coordinate system;

[0049] The fourth deflection angle is corrected according to the radial distortion coefficient in the internal parameter.

[0050] Compared with the prior art, the above embodiment has the following beneficial effects: the radial distance is calculated and obtained by correcting the offset of the camera relative to the ground. According to the radial distance, combined with the coordinate value of the first trajectory point in the optical axis direction of the camera coordinate system, the fourth deflection angle can be accurately obtained, and further combined with the radial distortion coefficient, the fourth deflection angle is corrected to improve the accuracy of the subsequent third trajectory line.

[0051] In a second aspect, an embodiment of the present application further provides a trajectory line generation device for a vehicle-mounted panoramic image, comprising: a first trajectory line calculation module, a second trajectory line calculation module, and a third trajectory line calculation module;

[0052] The first trajectory calculation module is used to calculate a first trajectory of a preset wheel in an Ackerman steering coordinate system according to a first deflection angle of a steering wheel of the current vehicle;

[0053] The second trajectory line calculation module is used to obtain a second trajectory line of the first trajectory line in the bird's-eye view corresponding to the vehicle according to the first trajectory line;

[0054] The third trajectory line calculation module is used to project the second trajectory line to the single view plane through a reverse projection algorithm to obtain the third trajectory line in the single view. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 A schematic flow chart of a method for generating a trajectory line of a vehicle-mounted panoramic image provided in some embodiments of the present application;

[0056] Figure 2 A schematic diagram of a bird's-eye view trajectory obtained according to a trajectory generation method of a vehicle-mounted panoramic image provided in some embodiments of the present application;

[0057] Figure 3 A schematic diagram of a single-view trajectory obtained according to a trajectory generation method of a vehicle-mounted panoramic image provided in some embodiments of the present application;

[0058] Figure 4 This is a schematic structural diagram of a trajectory generation device for a vehicle-mounted panoramic image provided in some embodiments of the present application. DETAILED DESCRIPTION

[0059] When predicting real-time trajectory lines, existing technologies often require real-time calculations based on the vehicle's steering angle and wheel position. When generating real-time trajectory lines, existing technologies dynamically adjust existing trajectory lines based on the existing trajectory lines, the current vehicle's steering, and the homography matrix between each single view. The adjustment process depends on the coordinate transformation of the homography matrix. Under complex perspective distortion, such as the nonlinear distortion of a fisheye lens, the correction effect of the homography matrix is ​​limited, resulting in a difference between the displayed trajectory line and the actual trajectory.

[0060] In order to solve the above technical problems and ensure the real-time generation of trajectory lines, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of them. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0061] Embodiment 1

[0062] Please refer to Figure 1 , a method for generating a trajectory line of a vehicle-mounted panoramic image provided by an embodiment of the present application, including S10 to S30, specifically:

[0063] S10: According to a first deflection angle of a steering wheel of the current vehicle, a first trajectory line of a preset wheel in an Ackerman steering coordinate system is predicted by using a preset single-vehicle model.

[0064] Further, in some embodiments of the present application, the above S10, including S11 to S13, is specifically:

[0065] S11: Calculating a second deflection angle of the front of the vehicle within a plurality of preset time intervals based on the first deflection angle and using the preset bicycle model.

[0066] Among them, the single-vehicle model is a kinematic model of the vehicle in motion. The model is based on the following assumptions: the vehicle is assumed to move in a two-dimensional plane, and the front and rear wheels of the vehicle are each one wheel, that is, the front left and right wheels and the rear left and right wheels have the same steering angle and speed. At the same time, it is assumed that the body and suspension system are rigid systems, and the movement and steering of the vehicle are driven by the front wheels, which greatly simplifies the simulation calculation complexity of the vehicle in motion. Through the single-vehicle model, the real-time generation of the first trajectory line during vehicle trajectory prediction can be greatly improved.

[0067] Specifically, in some embodiments of the present application, the above S11, including S111 to S113, is specifically:

[0068] S111: Acquire a third deflection angle of the front wheels according to the first deflection angle and a steering ratio between the steering wheel and the front wheels of the vehicle.

[0069] Preferably, in some embodiments of the present application, the third deflection angle can be calculated and obtained by the following formula:

[0070] β=α / i

[0071] Wherein, β is the third deflection angle; α is the first deflection angle; and i is the steering ratio of the vehicle steering wheel.

[0072] S112: Acquire the rotation radius of the front wheel according to the third deflection angle and the wheelbase of the vehicle.

[0073] Preferably, in some embodiments of the present application, the rotation radius of the front wheel can be calculated by the following formula:

[0074] R=L / tanβ

[0075] Wherein, L is the wheelbase of the vehicle; R is the turning radius of the front wheel.

[0076] S113: According to the rotation radius and the current speed of the vehicle, in combination with the preset bicycle model, obtaining second deflection angles corresponding to a plurality of preset time intervals.

[0077] Preferably, in some embodiments of the present application, the second deflection angle corresponding to each preset time interval can be obtained by the following formula:

[0078] θ=vt / R

[0079] Wherein, θ is the second deflection angle corresponding to the preset time interval t; v is the speed of the vehicle when the third trajectory is currently generated. Since the third trajectory is updated dynamically in real time, the second deflection angles generated each time the third trajectory is updated are generated and obtained by fixing β and v and adjusting the size of t. Wherein, each second deflection angle corresponds to a trajectory point in the third trajectory.

[0080] Preferably, in some embodiments of the present application, when taking the value of t, the acquisition can be determined by gradually increasing the time interval by 1 second starting from 0 seconds.

[0081] It can be seen from the above S111 to S113 that the present application can obtain the third deflection angle of the front wheel through the steering ratio and the first deflection angle, and further obtain the rotation radius of the front wheel based on the third deflection angle. Under the premise of simplifying the bicycle model, the second deflection angle of the front of the vehicle at a certain time interval and with a fixed front wheel rotation radius and vehicle speed is obtained. By changing the length of the time interval, the second deflection angle at different preset time intervals is obtained, which improves the data calculation efficiency and thus improves the real-time performance of the subsequent dynamic update of the third trajectory line.

[0082] S12: Calculating a displacement matrix corresponding to the vehicle in the Ackerman steering coordinate system according to the second deflection angle.

[0083] Under the assumption of a single-vehicle model, based on the second deflection angle of the vehicle front at different preset time intervals, the real-time displacement matrix is ​​updated in the Ackerman steering coordinate system to improve the real-time update of the first trajectory line, thereby improving the real-time and accuracy of the final third trajectory line update.

[0084] Preferably, in some embodiments of the present application, the Ackerman steering coordinate system is established with the center point of the rear wheel axle of the vehicle as the origin, the horizontal left direction of the vehicle as the positive direction of the y-axis, and the vertical forward direction of the front of the vehicle as the square of the x-axis. At the same time, the coordinates of the corresponding pixel points of the current origin in the bird's-eye view (X o ,Y o ) to facilitate the subsequent projection of the first trajectory line into the ground three-dimensional coordinate system.

[0085] Preferably, in some embodiments of the present application, the displacement matrix includes a rotation matrix Kr and a translation matrix Kt; wherein the rotation matrix Kr and the translation matrix Kt are respectively calculated and obtained by the following formulas:

[0086]

[0087] Kt=P+ΔP-Kr·P

[0088] ΔP=[R sinθ,RR cosθ]

[0089] Among them, Kr is the rotation matrix; θ is the second deflection angle; Kt is the translation matrix; P is the origin coordinate; R is the rotation radius of the front wheel of the vehicle; ΔP is the translation parameter.

[0090] S13: Acquire the first trajectory line according to each of the displacement matrices and the current trajectory point of the preset wheel.

[0091] When the displacement matrix corresponding to each rotation angle is obtained, the coordinate position of the preset wheel at different times can be obtained through simple linear operations in combination with the trajectory points of the preset wheel in the current Ackerman steering coordinate system, thereby obtaining the first trajectory line, thereby improving the real-time performance of the first trajectory line calculation.

[0092] Preferably, in some embodiments of the present application, the first trajectory line can be obtained by calculating using the following formula:

[0093] P t =P0·Kr+Kt

[0094] Among them, P0 is the current trajectory point; P t The preset track point of the wheel when the preset time interval is t in the first track line.

[0095] It can be seen from the above S11 to S13 that the present application uses a single-vehicle model and, based on the first deflection angle of the steering wheel, calculates in real time the second deflection angle of the front of the current vehicle corresponding to several time points in the future, and updates in real time the displacement matrix corresponding to each time point in the Ackerman steering coordinate system of the vehicle according to the acquired deflection angle, thereby reducing the amount of data calculation, improving the real-time performance of the subsequent dynamic update of the third trajectory line, and improving the accuracy of the trajectory line.

[0096] S20: According to the first trajectory line, obtain a second trajectory line of the first trajectory line in the bird's-eye view corresponding to the vehicle.

[0097] Preferably, in some embodiments of the present application, according to the coordinates of the origin in the bird's-eye view recorded in the previously established Ackerman steering coordinate system, the first trajectory line is projected into the bird's-eye view to obtain the second trajectory line, which is specifically obtained by the following formula:

[0098]

[0099] Among them, (X t ,Y t ) is the point Pt The corresponding projection point in the second trajectory, P t =(P t,x ,P t,y ).

[0100] Preferably, in some embodiments of the present application, assuming that the width and height of the bird's-eye view are width and height respectively, the coordinates (X t ,Y t ) corresponds to the vertex coordinates (X d ,Y d ), and input the vertex coordinates into the shader, so as to obtain the following based on OpenGL GPU rendering: Figure 2 The second trajectory is shown:

[0101]

[0102] S30: Projecting the second trajectory line onto a single view plane through a reverse projection algorithm to obtain a third trajectory line in the single view.

[0103] Further, in some embodiments of the present application, S30 includes S31 to S33, specifically:

[0104] S31: Projecting the second trajectory line into a three-dimensional ground coordinate system in combination with an offset of the camera relative to the ground to obtain a fifth trajectory line.

[0105] Since the camera has a certain offset relative to the ground, if the error caused by the offset is not considered in the subsequent calculation of the fifth trajectory, the accuracy of the third trajectory will be affected. Therefore, it is necessary to consider the offset in the projection of the bird's-eye view to the three-dimensional ground coordinate system, so as to improve the accuracy of the subsequent third trajectory.

[0106] Preferably, in some embodiments of the present application, the fifth trajectory line is obtained by calculating using the following formula:

[0107]

[0108] Among them, the coordinates of the trajectory point in the fifth trajectory line are (X g ,Y g ,Z g ), z-axis coordinate value Z g is the height of the camera from the ground, X g and Y g are the coordinate values ​​of the x-axis and y-axis respectively; scale is the actual physical length corresponding to the unit pixel in the bird's-eye view; (x move ,y move ) is the offset of the camera relative to the ground.

[0109] S32: Project the fifth trajectory line into the camera coordinate system to obtain a sixth trajectory line.

[0110] Preferably, in some embodiments of the present application, in order to obtain the trajectory line in the single view captured by the camera, it is still necessary to project the fifth trajectory line in the three-dimensional ground coordinate system to the camera coordinate system. The projection can be obtained by converting the external parameters of the camera, specifically, by multiplying the inverse matrix of the external parameters of the camera with the coordinate value of each trajectory point in the fifth trajectory line to obtain the coordinate point (X c ,Y c ,Z c ).

[0111] S33: According to the intrinsic parameters of the camera corresponding to the camera coordinate system, project the sixth trajectory line into the single view corresponding to the camera to obtain the third trajectory line.

[0112] It can be seen from S31 to S33 that the present application takes into account the influence of the camera position offset on the accuracy of the reverse projection, adopts the second trajectory line with higher accuracy generated under the bird's-eye view, and combines the offset of the camera relative to the ground to project the second trajectory line to the three-dimensional ground coordinates, corrects the error caused by the camera position offset, and then projects the three-dimensional ground coordinates into the camera coordinate system, so that the camera coordinate system and the single view plane can be converted into each other in combination with the camera's intrinsic parameters, thereby improving the accuracy of the third trajectory line.

[0113] Further, in some embodiments of the present application, S33 includes S331 to S333, specifically:

[0114] S331: traverse each first track point in the sixth track line, and calculate the radial distance of the first track point in the camera coordinate system.

[0115] Preferably, in some embodiments of the present application, the radial distance can be obtained by the following calculation formula:

[0116]

[0117] Where r is the radial distance.

[0118] S332: Correcting a fourth deflection angle of each of the first trajectory points relative to the optical axis of the camera according to the internal parameter.

[0119] Preferably, in some embodiments of the present application, the intrinsic parameters of the camera include focal length (f x ,f y ), main point (c x ,c y ) and radial distortion coefficients (k1, k2, k3, k4).

[0120] Further, in some embodiments of the present application, S332 includes S3321 to S3322, specifically:

[0121] S3321: Calculate the fourth deflection angle according to the radial distance and the coordinate value of the optical axis of the first trajectory point in the camera coordinate system.

[0122] Preferably, in some embodiments of the present application, the fourth deflection angle can be obtained by the following calculation formula:

[0123] o=atan2(r,Z c )

[0124] Wherein, o is the fourth deflection angle; atan2(*) is the inverse tangent function.

[0125] S3322: Correct the fourth deflection angle according to the radial distortion coefficient in the internal parameter.

[0126] It can be seen from S3321 to S3322 that the radial distance is calculated and obtained by the first trajectory point after correction of the offset of the camera relative to the ground. Based on the radial distance and the coordinate value of the first trajectory point in the optical axis direction of the camera coordinate system, the fourth deflection angle can be accurately obtained. Further combined with the radial distortion coefficient, the fourth deflection angle can be corrected to improve the accuracy of the subsequent third trajectory line.

[0127] S333: Acquire the third trajectory line according to the internal parameter, the fourth deflection angle, and the radial distance in combination with each of the first trajectory points.

[0128] Preferably, in some embodiments of the present application, the third trajectory line is obtained by the following calculation formula:

[0129]

[0130] Wherein, (w, m) is the coordinate of the second trajectory point corresponding to the first trajectory point in the third trajectory line; (X c ,Y c ) are the x-axis and y-axis coordinates of the first trajectory point in the camera coordinate system; d is the fourth deflection angle after correction.

[0131] Preferably, in some embodiments of the present application, the trajectory points in the third trajectory line are rendered by OpenGL rendering to obtain Figure 3 shown Figure 2 The trajectory line in the bird's-eye view corresponds to the trajectory line in the single view ( Figure 3 ).

[0132] It can be seen from S331 to S333 that, since the camera's intrinsic parameters include the radial distortion coefficient, the present application takes into account the impact of nonlinear distortion on the accuracy of the reverse projection, and corrects the fourth deflection angle corresponding to the first trajectory point through the camera's intrinsic parameters. Furthermore, the third trajectory line is calculated based on the first trajectory point corrected for the camera's position offset, the fourth deflection angle after distortion correction, and the camera's intrinsic parameters, thereby improving the accuracy of the third trajectory line finally obtained.

[0133] In summary, it can be seen that the trajectory line generation method of a vehicle-mounted panoramic image provided by an embodiment of the present application has the following beneficial effects: since the single-vehicle model assumes that the front and rear axles of the vehicle are each represented by a wheel, the computational complexity of the vehicle movement is simplified and the real-time prediction of the vehicle's motion trajectory is improved; further combined with the Ackerman steering coordinate system, the difference in the front and rear wheel trajectories of the vehicle when turning is accurately described, and the accuracy of the second trajectory line in the bird's-eye view is improved through mutual conversion between the Ackerman coordinate system and the plane coordinate system of the bird's-eye view; finally, through the reverse projection algorithm, the second trajectory line under the bird's-eye view perspective is projected to a single-view plane under a nonlinear distortion perspective, which effectively solves the coordinate mapping problem under complex perspective distortion. At the same time, since the second trajectory line under the bird's-eye view has higher accuracy, the accuracy of the third trajectory line in the single view can be further improved.

[0134] Embodiment 2

[0135] refer to Figure 4 , a trajectory line generation device for a vehicle-mounted panoramic image provided in an embodiment of the present application, includes: a first trajectory line calculation module 11, a second trajectory line calculation module 12 and a third trajectory line calculation module 13.

[0136] Furthermore, in some embodiments of the present application, the first trajectory calculation module 11 is used to calculate a first trajectory of a preset wheel in an Ackerman steering coordinate system according to a first deflection angle of a steering wheel of the current vehicle; the second trajectory calculation module 12 is used to obtain a second trajectory of the first trajectory in a bird's-eye view corresponding to the vehicle according to the first trajectory; the third trajectory calculation module 13 is used to project the second trajectory to a single view plane through a reverse projection algorithm to obtain a third trajectory in the single view.

[0137] Furthermore, in some embodiments of the present application, the method predicts a first trajectory line of a preset wheel in an Ackermann steering coordinate system through a preset bicycle model based on a first deflection angle of a steering wheel of the current vehicle, including: calculating a second deflection angle of the front of the vehicle within a number of preset time intervals through the preset bicycle model based on the first deflection angle; calculating a displacement matrix corresponding to the vehicle in the Ackermann steering coordinate system based on the second deflection angle; and obtaining the first trajectory line based on each of the displacement matrices and the current trajectory point of the preset wheel.

[0138] Furthermore, in some embodiments of the present application, the second deflection angle of the front of the vehicle within a number of preset time intervals is calculated based on the first deflection angle through the preset bicycle model, including: obtaining a third deflection angle of the front wheel based on the first deflection angle and the steering ratio of the steering wheel and the front wheels of the vehicle; obtaining a rotation radius of the front wheel based on the third deflection angle and the wheelbase of the vehicle; and obtaining a second deflection angle corresponding to a number of preset time intervals based on the rotation radius and the current speed of the vehicle, combined with the preset bicycle model.

[0139] Further, in some embodiments of the present application, the calculating, according to the second deflection angle, a displacement matrix corresponding to the vehicle in the Ackerman steering coordinate system includes: the displacement matrix includes a rotation matrix and a translation matrix;

[0140] The rotation matrix is ​​obtained by the following formula:

[0141]

[0142] The translation matrix is ​​obtained by the following formula:

[0143] Kt=P+ΔP-Kr·P

[0144] ΔP=[R sinθ,RR cosθ]

[0145] Among them, Kr is the rotation matrix; θ is the second deflection angle; Kt is the translation matrix; P is the origin coordinate; R is the rotation radius of the front wheel of the vehicle; ΔP is the translation parameter.

[0146] Further, in some embodiments of the present application, acquiring the first trajectory line according to the displacement matrix and the current trajectory point of the preset wheel includes:

[0147] The calculation formula of the first trajectory line is specifically:

[0148] P t =P0·Kr+Kt

[0149] Among them, P0 is the current trajectory point; P t is the trajectory point where the wheel is preset when the preset time interval is t in the first trajectory line; Kr is the rotation matrix; Kt is the translation matrix.

[0150] Furthermore, in some embodiments of the present application, the second trajectory line is projected onto a single view plane through a reverse projection algorithm to obtain a third trajectory line in the single view, including: combining the offset of the camera relative to the ground, projecting the second trajectory line into a three-dimensional ground coordinate system to obtain a fifth trajectory line; projecting the fifth trajectory line into a camera coordinate system to obtain a sixth trajectory line; and projecting the sixth trajectory line into the single view corresponding to the camera according to the intrinsic parameters of the camera coordinate system to obtain the third trajectory line.

[0151] Furthermore, in some embodiments of the present application, the sixth trajectory line is projected onto the single view corresponding to the camera according to the intrinsic parameters of the camera coordinate system to obtain the third trajectory line, including: traversing each first trajectory point in the sixth trajectory line, and calculating the radial distance of the first trajectory point in the camera coordinate system; correcting the fourth deflection angle of each first trajectory point relative to the optical axis of the camera according to the intrinsic parameters; and obtaining the third trajectory line according to the intrinsic parameters, the fourth deflection angle and the radial distance in combination with each first trajectory point.

[0152] Further, in some embodiments of the present application, acquiring the third trajectory line according to the internal parameter, the fourth deflection angle, and the radial distance in combination with each of the first trajectory points includes:

[0153] The second trajectory point corresponding to the first trajectory point in the third trajectory line is calculated by the following formula:

[0154]

[0155] Where, (w,m) is the coordinate of the second trajectory point; (f x ,f y ) is the focal length in the internal reference; (X c ,Y c ) are the x-axis and y-axis coordinates of the first trajectory point in the camera coordinate system; d is the fourth deflection angle after correction; (c x ,c y ) is the principal point in the internal parameter; r is the radial distance of the first trajectory point in the camera coordinate system.

[0156] Furthermore, in some embodiments of the present application, correcting the fourth deflection angle of each of the first trajectory points relative to the optical axis of the camera based on the internal parameter includes: calculating the fourth deflection angle based on the radial distance and the coordinate value of the optical axis of the first trajectory point in the camera coordinate system; and correcting the fourth deflection angle based on the radial distortion coefficient in the internal parameter.

[0157] It can be understood that the above-mentioned device item embodiments correspond to the method item embodiments of the present invention. The trajectory line generation device for vehicle-mounted panoramic images provided by the embodiments of the present invention can implement any method item embodiment of the present invention, that is, the trajectory line generation method for vehicle-mounted panoramic images provided in Embodiment 1.

[0158] Embodiment 3

[0159] Based on the above-mentioned embodiment of the trajectory line generation method of the vehicle-mounted panoramic image, another embodiment of the present application provides a vehicle-mounted panoramic image trajectory line generation terminal device, and the vehicle-mounted panoramic image trajectory line generation terminal device includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, the vehicle-mounted panoramic image trajectory line generation method of any embodiment of the present application is implemented.

[0160] Exemplarily, in this embodiment, the computer program may be divided into one or more modules, which are stored in the memory and executed by the processor to complete the present application. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program in the vehicle-mounted panoramic image trajectory generation device.

[0161] The vehicle-mounted panoramic image trajectory generation device may be a computing device such as a desktop computer, a notebook, a palm computer, a cloud server, etc. The vehicle-mounted panoramic image trajectory generation terminal device may include, but is not limited to, a processor and a memory.

[0162] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc. The processor is the control center of the vehicle-mounted panoramic image trajectory generation device, and uses various interfaces and lines to connect the various parts of the vehicle-mounted panoramic image trajectory generation device. The memory may be used to store the computer program and / or module, and the processor implements various functions of the vehicle-mounted panoramic image trajectory generation device by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory. The memory may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the mobile phone, etc. In addition, the memory may include high-speed random access memory and may also include non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0163] Embodiment 4

[0164] Based on the above-mentioned embodiment of the trajectory line generation method of the vehicle-mounted panoramic image, another embodiment of the present application provides a storage medium, wherein the storage medium includes a stored computer program, wherein when the computer program is running, the device where the storage medium is located is controlled to execute the trajectory line generation method of the vehicle-mounted panoramic image of any embodiment of the present application.

[0165] In this embodiment, the storage medium is a computer-readable storage medium, and the computer program includes computer program code, which may be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0166] The specific embodiments described above further describe the purpose, technical solutions and beneficial effects of the present application in detail. It should be understood that the above description is only a specific embodiment of the present application and is not intended to limit the scope of protection of the present application. It is particularly pointed out that for those skilled in the art, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of protection of the present application.

Claims

1. A method for generating a trajectory line of a vehicle-mounted panoramic image, characterized in that: include: According to a first deflection angle of the steering wheel of the current vehicle, a first trajectory line of a preset wheel in an Ackerman steering coordinate system is predicted by using a preset single-vehicle model; According to the first trajectory line, obtain a second trajectory line of the first trajectory line in the bird's-eye view corresponding to the vehicle; The second trajectory line is projected onto the single view plane through a reverse projection algorithm to obtain a third trajectory line in the single view.

2. The method for generating a trajectory line of a vehicle-mounted panoramic image according to claim 1, characterized in that: The method predicts a first trajectory of a preset wheel in an Ackerman steering coordinate system according to a first deflection angle of a steering wheel of the current vehicle by using a preset single-vehicle model, including: According to the first deflection angle, a second deflection angle of the front of the vehicle within a plurality of preset time intervals is calculated by using the preset single-vehicle model; Calculating a displacement matrix corresponding to the vehicle in the Ackerman steering coordinate system according to the second deflection angle; The first trajectory line is obtained according to each of the displacement matrices and the current trajectory point of the preset wheel.

3. The method for generating a trajectory line of a vehicle-mounted panoramic image according to claim 2, characterized in that: The step of calculating the second deflection angle of the front of the vehicle within a plurality of preset time intervals according to the first deflection angle by using the preset single-vehicle model includes: According to the first deflection angle and the steering ratio between the steering wheel and the front wheels of the vehicle, a third deflection angle of the front wheels is obtained: Obtaining a rotation radius of the front wheel according to the third deflection angle and the wheelbase of the vehicle; According to the rotation radius and the current speed of the vehicle, in combination with the preset bicycle model, a second deflection angle corresponding to a plurality of preset time intervals is obtained.

4. The method for generating a trajectory line of a vehicle-mounted panoramic image according to claim 2, characterized in that: Calculating a displacement matrix corresponding to the vehicle in the Ackerman steering coordinate system according to the second deflection angle includes: The displacement matrix includes a rotation matrix and a translation matrix; The rotation matrix is ​​obtained by the following formula: The translation matrix is ​​obtained by the following formula: Kt=P+ΔP-Kr·P ΔP=[R sinθ,RR cosθ] Among them, Kr is the rotation matrix; θ is the second deflection angle; Kt is the translation matrix; P is the origin coordinate; R is the rotation radius of the front wheel of the vehicle; ΔP is the translation parameter.

5. The method for generating a trajectory line of a vehicle-mounted panoramic image according to claim 4, characterized in that: The acquiring the first trajectory line according to the displacement matrix and the current trajectory point of the preset wheel includes: The calculation formula of the first trajectory line is specifically: P t =P0·Kr+Kt Among them, P0 is the current trajectory point; P t is the trajectory point where the wheel is preset when the preset time interval is t in the first trajectory line; Kr is the rotation matrix; Kt is the translation matrix.

6. The method for generating a trajectory line of a vehicle-mounted panoramic image according to claim 1, characterized in that: The method of projecting the second trajectory line onto a single view plane by a reverse projection algorithm to obtain a third trajectory line in the single view includes: In combination with the offset of the camera relative to the ground, the second trajectory line is projected into a three-dimensional ground coordinate system to obtain a fifth trajectory line; Projecting the fifth trajectory line into the camera coordinate system to obtain a sixth trajectory line; According to the intrinsic parameters of the camera corresponding to the camera coordinate system, the sixth trajectory line is projected into the single view corresponding to the camera to obtain the third trajectory line.

7. The method for generating a trajectory line of a vehicle-mounted panoramic image according to claim 6, characterized in that: The step of projecting the sixth trajectory line onto the single view corresponding to the camera according to the intrinsic parameters of the camera corresponding to the camera coordinate system to obtain the third trajectory line includes: Traversing each first track point in the sixth track line, and calculating the radial distance of the first track point in the camera coordinate system; Correcting a fourth deflection angle of each of the first trajectory points relative to the optical axis of the camera according to the internal parameter; The third trajectory line is acquired according to the internal parameter, the fourth deflection angle, and the radial distance in combination with each of the first trajectory points.

8. The method for generating a trajectory line of a vehicle-mounted panoramic image according to claim 7, characterized in that: The acquiring the third trajectory line according to the internal parameter, the fourth deflection angle, and the radial distance in combination with each of the first trajectory points includes: The second trajectory point corresponding to the first trajectory point in the third trajectory line is calculated by the following formula: Where, (w,m) is the coordinate of the second trajectory point; (f x ,f y ) is the focal length in the internal reference; (X c ,Y c ) are the x-axis and y-axis coordinates of the first trajectory point in the camera coordinate system; d is the fourth deflection angle after correction; (c x ,c y ) is the principal point in the internal parameter; r is the radial distance of the first trajectory point in the camera coordinate system.

9. The method for generating a trajectory line of a vehicle-mounted panoramic image according to claim 7, characterized in that: The step of correcting the fourth deflection angle of each of the first trajectory points relative to the optical axis of the camera according to the internal parameter includes: Calculating the fourth deflection angle according to the radial distance and the coordinate value of the optical axis of the first trajectory point in the camera coordinate system; The fourth deflection angle is corrected according to the radial distortion coefficient in the internal parameter.

10. A trajectory line generating device for vehicle-mounted panoramic images, characterized in that: include: A first trajectory calculation module, a second trajectory calculation module and a third trajectory calculation module; The first trajectory calculation module is used to calculate a first trajectory of a preset wheel in an Ackerman steering coordinate system according to a first deflection angle of a steering wheel of the current vehicle; The second trajectory line calculation module is used to obtain a second trajectory line of the first trajectory line in the bird's-eye view corresponding to the vehicle according to the first trajectory line; The third trajectory line calculation module is used to project the second trajectory line to the single view plane through a reverse projection algorithm to obtain the third trajectory line in the single view.

Images