Road vehicle trajectory extraction method based on unmanned aerial vehicle monocular camera

Abstract

The application discloses a road vehicle trajectory extraction method based on a UAV monocular camera, and comprises the following steps: calibrating the UAV monocular camera based on a video stream containing a marker shot by the UAV, and acquiring camera intrinsic parameters; acquiring the UAV pose and adjusting the UAV attitude based on a video stream containing a marker shot by the UAV in a hovering mode, calculating the UAV pose after the parallelism between the UAV and the road plane meets the requirements, and obtaining the height of the UAV relative to the road plane; acquiring the pixel coordinates of the vehicle center point and the pixel coordinates of the edges on both sides of the road based on a road vehicle video stream shot by the UAV; establishing a world coordinate system with the hovering position of the UAV as the base point, converting the pixel coordinates of the vehicle center point and the pixel coordinates of the edges on both sides of the road into world coordinates based on the camera intrinsic parameters, the height of the UAV relative to the road plane and the conversion relationship between the camera coordinate system and the pixel coordinate system; and arranging the world coordinates in a sequence to obtain a vehicle trajectory graph. The application can acquire a high-definition vehicle trajectory.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of vehicle trajectory extraction, in particular to a road vehicle trajectory extraction method based on a monocular camera of a UAV. BACKGROUND

[0002] Mountainous area highways have many sharp curves, many sections with small radii of circular curves, many sections near rivers and cliffs, less traffic safety protection measures, poor vehicle driving visibility, serious driving risks, and are prone to traffic accidents. The traffic safety of mountainous area highways faces great safety problems, and the traffic safety situation is not optimistic. Therefore, targeted improvement measures must be taken to assist drivers in safe driving to reduce the occurrence of traffic accidents. The traditional satellite positioning technology has the limitation of poor signal quality in some mountainous area sections, the extracted vehicle trajectory deviates greatly from the actual trajectory, and cannot provide effective vehicle driving information. Therefore, it is necessary to design a system that can provide effective vehicle driving information for vehicles driving on mountainous area roads to improve the driving safety factor of mountainous area roads and reduce the incidence of traffic accidents. SUMMARY

[0003] The purpose of the present application is to solve the problems in the prior art and provide a road vehicle trajectory extraction method based on a monocular camera of a UAV. The driving video data of the vehicle on the mountainous area road is shot by the monocular camera of the UAV, and the vehicle trajectory is extracted using computer vision methods. The high-definition vehicle trajectory can provide effective vehicle driving information for the driver.

[0004] The present application is implemented as follows: a road vehicle trajectory extraction method based on a monocular camera of a UAV, comprising the steps of:

[0005] S1. Calibrate the monocular camera of the UAV based on the video stream containing the markers shot by the UAV, and obtain the camera intrinsic parameters;

[0006] S2. Obtain the UAV pose and adjust the UAV attitude based on the video stream containing the markers shot by the UAV in hovering mode, calculate the UAV pose after the parallelism of the UAV and the road plane meets the requirements, and obtain the height of the UAV relative to the road plane;

[0007] S3. Obtain the vehicle center point pixel coordinates and the road side edge pixel coordinates based on the road vehicle video stream shot by the UAV;

[0008] S4. Establish a first world coordinate system with the hovering position of the UAV as the base point, convert the vehicle center point pixel coordinates and the road side edge pixel coordinates into world coordinates based on the camera intrinsic parameters, the height of the UAV relative to the road plane, and the conversion relationship between the camera coordinate system and the pixel coordinate system;

[0009] S5. Arrange the world coordinates in order to obtain a vehicle trajectory graph.

[0010] In step S1, the video stream containing the marker shot by the UAV is used to calibrate the monocular camera of the UAV, and the camera intrinsic parameters are obtained, including the steps of:

[0011] The calibration video stream containing the marker shot by the UAV in the traffic scene is frame-extracted to obtain multiple pictures containing the marker;

[0012] The corner points of the marker in the pictures are detected to obtain the pixel coordinate values of the corner points of the marker;

[0013] According to the size of the marker and the second world coordinate system established with the center point of the marker as the origin, the physical coordinate values of the corner points of the marker are obtained;

[0014] Based on the correspondence between the pixel coordinate values and the physical coordinate values of the monocular camera, the camera intrinsic parameter matrix of the monocular camera of the UAV is obtained.

[0015] The camera intrinsic parameter matrix is repeatedly obtained multiple times to obtain multiple camera intrinsic parameter matrix calibration results, and one camera intrinsic parameter matrix calibration result is selected as the camera intrinsic parameter.

[0016] In step S2, the video stream containing the marker shot by the UAV in the traffic scene is used to calibrate the monocular camera of the UAV, and the camera intrinsic parameters are obtained, including the steps of:

[0017] Detect the preset information of the marker in the video stream to determine whether the marker is a preset marker; if so, obtain the coordinate values of the corner points of the preset marker on a single frame image in the image coordinate system;

[0018] Based on the third world coordinate system established with the center point of the marker as the origin, the physical coordinate values of the corner points of the marker in the third world coordinate system are obtained according to the size of the marker;

[0019] Obtain the homography matrix corresponding to the mapping relationship between the coordinate values of the corner points of the marker in the image coordinate system and the physical coordinate values in the third world coordinate system;

[0020] According to the camera intrinsic parameters, the homography matrix and the unit orthogonal relationship of the rotation vector in the camera extrinsic parameters, the camera extrinsic parameters of the monocular camera are calculated, and the pose information of the UAV relative to the target road plane is obtained.

[0021] In step S3, the video stream of the road vehicle shot by the UAV is used to obtain the pixel coordinate of the vehicle center point, and the target detection algorithm is used to realize the step of:

[0022] The video of each frame of the road vehicle video stream is cropped and scaled, the target vehicle in the picture is labeled, and a data set of a vehicle target detection network model is prepared; and the data set is enhanced by rotation, scaling and cropping;

[0023] The vehicle target detection network model is trained by using the training set of the data set, and the optimal weight parameter is obtained by using the verification set of the data set;

[0024] The optimal weight parameter is imported into the vehicle target detection network model, the road vehicle video stream photographed by the monocular camera of the unmanned aerial vehicle is detected, the pixel coordinates of the four corners of the target detection frame are recorded, the pixel coordinates of the four corners of the target detection frame are averaged, the center pixel coordinates of the target detection frame are preliminarily obtained, the misrecognized center pixel coordinates are eliminated from the preliminarily obtained center pixel coordinates of the target detection frame, and the final center pixel coordinates of the target detection frame are obtained as the pixel coordinates of the center of the target vehicle.

[0025] The center pixel coordinates of the target detection frame are preliminarily obtained, and the final center pixel coordinates of the target detection frame are obtained, which comprises:

[0026] The preliminarily obtained center pixel coordinates of the target detection frame are classified into multiple categories / regions;

[0027] The preliminarily obtained center pixel coordinates of the multiple categories / regions are classified by using a density / distance clustering method;

[0028] The category / region to which the pixel coordinates of the center of the target vehicle belong is extracted from the classification result, the misrecognized pixel coordinates are eliminated, and the final center pixel coordinates of the target detection frame are obtained.

[0029] Preferably, the vehicle target detection neural network model uses YOLOv5 algorithm as the target detection algorithm.

[0030] In step S3, the road vehicle video stream photographed by the unmanned aerial vehicle is used to obtain the edge pixel coordinates of the two sides of the road, which is realized based on an edge detection algorithm, and comprises:

[0031] The video of each frame of the road vehicle video stream is cropped and scaled, the target road area is labeled, and a training data set of a road detection neural network model is prepared;

[0032] The road detection neural network model is trained based on the training data set, and the optimal road detection neural network model is obtained;

[0033] Detecting the road vehicle video stream captured by the monocular camera of the unmanned aerial vehicle by using the optimal road detection neural network model to obtain a sequence of gray images of road extraction results;

[0034] Detecting pixel coordinates of road edges in the pixel coordinate system of the monocular camera in the sequence of gray images as road edge pixel coordinates on both sides of the road based on an edge detection algorithm.

[0035] Preferably, the road detection neural network model selects U-Net as the road detection neural network.

[0036] In step S4, the first world coordinate system is established based on the hovering position of the unmanned aerial vehicle, the camera intrinsic parameter, the height of the unmanned aerial vehicle relative to the road plane, and the conversion relationship between the camera coordinate system and the pixel coordinate system, and the center point pixel coordinates of the vehicle and the road edge pixel coordinates on both sides of the road are converted into world coordinates by using the following expression:

[0037]

[0038] wherein A is the camera intrinsic parameter of the monocular camera of the unmanned aerial vehicle, T z is the height of the unmanned aerial vehicle relative to the target road plane, (u, v) is the center point pixel coordinates or the road edge pixel coordinates on both sides of the road, (U, V, T z ) is the world coordinates of the center point pixel coordinates or the road edge pixel coordinates on both sides of the road in the first world coordinate system, dX and dY respectively represent the physical length of one pixel in the X and Y directions on the photosensitive plate of the monocular camera, f is the image distance, u0 and v0 respectively represent the coordinates of the center of the photosensitive plate of the monocular camera in the pixel coordinate system, and alpha represents the angle between the horizontal side and the vertical side of the photosensitive plate of the monocular camera.

[0039] The marker comprises a checkerboard and / or a nested two-dimensional code picture, the nested two-dimensional code picture is composed of two two-dimensional code marks with different sizes, the smaller two-dimensional code mark is placed in a predetermined position in the inner blank area of the larger two-dimensional code mark to form the nested two-dimensional code picture, and the marker is laid on both sides of the road

[0040] The unmanned aerial vehicle hovering in the air captures the road vehicle video stream in the traffic scene as the data source, the vehicle trajectory is extracted by using the computer vision method, the problem of unstable signal of satellite positioning technology in mountainous roads can be avoided, and the high-definition vehicle trajectory can be obtained; the safety degree of the driving trajectory at each time point can be analyzed by using the obtained vehicle trajectory, the driving safety factor can be effectively improved, and the traffic accident rate can be reduced. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1A flowchart of a road vehicle trajectory extraction method based on a monocular camera of a UAV according to an embodiment of the present application.

[0042] Figure 2 A schematic diagram of a nested two-dimensional code marker according to an embodiment of the present application. DETAILED DESCRIPTION

[0043] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in a clear and complete manner in conjunction with the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application.

[0044] The embodiments of the present application use a road vehicle video stream in a traffic scene photographed by a UAV hovering in the air as a data source, use a computer vision method to extract a vehicle trajectory in a road, can obtain a high-definition vehicle trajectory, can be used to analyze the safety of a driving trajectory at each time point, can effectively improve a driving safety factor, and can reduce a traffic accident rate.

[0045] As shown in FIG. 1, the road vehicle trajectory extraction method based on a monocular camera of a UAV includes the following steps: Figure 1

[0046] S1: calibrating a monocular camera of a UAV based on a calibration video stream containing a marker photographed by the monocular camera of the UAV in a traffic scene, and obtaining camera intrinsic parameters of the monocular camera.

[0047] In step S1, the following sub-steps are included:

[0048] S11: acquiring a calibration video stream containing a marker from different heights and different angles by using the monocular camera of the UAV, the flight height of the UAV should not exceed 1.5 meters, and obtaining multiple pictures containing the marker by frame extraction on the obtained calibration video stream. The shape and size information of the marker should be known, and there are at least four known corner points (feature points).

[0049] The marker can be a two-dimensional code icon, an H-shaped landmark, a checkerboard icon, etc., and the road surface of a target road already has corresponding marker information, such as lane marking, lane marker, and other traffic facilities, etc.

[0050] S12: detecting the corner points of the marker in the picture and obtaining pixel coordinate values of the corner points of the marker; a computer vision algorithm such as Harris algorithm or SIFT algorithm can be used to detect feature points such as marker corner points in the marker picture and obtain pixel coordinate values of the marker corner points.

[0051] ​Taking a checkerboard icon as a marker as an example, when the checkerboard icon marker is detected, a monocular camera of the unmanned aerial vehicle can be calibrated based on Zhang Zhengyou camera calibration method.

[0052] S13, according to the actual size of the known marker and a world coordinate system established with the center point of the marker as the origin, physical coordinate values of the corner points of the marker are obtained;

[0053] S14, based on the corresponding relationship between the pixel coordinate values and the physical coordinate values, a camera intrinsic matrix of the monocular camera of the unmanned aerial vehicle is obtained;

[0054] Specifically, firstly, a homography matrix corresponding to the mapping relationship between the pixel coordinate values and the physical coordinate values is obtained based on the camera optical imaging principle, and the method is as follows:

[0055] In the camera imaging system, four coordinate systems including a world coordinate system, a camera coordinate system, an image coordinate system and a pixel coordinate system are included. In the embodiment of the application, the world coordinate system is fixed on the marker picture, and a world coordinate system is established with the center point of the marker as the origin, so that the physical coordinates (U, V, W) of any point on the marker picture under the world coordinate are W=0. The conversion relationship between the world coordinate system established in this step and the pixel coordinate system is as follows:

[0056]

[0057]

[0058] Wherein, A is the intrinsic parameter or intrinsic matrix of the camera, (U, V, W) is the physical coordinates of a point under the world coordinate system established in this step, wherein W=0, (u, v) is the pixel coordinates of the corresponding point under the pixel coordinate system, Z is a scale factor, f is an image distance, dX and dY respectively represent the physical length of one pixel in the X and Y directions on the camera photosensitive plate, u0 and v0 respectively represent the coordinates of the center of the camera photosensitive plate under the pixel coordinate system, and a represents the angle between the horizontal side and the vertical side of the camera photosensitive plate. R1 and R2 represent the first two column vectors of the rotation matrix R of the relative position relationship between the camera coordinate system and the world coordinate system established in this step, and T represents the translation vector of the relative position relationship between the camera coordinate system and the world coordinate system established in this step.

[0059] A (R1 R2 T) is denoted as G, G is the homography matrix corresponding to the mapping relationship between the pixel coordinate values and the physical coordinate values, and G is denoted as (G1, G2, G3), so that:

[0060]

[0061] The above formula is established for the corner points on the same marker picture. The pixel coordinate value (u, v) and the physical coordinate value (U, V) of the corner points on the same marker picture are substituted into the above formula, and then a homography matrix G corresponding to the marker picture is obtained. Further, the pixel coordinate value and the world coordinate value of the corner points of other marker pictures are substituted into the above formula, and then a plurality of different homography matrices are obtained.

[0062] Because the camera intrinsic matrix is only related to the internal structure parameters of the camera, the obtained plurality of homography matrices are combined with the unit orthogonal relationship of the rotation vector in the camera extrinsic parameters, as shown in the following formula, and the least square method is used for regression, so that the intrinsic matrix A of the monocular camera of the unmanned aerial vehicle is obtained.

[0063] G=A(R1 R2 T)

[0064] R1 T R2=0

[0065] R1 T R1=R2 T R2=1

[0066] The above steps are repeated multiple times to obtain a plurality of calibration results of the intrinsic matrix, and one of the stable results is selected as the final camera intrinsic parameter or camera intrinsic matrix.

[0067] S2: based on the video stream containing the marker shot by the unmanned aerial vehicle hovering, obtaining the unmanned aerial vehicle pose and integrating the unmanned aerial vehicle attitude, calculating the unmanned aerial vehicle pose after the parallelism of the unmanned aerial vehicle and the road plane meets the requirements, and obtaining the height of the unmanned aerial vehicle relative to the road plane;

[0068] Specifically, when obtaining the height of the unmanned aerial vehicle relative to the target road plane, the unmanned aerial vehicle hovers in the air, shoots a marker video stream containing a marker, and then calculates the pose information of the unmanned aerial vehicle relative to the target road plane based on the marker information in the marker video stream and the camera intrinsic parameter. Then, according to the pose information, the flight attitude of the unmanned aerial vehicle is adjusted, so that the parallelism of the unmanned aerial vehicle and the target road plane meets the parallelism threshold, and then the pose information of the unmanned aerial vehicle after the attitude adjustment is calculated. In this way, based on the pose information of the unmanned aerial vehicle after the attitude adjustment, the height of the unmanned aerial vehicle relative to the target road plane can be obtained.

[0069] In step S2, when obtaining the pose information of the unmanned aerial vehicle relative to the target road plane, the pose estimation algorithm is realized, which can include the following sub-steps:

[0070] S21, obtaining the coordinate value of the corner point of the marker in the image coordinate system in a single frame image of the marker video stream;

[0071] As an embodiment, preferably, in the present application, the marker used in calculating the height of the UAV is a nested two-dimensional code picture, as shown in Figure 2 The picture is composed of two two-dimensional codes with obvious size difference, and the nested two-dimensional code picture is formed by inserting a small two-dimensional code into the middle blank part of a large two-dimensional code. The picture is laid on both sides of the road as a marker, which can adapt to the two-dimensional code recognition task of different camera angles, as shown in Figure 2 The nested two-dimensional code picture as a marker is taken as an example, and an AprilTag-based pose estimation algorithm is used to obtain the pose information of the UAV, and the specific calculation process is as follows:

[0072] Taking the marker shown in Figure 2 As an example, first, the intensity and direction of the gradient of all pixels in each frame of the video stream are calculated using computer vision algorithms, and then the intensity and direction of the gradient are clustered, and the equation of a straight line is fitted using a weighted least squares method to detect line segments in the image. After detecting all the line segments, extract the line segments that are close enough to the end of the previous line segment within a distance threshold and comply with the counterclockwise direction, and detect the rectangular information; according to the obtained marker corner point, line segment and rectangular information, judge whether the marker is a correct marker, and prevent interference from other shape information; obtain the coordinates or coordinate values of the corner points of the marker judged as the correct marker in the image coordinate system, which are used for subsequent calculation and processing of the coordinate values of the corner points of the marker in the image coordinate system;

[0073] S22, taking the center point of the marker as the origin of a world coordinate system, obtaining the physical coordinate values of the corner points of the marker according to the known size of the marker;

[0074] S23, based on the optical imaging principle of the camera, obtaining the homography matrix corresponding to the mapping relationship between the coordinate values of the corner points of the marker in the image coordinate system and the physical coordinate values;

[0075] S24, according to the monocular camera intrinsic parameter obtained in step S1, combining the obtained homography matrix, and combining the unit orthogonal relationship of the rotation vector in the camera extrinsic parameter, obtaining the extrinsic parameter of the camera, and finally obtaining the pose information of the UAV relative to the target road plane.

[0076] The calculation method is as follows:

[0077]

[0078] Wherein, H is a homography matrix, B is the transformation matrix of the world coordinate system established in this step to the camera coordinate system, f is the image distance, dX and dY respectively represent the physical length of one pixel in the X and Y directions on the camera photosensitive plate, R1 and R2 represent the first two column vectors of the rotation matrix R of the relative position relationship between the camera coordinate system and the world coordinate system established in this step, T represents the translation vector of the relative position relationship between the camera coordinate system and the world coordinate system established in this step. S is the scale factor, the homography matrix H is a homogeneous matrix, so the scale factor s needs to be added.

[0079] Expanding the formula H = sB(R1 R2 T) can obtain:

[0080]

[0081]

[0082] According to the corresponding relationship between the matrix B and the monocular camera internal parameter A obtained in step 1, It can be approximately considered that the monocular camera internal parameter A is θ = 90°, so

[0083] Because It is known that R1 and R2 are unit vectors, and there is an orthogonal relationship, so R1, R2 and T can be calculated from the above formula, wherein T z The absolute value of the height relative to the target road plane.

[0084] Because R1 and R2 satisfy the following equation relationship

[0085]

[0086] Wherein, θ and φ are the rotation angles of the world coordinate system established in this step around the x and y axes when transforming to the camera coordinate system, so θ and φ can be obtained according to the above formula θ and φ, and then the attitude information of the unmanned aerial vehicle relative to the landmark, that is, the attitude information of the unmanned aerial vehicle relative to the ground, that is, the attitude information of the unmanned aerial vehicle relative to the target road plane, is obtained.

[0087] S3: based on the road vehicle video stream photographed by the unmanned aerial vehicle monocular camera after attitude adjustment, the vehicle center point pixel coordinates and the road side edge pixel coordinates of the target vehicle are obtained;

[0088] Wherein, according to the road vehicle video stream photographed by the unmanned aerial vehicle monocular camera without attitude adjustment, the pixel coordinates of the vehicle center point in the video stream are obtained, and based on the target detection algorithm, the following substeps are included:

[0089] S31, based on the unmanned aerial vehicle monocular camera shooting road traffic, extract video every frame picture, adopt data set labeling tool to label the vehicle in the picture, make the data set of target detection network model;

[0090] The data set is enhanced by rotation, scaling and cropping methods to ensure the effect of the picture itself and improve the generalization ability of the network model.

[0091] S32, select YOLOv5 algorithm as the target detection algorithm, divide the data set into training set and validation set, use the training set and validation set to train the target detection network model, and obtain the optimal weight parameter.

[0092] S33, import the optimal weight parameter into the test algorithm of the target detection network model, detect the road traffic shot by the unmanned aerial vehicle monocular camera, record the pixel coordinates of the four corners of the detection frame, calculate the average value of the pixel coordinates of the four corners of the detection frame, and obtain the center pixel coordinates of the detection frame. The coordinates are regarded as the center point pixel coordinates of the vehicle.

[0093] S34, eliminate the misrecognition center pixel coordinates output by the target detection algorithm, and obtain the final target detection frame center pixel coordinates as the target vehicle center point pixel coordinates.

[0094] In step S34, a clustering algorithm based on density or distance is used to eliminate the misrecognition center pixel coordinates output by the target detection algorithm. By using a density-based clustering method and setting a suitable density threshold, or using a distance-based clustering method and setting a suitable number of classes, the obtained center pixel coordinates are classified. Taking the density-based clustering algorithm as an example, the following sub-steps are included:

[0095] The center pixel coordinates data output by the target detection algorithm in step S33 are divided into classes to obtain multiple classes (i.e. multiple regions). In the density-based clustering algorithm, DNSCAN algorithm is selected. The DNSCAN algorithm first sets two hyperparameters, search radius eps and number of data within search radius minPts; according to the two parameters, search all data within search radius eps, if the number of data is greater than minPts, it is considered as a class, otherwise it is not, continue to traverse the next point, until all points are traversed, and the classification of center pixel coordinates data is completed; in the classification result, the class to which the vehicle center pixel coordinates of the target vehicle belong is extracted, and the misrecognized pixel coordinates are eliminated, so as to obtain the final target detection frame center pixel coordinates.

[0096] According to the road vehicle video stream shot by the unmanned aerial vehicle monocular camera, the pixel coordinates of the edges of the road on both sides in the video stream are obtained, and based on the road detection algorithm, the following sub-steps are included:

[0097] S35, based on the unmanned aerial vehicle monocular camera shooting road traffic, extract video every frame picture, the picture is cut and zoomed, such as making the image size 1024*1024 pixels, using the annotation software to label the area where the road is, the pixel value of the marked road area is 255, and the pixel value of the remaining area is 0, complete the training data set of road detection neural network.

[0098] S35, select U-Net as the road detection neural network (of course, not limited to U-Net network, also can be other network), based on the training data set, train U-Net, obtain the optimal U-Net model, use the model to detect the road traffic photographed by the unmanned aerial vehicle monocular camera, and obtain the gray image sequence of the road extraction result.

[0099] S35, based on the Sobel operator, the road edge in the gray image sequence of the gray image is detected, and the pixel coordinates of the point with large gradient, that is, the pixel coordinates of the road edge line, are obtained.

[0100] S4. A world coordinate system is established based on the hovering position of the unmanned aerial vehicle, based on the camera internal parameter, the height of the unmanned aerial vehicle relative to the target road plane, and according to the conversion relationship between the camera coordinate system and the pixel coordinate system, the vehicle center pixel coordinates and the road side edge pixel coordinates are converted into the world coordinates in the world coordinate system established in this step, so that the world coordinates of the target vehicle in the world coordinate system established in this step are obtained:

[0101] Based on the monocular camera internal parameter obtained in step S1 and the height of the unmanned aerial vehicle relative to the HHSF road plane obtained in step S2, a world coordinate system is established based on the hovering position of the unmanned aerial vehicle, and according to the conversion relationship between the camera coordinate system and the pixel coordinate system, the vehicle center pixel coordinates and the road edge line pixel coordinates output in step S4 are converted into world coordinates, and the world coordinates of the target vehicle in the world coordinate system established in this step are taken.

[0102] The calculation method is as follows:

[0103]

[0104] Wherein, A is the internal parameter matrix of the monocular camera of the unmanned aerial vehicle obtained in step 1, T z The height of the unmanned aerial vehicle relative to the road plane obtained in step 2, (u, v) is the pixel coordinates output by the clustering algorithm or the road detection algorithm, (U, V, T z ) is the world coordinates of the point in the world coordinate system established in this step.

[0105] S5. Arrange the world coordinates in order to obtain the vehicle trajectory graph;

[0106] That is, the world coordinates of the target vehicle in the world coordinate system are arranged in turn according to the sequence of the video sequences of the road vehicle video stream, and then the trajectory graph information of the target vehicle in the road vehicle video stream is obtained.

[0107] The embodiment of the present application takes the video photographed by the unmanned aerial vehicle as the data source, uses the computer vision method to extract the vehicle trajectory, can obtain the high-definition vehicle trajectory, provides reliable vehicle driving information for the driver, can be used for analyzing the safety degree of the driving trajectory at each time point, can effectively improve the driving safety factor, and reduces the traffic accident rate.

[0108] The above only describes the preferred embodiments of the present application, and it should be pointed out that for ordinary skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, and these improvements and refinements should be considered as the protection scope of the present application.

Claims

1. A method for road vehicle trajectory extraction based on unmanned aerial vehicle monocular camera, characterized in that, The method comprises the steps of: S1. Calibrating a monocular camera of a UAV based on a video stream containing a marker shot by the UAV, and obtaining camera intrinsic parameters; S2. Obtaining a UAV pose based on a video stream containing a marker shot by the UAV in a hovering state, adjusting the attitude of the UAV, and calculating the UAV pose after the parallelism between the UAV and the road plane meets the requirements, to obtain the height of the UAV relative to the road plane; S3. Obtaining vehicle center point pixel coordinates and road side edge pixel coordinates based on a road vehicle video stream shot by the UAV; S4. Establishing a first world coordinate system with the hovering position of the UAV as the base point, converting the vehicle center point pixel coordinates and the road side edge pixel coordinates into world coordinates based on the camera intrinsic parameters, the height of the UAV relative to the road plane, and the conversion relationship between the camera coordinate system and the pixel coordinate system; the following expression is used: wherein A is an intrinsic parameter of the monocular camera of the UAV, T z is a height of the UAV relative to a target road plane, (u, v) is a central point pixel coordinate or a road two-side edge pixel coordinate, (U, V, T z ) is a world coordinate in a first world coordinate system corresponding to the central point pixel coordinate or the road two-side edge pixel coordinate, dX, dY respectively represent a physical length of one pixel in the X and Y directions on a photosensitive plate of the monocular camera, f is an image distance, u0, v0 respectively represent coordinates of a center of the photosensitive plate of the monocular camera in a pixel coordinate system, and a represents an angle between a horizontal side and a vertical side of the photosensitive plate of the monocular camera. S5. Arranging the world coordinates in order to obtain a vehicle trajectory diagram. 2.The method of claim 1, wherein, In step S1, the camera intrinsic parameters are obtained by calibrating the monocular camera of the UAV based on the video stream containing the marker shot by the UAV, which comprises the steps of: Frame extraction is performed on the calibration video stream containing the marker shot by the UAV in the traffic scene to obtain multiple pictures containing the marker; The corner points of the marker in the pictures are detected to obtain the pixel coordinate values of the corner points of the marker; The physical coordinate values of the corner points of the marker are obtained according to the size of the marker and a second world coordinate system established with the center point of the marker as the origin; The camera intrinsic parameter matrix of the monocular camera of the UAV is obtained based on the corresponding relationship between the pixel coordinate values and the physical coordinate values of the monocular camera; The camera intrinsic parameter matrix is repeatedly obtained multiple times to obtain multiple calibration results of the camera intrinsic parameter matrix, and one calibration result of the camera intrinsic parameter matrix is selected as the camera intrinsic parameter. 3.The method of claim 1, wherein, In step S2, the UAV pose is obtained based on the video stream containing the marker shot by the UAV in the hovering state, which is realized based on a pose estimation algorithm, and comprises the steps of: Detecting preset information of the marker in the video stream to determine whether the marker is a preset marker; if yes, obtaining the coordinate values of the corner points of the preset marker in a single image in the marker video stream in an image coordinate system; A third world coordinate system is established with the center point of the marker as the origin, and the physical coordinate values of the corner points of the marker in the third world coordinate system are obtained according to the size of the marker; A homography matrix corresponding to the mapping relationship between the coordinate values of the corner points of the marker in the image coordinate system and the physical coordinate values in the third world coordinate system is obtained; The camera extrinsic parameters of the monocular camera are calculated according to the camera intrinsic parameters, combined with the homography matrix and the unit orthogonal relationship of the rotation vector in the camera extrinsic parameters, to obtain the pose information of the UAV relative to the target road plane.

4. The method of claim 1, wherein, In step S3, the vehicle center point pixel coordinates are obtained based on the road vehicle video stream shot by the UAV, which is realized based on a target detection algorithm, and comprises the steps of: Each frame of video of the road vehicle video stream is extracted, cropped and scaled to label the target vehicle in the picture, and a data set of a vehicle target detection network model is prepared; The data set is enhanced by rotation, scaling and cropping; A training set of the data set is used to train a vehicle target detection network model, and a verification set of the data set is used for verification to obtain optimal weight parameters; The optimal weight parameters are imported into the vehicle target detection network model to detect a road vehicle video stream captured by the monocular camera of the unmanned aerial vehicle, record pixel coordinates of four corners of a target detection frame, average the pixel coordinates of the four corners of the target detection frame, preliminarily obtain a center pixel coordinate of the target detection frame, eliminate a misrecognized center pixel coordinate from the preliminarily obtained center pixel coordinate of the target detection frame, and obtain a final center pixel coordinate of the target detection frame as a target vehicle center pixel coordinate.

5. The method of claim 4, wherein, The method for eliminating the misrecognized center pixel coordinate from the preliminarily obtained center pixel coordinate of the target detection frame to obtain the final center pixel coordinate of the target detection frame includes: Classifying the preliminarily obtained center pixel coordinates of the target detection frame to obtain a plurality of classes or regions; Classifying the preliminarily obtained center pixel coordinates of the target detection frame corresponding to the plurality of classes or regions by using a density or distance clustering method; Extracting a class or region to which a vehicle center pixel coordinate of a target vehicle belongs from a classification result, eliminating a misrecognized pixel coordinate, and obtaining a final center pixel coordinate of the target detection frame.

6. The method of claim 4, wherein, The vehicle target detection network model uses a YOLOv5 algorithm as a target detection algorithm.

7. The method of claim 1, wherein, In step S3, the road vehicle video stream captured by the unmanned aerial vehicle is used to obtain road edge pixel coordinates, which are achieved based on an edge detection algorithm, including: Extracting each frame of a video of the road vehicle video stream, cropping and scaling the extracted frame, labeling a target road region, and preparing a training data set of a road detection neural network model; Training the road detection neural network model based on the training data set to obtain an optimal road detection neural network model; Using the optimal road detection neural network model to detect the road vehicle video stream captured by the monocular camera of the unmanned aerial vehicle to obtain a gray image sequence of a road extraction result; Detecting pixel coordinates of a road edge in the gray image sequence in a pixel coordinate system of the monocular camera as road edge pixel coordinates based on an edge detection algorithm. 8.The method of claim 7, wherein, The road detection neural network model selects U-Net as a road detection neural network.

9. The method of claim 1, wherein, The marker includes a checkerboard and / or a nested two-dimensional code picture; the nested two-dimensional code picture is composed of two two-dimensional code markers with different sizes, the smaller two-dimensional code marker is placed in a predetermined position in an inner blank area of the larger two-dimensional code marker to form the nested two-dimensional code picture, and the marker is laid on both sides of the road.

Images