Vehicle four-dimensional trajectory generation method based on road end camera picture

Abstract

The invention relates to the technical field of automatic driving, and particularly discloses a vehicle four-dimensional track generation method based on road end camera pictures, which comprises the following steps: acquiring crossroad images shot by a plurality of cameras at a crossroad, and performing synchronous processing on the crossroad images; inputting the plurality of synchronized crossroad images into an extraction network model to obtain crossroad features; inputting the crossroad features into a tracking network model to obtain a multi-target tracking result of the crossroad, and generating a plurality of vehicle four-dimensional tracks of the crossroad according to the multi-target tracking result; inputting the crossroad features into a prediction network model to obtain a crossroad road boundary; and abandoning the vehicle four-dimensional trajectory outside the road boundary and the vehicle four-dimensional trajectory with too short tracking time in the plurality of vehicle four-dimensional trajectories to obtain a final vehicle four-dimensional trajectory of the crossroad. According to the invention, the vehicle can be helped to more accurately understand the traffic information of the crossroad, so that the spatial modeling of the crossroad is more accurate.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a method for generating a vehicle four-dimensional trajectory based on road-side camera images. Background Art

[0002] Vehicle trajectory tracking is an important part of autonomous driving. With trajectory information for reference, downstream algorithms can better understand road information, including the location of crosswalks, lane information, etc., thereby helping the autonomous driving system to output more reasonable decisions.

[0003] At present, the mainstream vehicle trajectory generation method is to use the vehicle driving on the road, record the road surface information around the vehicle through the camera on the roof and other equipment, and use the deep learning method to generate the three-dimensional trajectory of the surrounding vehicles in the plane trajectory map with the two-dimensional spatial coordinates x, y and time t. The disadvantage of this method is that due to the limited field of view of the vehicle-side sensor, the limited overlap of the coverage area of ​​multiple sensors, and the fact that the vehicle is always in motion, it is difficult to accurately capture the continuous trajectory information of other vehicles, and it is difficult to cover all vehicle trajectories in the specified area. Summary of the invention

[0004] In order to overcome the problems of information loss and inaccurate generation in existing vehicle trajectory generation methods, the present invention provides a vehicle four-dimensional trajectory generation method based on road-side camera images. For multiple road-side cameras with consistent timing at an intersection, continuous vehicle four-dimensional trajectory information across perspectives of the road section is generated, and the method is highly robust to the problem of inconsistent data streams of multiple cameras caused by network delays, data loss, etc.

[0005] As a first aspect of the present invention, a method for generating a four-dimensional trajectory of a vehicle based on a roadside camera image is provided, and the method for generating a four-dimensional trajectory of a vehicle based on a roadside camera image comprises:

[0006] Step S1: acquiring intersection images captured by multiple cameras at the intersection, and synchronously processing the intersection images captured by the multiple cameras to obtain multiple synchronized intersection images;

[0007] Step S2: inputting the synchronized multiple intersection images into a road-end feature information extraction network model to extract features, so as to obtain intersection features;

[0008] Step S3: inputting the intersection feature into a three-dimensional multi-target tracking network model to perform target tracking to obtain a multi-target tracking result of the intersection, and then generating a plurality of four-dimensional trajectories of vehicles at the intersection according to the multi-target tracking result of the intersection;

[0009] Step S4: inputting the intersection features into the intersection boundary prediction network model to perform road boundary prediction to obtain the intersection road boundary;

[0010] Step S5: Discard the vehicle four-dimensional trajectories outside the intersection road boundary and the vehicle four-dimensional trajectories with tracking time less than a preset value in the multiple vehicle four-dimensional trajectories to obtain the final vehicle four-dimensional trajectory of the intersection

[0011] Furthermore, the step of acquiring the intersection images captured by multiple cameras at the intersection and synchronously processing the intersection images captured by the multiple cameras to obtain multiple synchronized intersection images further includes:

[0012] N through the intersection c A camera captures N images in a period of time. c Data streams from different angles to form a data stream set Among them, each data stream S i Contains T i The intersection image at each time stamp is

[0013] Select the latest first frame timestamp in the data stream set S as the starting timestamp t start , and select the earliest last frame timestamp in the data stream set S as the end timestamp t end , at the starting timestamp t start To the end timestamp t end Take a sampling point every Δt in the time interval between sampling points;

[0014] For each sampling point, in each data stream S i Select the sampling point whose timestamp difference is less than the threshold θ t And the data corresponding to the timestamp with the smallest difference value is taken as the data stream S i The data under the timestamp of the sampling point to achieve clock synchronization;

[0015] If the data stream S i There is no value less than the threshold θ t timestamp, the data stream S i The data at the sampling point timestamp defaults to null values; discard the data streams with the number of null values ​​greater than the threshold α, and for the remaining data streams, select the frame interpolation method or frame replication method to fill the null values, or choose to retain the null values, so as to obtain the synchronized data stream set Among them, each synchronized data stream S i 'Include N c images of the intersection from different angles.

[0016] Furthermore, the step of inputting the synchronized multiple intersection images into a roadside feature information extraction network model to extract features to obtain intersection features further includes:

[0017] For each synchronized data stream S i ', that is, N at each sampling point timestamp c The N intersection images under the current sampling point timestamp t c The intersection image is input into the road-end feature information extraction network model for feature extraction to obtain the intersection feature f corresponding to the current sampling point timestamp t t Finally, the intersection is obtained from the starting timestamp t start To the end timestamp t end N t intersection features to form an intersection feature set

[0018] Furthermore, the step of inputting the intersection features into a three-dimensional multi-target tracking network model for target tracking to obtain a multi-target tracking result for the intersection also includes:

[0019] Get the detection query vector Q corresponding to the current sampling point timestamp t t The tracking query vector T corresponding to the previous sampling point timestamp t-1 t-1 ;

[0020] The tracking query vector T corresponding to the last sampling point timestamp t-1 t-1 , the detection query vector Q corresponding to the current sampling point timestamp t t The intersection feature f corresponding to the current sampling point timestamp t t are input into the three-dimensional multi-target tracking network model for target tracking, so as to output the tracking query vector T corresponding to the current sampling point timestamp t t Multi-target tracking results corresponding to the current sampling point timestamp t Finally, the intersection is obtained from the starting timestamp t start To the end timestamp t end N t Multi-target tracking results to form a multi-target tracking result set for the intersection

[0021] Furthermore, the multi-target tracking result corresponding to the current sampling point timestamp t It includes the three-dimensional position information, length, width and height information, tilt angle information and speed information of all vehicles tracked at the current sampling point timestamp t.

[0022] Furthermore, the generating of a plurality of vehicle four-dimensional trajectories at the intersection according to the multi-target tracking result at the intersection further includes:

[0023] According to the multi-target tracking result set at the intersection Generate multiple four-dimensional trajectories of vehicles at the intersection.

[0024] Furthermore, the step of inputting the intersection features into an intersection boundary prediction network model to perform road boundary prediction to obtain the intersection road boundary also includes:

[0025] The intersection is limited to a square of length E, and then the intersection feature set F r Inputting the information into the intersection position prediction network model for position prediction to predict the position information of the intersection, and establishing the intersection coordinate system according to the position information of the intersection;

[0026] The intersection feature set F r The data are input into the intersection boundary prediction network model to perform road boundary prediction, so as to predict the intersection road boundary.

[0027] Furthermore, the road-side feature information extraction network model and the three-dimensional multi-target tracking network model constitute a road-side vehicle tracking network model, which also includes:

[0028] Obtaining a training set, and training the road-side vehicle tracking network model according to the training set;

[0029] Among them, the training set includes a first data set, a second data set and a third data set. The first data set includes intersection images taken by some cameras at the intersection during a historical time period, the second data set includes intersection images taken by all cameras at the intersection during a historical time period, and the third data set includes intersection images after noise processing is performed on the intersection images in the second data set.

[0030] The method for generating vehicle four-dimensional trajectories based on road-side camera images provided by the present invention has the following advantages: a coordinate system model of the intersection is established by using the camera at the intersection, multi-data stream image information of multiple cameras in a specified time interval is used to synchronize timestamps, and three-dimensional target tracking is performed, thereby generating accurate vehicle four-dimensional trajectories, and discarding vehicle four-dimensional trajectories outside the road boundary and vehicle four-dimensional trajectories with too short tracking time, to obtain the final vehicle four-dimensional trajectory of the intersection, which can help vehicles understand the traffic information of the road section more accurately, making the spatial modeling of the road section more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the present invention, but do not constitute a limitation of the present invention.

[0032] Figure 1 The present invention provides a flow chart of a method for generating a vehicle four-dimensional trajectory based on roadside camera images.

[0033] Figure 2 A schematic diagram of the process of extracting intersection features provided by the present invention.

[0034] Figure 3 A schematic diagram of the generation process of the three-dimensional target tracking results provided by the present invention.

[0035] Figure 4 A schematic diagram of the generation process of the vehicle four-dimensional trajectory provided by the present invention.

[0036] Figure 5 A schematic diagram of the training process of the road-side vehicle tracking network model provided by the present invention. DETAILED DESCRIPTION

[0037] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0038] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is only a part of the embodiment of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work should fall within the scope of protection of the present invention.

[0039] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so as to describe the embodiments of the present invention described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0040] In this embodiment, a method for generating a vehicle four-dimensional trajectory based on roadside camera images is provided. Figure 1The flowchart of the method for generating a vehicle four-dimensional trajectory based on road-side camera images provided by the present invention is as follows: Figure 1 As shown, the vehicle four-dimensional trajectory generation method based on roadside camera images includes:

[0041] Step S1: acquiring intersection images captured by multiple cameras at the intersection, and synchronously processing the intersection images captured by the multiple cameras to obtain multiple synchronized intersection images;

[0042] Preferably, Figure 2 As shown, the method of acquiring the intersection images captured by multiple cameras at the intersection and synchronously processing the intersection images captured by the multiple cameras to obtain multiple synchronized intersection images also includes:

[0043] N through the intersection c A camera captures N images in a period of time. c Data streams from different angles to form a data stream set Among them, each data stream S i Contains T i The intersection image at each time stamp is

[0044] Select the latest first frame timestamp in the data stream set S as the starting timestamp t start , and select the earliest last frame timestamp in the data stream set S as the end timestamp t end , at the starting timestamp t start To the end timestamp t end Take a sampling point every Δt in the time interval between sampling points;

[0045] For each sampling point, in each data stream S i Select the sampling point whose timestamp difference is less than the threshold θ t And the data corresponding to the timestamp with the smallest difference value is taken as the data stream S i The data under the timestamp of the sampling point to achieve clock synchronization;

[0046] If the data stream S i There is no value less than the threshold θ t timestamp, the data stream S i The data at the sampling point timestamp defaults to null values; discard the data streams with the number of null values ​​greater than the threshold α, and for the remaining data streams, select the frame interpolation method or frame replication method to fill the null values, or choose to retain the null values, so as to obtain the synchronized data stream set Among them, each synchronized data stream Si 'Include N c images of the intersection from different angles.

[0047] Step S2: inputting the synchronized multiple intersection images into a road-end feature information extraction network model to extract features, so as to obtain intersection features;

[0048] Preferably, Figure 2 As shown, the step of inputting the synchronized multiple intersection images into a roadside feature information extraction network model for feature extraction to obtain intersection features also includes:

[0049] For each synchronized data stream S i ', that is, N at each sampling point timestamp c The N intersection images under the current sampling point timestamp t c The intersection image is input into the road-end feature information extraction network model based on convolutional neural network and Trasnsformer for feature extraction to obtain the intersection feature f corresponding to the current sampling point timestamp t t Finally, the intersection is obtained from the starting timestamp t start To the end timestamp t end N t intersection features to form an intersection feature set

[0050] Step S3: inputting the intersection feature into a three-dimensional multi-target tracking network model to perform target tracking to obtain a multi-target tracking result of the intersection, and then generating a plurality of four-dimensional trajectories of vehicles at the intersection according to the multi-target tracking result of the intersection;

[0051] Preferably, Figure 3 As shown, the inputting of the intersection feature into the three-dimensional multi-target tracking network model for target tracking to obtain the multi-target tracking result of the intersection also includes:

[0052] Get the learnable detection query vector Q corresponding to the current sampling point timestamp t t The tracking query vector T corresponding to the previous sampling point timestamp t-1 t-1 ;

[0053] The tracking query vector T corresponding to the last sampling point timestamp t-1 t-1 , the detection query vector Q corresponding to the current sampling point timestamp t t The intersection feature f corresponding to the current sampling point timestamp t tare input into the three-dimensional multi-target tracking network model based on the Transformer decoder for target tracking, so as to output the tracking query vector T corresponding to the current sampling point timestamp t t Multi-target tracking results corresponding to the current sampling point timestamp t Finally, the intersection is obtained from the starting timestamp t start To the end timestamp t end N t Multi-target tracking results to form a multi-target tracking result set for the intersection

[0054] It should be noted that both the detection query vector and the tracking query vector are high-dimensional feature representation vectors, and their parameters will be updated as the model is trained.

[0055] Specifically, the multi-target tracking result corresponding to the current sampling point timestamp t It includes the three-dimensional position information, length, width and height information, tilt angle information and speed information of all vehicles tracked at the current sampling point timestamp t.

[0056] Preferably, the step of generating a plurality of four-dimensional trajectories of vehicles at the intersection according to the multi-target tracking result at the intersection further includes:

[0057] According to the multi-target tracking result set at the intersection Generate multiple four-dimensional trajectories of vehicles at the intersection.

[0058] Step S4: inputting the intersection features into the intersection boundary prediction network model to perform road boundary prediction to obtain the intersection road boundary;

[0059] Preferably, Figure 4 As shown, the step of inputting the intersection features into the intersection boundary prediction network model to perform road boundary prediction to obtain the intersection road boundary also includes:

[0060] The intersection is limited to a square of length E, and then the intersection feature set F r Input into the Transformer-based intersection position prediction network model for position prediction, so as to predict the position information of the intersection, and establish an intersection coordinate system according to the position information of the intersection; wherein the position information of the intersection includes the center point O of the intersection and the road deviation angle ω of the intersection;

[0061] The intersection feature set F rThe data is input into the intersection boundary prediction network model based on the convolutional neural network to predict the road boundary, so as to predict the intersection road boundary.

[0062] Step S5: Figure 4 As shown, the vehicle four-dimensional trajectories outside the road boundary of the intersection and the vehicle four-dimensional trajectories with tracking time less than a preset value among the multiple vehicle four-dimensional trajectories are discarded to obtain the final vehicle four-dimensional trajectory of the intersection.

[0063] Specifically, Figure 5 As shown, the road-side feature information extraction network model and the three-dimensional multi-target tracking network model constitute a road-side vehicle tracking network model, which also includes:

[0064] A training set is obtained, and the road-end vehicle tracking network model is trained according to the training set; wherein the training set includes a first data set, a second data set and a third data set, the first data set includes images of the intersection taken by some cameras at the intersection during a historical time period, the second data set includes images of the intersection taken by all cameras at the intersection during a historical time period, and the third data set includes images of the intersection after noise processing is performed on the images of the intersection in the second data set.

[0065] It should be noted that the three data sets in the training set were randomly shuffled and sent to the road-side vehicle tracking network model for training. For the first data set, the road-side vehicle tracking network model was trained to generate the vehicle's four-dimensional trajectory using partial data; for the second data set, the road-side vehicle tracking network model was trained to generate the vehicle's four-dimensional trajectory normally; for the third data set, the road-side vehicle tracking network model was trained to generate the vehicle's four-dimensional trajectory even when some information was missing.

[0066] The above cross-view image data enhancement training method can make the road-side vehicle tracking network model adapt to the number of cameras N at the intersection c Uncertain and unevenly distributed situations. Since there are many or few cameras on a real road, it is possible that a camera may fail. In addition, the image quality of the camera may be affected by weather or other reasons. To address these situations, the road-side vehicle tracking network model must also be able to stably generate the vehicle's four-dimensional trajectory.

[0067] The method for generating a vehicle four-dimensional trajectory based on roadside camera images provided by the present invention generates a continuous four-dimensional trajectory diagram of vehicles across perspectives in the intersection area based on multi-perspective images taken by multiple roadside cameras at the intersection, wherein the four dimensions include coordinates x, y, z in three-dimensional space, and time t, in addition to vehicle information such as the size of the vehicle and the speed at each moment. Since the roadside camera covers a wide range of road surfaces and has a large inclination, and multiple roadside cameras jointly shoot the same intersection area, it is possible to better detect and track vehicles traveling on the road, thereby generating an accurate four-dimensional trajectory of the vehicle, and discarding the four-dimensional trajectory of vehicles outside the road boundary and the four-dimensional trajectory of vehicles with too short tracking time, to obtain the final four-dimensional trajectory of the vehicle. The present invention can help vehicles understand the traffic information of the road section more accurately, making the spatial modeling of the road section more accurate, which is of great value to the decision-making planning of autonomous driving vehicles, the generation of road elements and topological information, etc.

[0068] It is to be understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of the present invention, but the present invention is not limited thereto. For those of ordinary skill in the art, various modifications and improvements can be made without departing from the spirit and essence of the present invention, and these modifications and improvements are also considered to be within the scope of protection of the present invention.

Claims

1. A method for generating a vehicle four-dimensional trajectory based on roadside camera images, characterized in that: The vehicle four-dimensional trajectory generation method based on roadside camera images includes: Step S1: acquiring intersection images captured by multiple cameras at the intersection, and synchronously processing the intersection images captured by the multiple cameras to obtain multiple synchronized intersection images; Step S2: inputting the synchronized multiple intersection images into a road-end feature information extraction network model to extract features, so as to obtain intersection features; Step S3: inputting the intersection feature into a three-dimensional multi-target tracking network model to perform target tracking to obtain a multi-target tracking result of the intersection, and then generating a plurality of four-dimensional trajectories of vehicles at the intersection according to the multi-target tracking result of the intersection; Step S4: inputting the intersection features into the intersection boundary prediction network model to perform road boundary prediction to obtain the intersection road boundary; Step S5: discarding the vehicle four-dimensional trajectories outside the road boundary of the intersection and the vehicle four-dimensional trajectories with tracking time less than a preset value among the multiple vehicle four-dimensional trajectories to obtain the final vehicle four-dimensional trajectory of the intersection.

2. The method for generating a vehicle four-dimensional trajectory based on roadside camera images according to claim 1, characterized in that: The method of acquiring the intersection images captured by multiple cameras at the intersection and synchronously processing the intersection images captured by the multiple cameras to obtain multiple synchronized intersection images further includes: N through the intersection c A camera captures N images in a period of time. c Data streams from different angles to form a data stream set Among them, each data stream S i Contains T i The intersection image at each time stamp is Select the latest first frame timestamp in the data stream set S as the starting timestamp t start , and select the earliest last frame timestamp in the data stream set S as the end timestamp t end , at the starting timestamp t start To the end timestamp t end Take a sampling point every Δt in the time interval between sampling points; For each sampling point, in each data stream S i Select the sampling point whose timestamp difference is less than the threshold θ t And the data corresponding to the timestamp with the smallest difference value is taken as the data stream S i The data under the timestamp of the sampling point to achieve clock synchronization; If the data stream S i There is no value less than the threshold θ t timestamp, the data stream S i The data at the sampling point timestamp defaults to null values; discard the data streams with the number of null values ​​greater than the threshold α, and for the remaining data streams, select the frame interpolation method or frame replication method to fill the null values, or choose to retain the null values, so as to obtain the synchronized data stream set Among them, each synchronized data stream S i ' contains N c images of the intersection from different angles.

3. The method for generating a vehicle four-dimensional trajectory based on roadside camera images according to claim 2, characterized in that: The step of inputting the synchronized multiple intersection images into a roadside feature information extraction network model to extract features to obtain intersection features also includes: For each synchronized data stream S i ′, that is, N at each sampling point timestamp c The N intersection images under the current sampling point timestamp t c The intersection image is input into the road-end feature information extraction network model for feature extraction to obtain the intersection feature f corresponding to the current sampling point timestamp t t Finally, the intersection is obtained from the starting timestamp t start To the end timestamp t end N t intersection features to form an intersection feature set 4. The method for generating a vehicle four-dimensional trajectory based on roadside camera images according to claim 3, characterized in that: The step of inputting the intersection features into a three-dimensional multi-target tracking network model to perform target tracking to obtain a multi-target tracking result for the intersection also includes: Get the detection query vector Q corresponding to the current sampling point timestamp t t The tracking query vector T corresponding to the previous sampling point timestamp t-1 t-1 ; The tracking query vector T corresponding to the last sampling point timestamp t-1 t-1 , the detection query vector Q corresponding to the current sampling point timestamp t t The intersection feature f corresponding to the current sampling point timestamp t t are input into the three-dimensional multi-target tracking network model for target tracking, so as to output the tracking query vector T corresponding to the current sampling point timestamp t t Multi-target tracking results corresponding to the current sampling point timestamp t Finally, the intersection is obtained from the starting timestamp t start To the end timestamp t end N t Multi-target tracking results to form a multi-target tracking result set for the intersection 5. The method for generating a vehicle four-dimensional trajectory based on roadside camera images according to claim 4, characterized in that: The multi-target tracking result corresponding to the current sampling point timestamp t It includes the three-dimensional position information, length, width and height information, tilt angle information and speed information of all vehicles tracked at the current sampling point timestamp t.

6. The method for generating a vehicle four-dimensional trajectory based on roadside camera images according to claim 4, characterized in that: The step of generating a plurality of vehicle four-dimensional trajectories at the intersection according to the multi-target tracking result at the intersection further includes: According to the multi-target tracking result set at the intersection Generate multiple four-dimensional trajectories of vehicles at the intersection.

7. The method for generating a vehicle four-dimensional trajectory based on roadside camera images according to claim 3, characterized in that: The step of inputting the intersection features into the intersection boundary prediction network model to perform road boundary prediction to obtain the intersection road boundary also includes: The intersection is limited to a square of length E, and then the intersection feature set F r Inputting the information into the intersection position prediction network model for position prediction to predict the position information of the intersection, and establishing the intersection coordinate system according to the position information of the intersection; The intersection feature set F r The data are input into the intersection boundary prediction network model to perform road boundary prediction, so as to predict the intersection road boundary.

8. The method for generating a vehicle four-dimensional trajectory based on roadside camera images according to claim 1, characterized in that: The road-end feature information extraction network model and the three-dimensional multi-target tracking network model constitute a road-end vehicle tracking network model, and further include: Obtaining a training set, and training the road-side vehicle tracking network model according to the training set; Among them, the training set includes a first data set, a second data set and a third data set. The first data set includes intersection images taken by some cameras at the intersection during a historical time period, the second data set includes intersection images taken by all cameras at the intersection during a historical time period, and the third data set includes intersection images after noise processing is performed on the intersection images in the second data set.

Images