A method for generating vehicle four-dimensional trajectory based on roadside camera images

Abstract

The present invention relates to the field of autonomous driving technology, and specifically discloses a method for generating a four-dimensional vehicle trajectory based on roadside camera images, comprising: obtaining intersection images captured by multiple cameras at the intersection and performing synchronous processing on them; inputting the synchronized multiple intersection images into an extraction network model to obtain intersection features; inputting the intersection features into a tracking network model to obtain multi-target tracking results at the intersection, and generating multiple four-dimensional vehicle trajectories at the intersection based on the multi-target tracking results; inputting the intersection features into a prediction network model to obtain the intersection road boundary; and discarding the four-dimensional vehicle trajectories outside the road boundary and the four-dimensional vehicle trajectories with too short tracking time in the multiple four-dimensional vehicle trajectories to obtain the final four-dimensional vehicle trajectory at the intersection. The present invention can help vehicles understand the traffic information of the intersection more accurately, making the spatial modeling of the intersection more precise.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a method for generating a four-dimensional trajectory of a vehicle based on roadside camera images. Background Art

[0002] Vehicle trajectory tracking is an important component of autonomous driving. With reference to trajectory information, downstream algorithms can better understand road information, including the location of crosswalks, lane information, etc., thereby helping the autonomous driving system output more reasonable decisions.

[0003] Currently, mainstream vehicle trajectory generation methods rely on a vehicle driving on the road, using devices such as roof-mounted cameras to record road conditions around the vehicle. Deep learning methods are then used to generate the three-dimensional trajectories of surrounding vehicles, represented by their two-dimensional spatial coordinates x, y and time t, within a two-dimensional trajectory map. This approach suffers from limitations in capturing the continuous trajectory of other vehicles due to the limited field of view of on-board sensors, the limited overlap of multiple sensor coverage areas, and the constant motion of the vehicle itself. Furthermore, it struggles to accurately capture the trajectory of all vehicles within a given area. Summary of the Invention

[0004] To overcome the problems of information loss and inaccurate generation in existing vehicle trajectory generation methods, the present invention provides a method for generating four-dimensional vehicle trajectories based on roadside camera images. This method generates continuous four-dimensional vehicle trajectory information across the entire road section for multiple roadside cameras with consistent timing at an intersection. The method is robust to inconsistencies in the data streams from multiple cameras caused by network latency, data loss, and other factors.

[0005] As a first aspect of the present invention, a method for generating a four-dimensional vehicle trajectory based on roadside camera images is provided. The method for generating a four-dimensional vehicle trajectory based on roadside camera images comprises:

[0006] Step S1: Acquire intersection images captured by multiple cameras at the intersection, and perform synchronization processing on the intersection images captured by the multiple cameras to obtain multiple synchronized intersection images;

[0007] Step S2: inputting the synchronized multiple intersection images into a road-end feature information extraction network model to perform feature extraction to obtain intersection features;

[0008] Step S3: inputting the intersection features into a three-dimensional multi-target tracking network model to perform target tracking to obtain a multi-target tracking result for the intersection, and then generating a plurality of four-dimensional trajectories of vehicles at the intersection based on the multi-target tracking result for the intersection;

[0009] Step S4: inputting the intersection features into the intersection boundary prediction network model to perform road boundary prediction to obtain the intersection road boundary;

[0010] Step S5: Discard the vehicle 4D trajectories outside the intersection road boundary and the vehicle 4D trajectories with tracking time less than a preset value in the multiple vehicle 4D trajectories to obtain the final vehicle 4D trajectory of the intersection

[0011] Furthermore, the step of acquiring intersection images captured by multiple cameras at the intersection and synchronously processing the intersection images captured by the multiple cameras to obtain multiple synchronized intersection images further includes:

[0012] N through the intersection c A camera captures N images in a period of time. c Data streams from different angles to form a data stream set Among them, each data stream S i Contains T i The intersection image at each time stamp, i.e.

[0013] Select the latest first frame timestamp in the data stream set S as the starting timestamp t start , and select the earliest last frame timestamp in the data stream set S as the end timestamp t end , at the starting timestamp t start To the end timestamp t end Take a sampling point every Δt in the time interval between sampling points;

[0014] For each sampling point, in each data stream S i The time stamp difference between the sampling point and the sampling point is less than the threshold θ t And the data corresponding to the timestamp with the smallest difference value is taken as the data stream S i The data at the sampling point timestamp is used to achieve clock synchronization;

[0015] If the data stream S i There is no value less than the threshold θ t timestamp, the data stream S i The data at the sampling point timestamp defaults to null values; discard the data streams with the number of null values ​​greater than the threshold α, and for the remaining data streams, select the frame interpolation method or the frame replication method to fill the null values, or choose to retain the null values, so as to obtain the synchronized data stream set Among them, each synchronized data stream S i 'Include N c images of the intersection from different angles.

[0016] Furthermore, the step of inputting the synchronized multiple intersection images into a roadside feature information extraction network model to extract features to obtain intersection features further includes:

[0017] For each synchronized data stream S i ', that is, N at each sampling point timestamp c The intersection image is N under the current sampling point timestamp t. c The intersection image is input into the road-end feature information extraction network model for feature extraction to obtain the intersection feature f corresponding to the current sampling point timestamp t t Finally, we get the intersection from the starting timestamp t start To the end timestamp t end N between t intersection features to form an intersection feature set

[0018] Furthermore, the step of inputting the intersection features into a three-dimensional multi-target tracking network model for target tracking to obtain a multi-target tracking result for the intersection further includes:

[0019] Get the detection query vector Q corresponding to the current sampling point timestamp t t The tracking query vector T corresponding to the last sampling point timestamp t-1 t-1 ;

[0020] The tracking query vector T corresponding to the last sampling point timestamp t-1 t-1 , the detection query vector Q corresponding to the current sampling point timestamp t t The intersection feature f corresponding to the current sampling point timestamp t t are input into the three-dimensional multi-target tracking network model for target tracking, so as to output the tracking query vector T corresponding to the current sampling point timestamp t t Multi-target tracking results corresponding to the current sampling point timestamp t Finally, the intersection is obtained from the starting timestamp t start To the end timestamp t end N between t Multi-target tracking results to form the multi-target tracking result set for the intersection

[0021] Furthermore, the multi-target tracking result corresponding to the current sampling point timestamp t It includes the three-dimensional position information, length, width and height information, tilt angle information and speed information of all vehicles tracked at the current sampling point timestamp t.

[0022] Furthermore, the step of generating a plurality of four-dimensional vehicle trajectories at the intersection based on the multi-target tracking results at the intersection further includes:

[0023] According to the multi-target tracking result set at the intersection Generate multiple four-dimensional trajectories of vehicles at the intersection.

[0024] Furthermore, the step of inputting the intersection features into an intersection boundary prediction network model to perform road boundary prediction to obtain the intersection road boundary further includes:

[0025] The intersection is limited to a square of length E, and then the intersection feature set F r Inputting the information into the intersection position prediction network model for position prediction to predict the position information of the intersection, and establishing an intersection coordinate system based on the position information of the intersection;

[0026] The intersection feature set F r The data is input into the intersection boundary prediction network model to perform road boundary prediction, so as to predict the intersection road boundary.

[0027] Furthermore, the road-end feature information extraction network model and the three-dimensional multi-target tracking network model constitute a road-end vehicle tracking network model, further comprising:

[0028] Obtaining a training set, and training the road-end vehicle tracking network model according to the training set;

[0029] Among them, the training set includes a first data set, a second data set and a third data set. The first data set includes intersection images taken by some cameras at the intersection during a historical time period. The second data set includes intersection images taken by all cameras at the intersection during a historical time period. The third data set includes intersection images after noise processing is performed on the intersection images in the second data set.

[0030] The method for generating four-dimensional vehicle trajectories based on roadside camera images provided by the present invention has the following advantages: a coordinate system model of the intersection is established using cameras at the intersection, multi-data stream image information from multiple cameras in a specified time interval is used to synchronize timestamps, and three-dimensional target tracking is performed to generate accurate four-dimensional vehicle trajectories. The four-dimensional trajectories of vehicles outside the road boundary and the four-dimensional trajectories of vehicles with too short tracking time are discarded to obtain the final four-dimensional vehicle trajectory for the intersection. This can help vehicles understand the traffic information of the road section more accurately, making the spatial modeling of the road section more precise. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the present invention, but do not constitute a limitation of the present invention.

[0032] Figure 1 This is a flow chart of the method for generating a vehicle's four-dimensional trajectory based on roadside camera images provided by the present invention.

[0033] Figure 2 Schematic diagram of the intersection feature extraction process provided by the present invention.

[0034] Figure 3 Schematic diagram of the process of generating three-dimensional target tracking results provided by the present invention.

[0035] Figure 4 Schematic diagram of the generation process of the vehicle four-dimensional trajectory provided by the present invention.

[0036] Figure 5 Schematic diagram of the training process of the road-side vehicle tracking network model provided by the present invention. DETAILED DESCRIPTION

[0037] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments of the present invention may be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0038] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0039] It should be noted that the terms "first," "second," and the like in the specification and claims of the present invention and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate for the embodiments of the present invention described herein. In addition, the terms "including," "having," and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to these processes, methods, products, or apparatuses.

[0040] In this embodiment, a method for generating a vehicle four-dimensional trajectory based on roadside camera images is provided. Figure 1The flowchart of the method for generating a vehicle four-dimensional trajectory based on roadside camera images provided by the present invention is as follows: Figure 1 As shown, the method for generating a vehicle four-dimensional trajectory based on roadside camera images includes:

[0041] Step S1: Acquire intersection images captured by multiple cameras at the intersection, and perform synchronization processing on the intersection images captured by the multiple cameras to obtain multiple synchronized intersection images;

[0042] Preferably, Figure 2 As shown, the method of acquiring intersection images captured by multiple cameras at the intersection and synchronously processing the intersection images captured by the multiple cameras to obtain multiple synchronized intersection images further includes:

[0043] N through the intersection c A camera captures N images in a period of time. c Data streams from different angles to form a data stream set Among them, each data stream S i Contains T i The intersection image at each time stamp, i.e.

[0044] Select the latest first frame timestamp in the data stream set S as the starting timestamp t start , and select the earliest last frame timestamp in the data stream set S as the end timestamp t end , at the starting timestamp t start To the end timestamp t end Take a sampling point every Δt in the time interval between sampling points;

[0045] For each sampling point, in each data stream S i The time stamp difference between the sampling point and the sampling point is less than the threshold θ t And the data corresponding to the timestamp with the smallest difference value is taken as the data stream S i The data at the sampling point timestamp is used to achieve clock synchronization;

[0046] If the data stream S i There is no value less than the threshold θ t timestamp, the data stream S i The data at the sampling point timestamp defaults to null values; discard the data streams with the number of null values ​​greater than the threshold α, and for the remaining data streams, select the frame interpolation method or the frame replication method to fill the null values, or choose to retain the null values, so as to obtain the synchronized data stream set Among them, each synchronized data stream Si 'Include N c images of the intersection from different angles.

[0047] Step S2: inputting the synchronized multiple intersection images into a road-end feature information extraction network model to perform feature extraction to obtain intersection features;

[0048] Preferably, Figure 2 As shown, the step of inputting the synchronized multiple intersection images into a road-end feature information extraction network model to extract features to obtain intersection features further includes:

[0049] For each synchronized data stream S i ', that is, N at each sampling point timestamp c The intersection image is N under the current sampling point timestamp t. c The intersection image is input into the road-end feature information extraction network model based on convolutional neural network and Transformer to extract features, so as to obtain the intersection feature f corresponding to the current sampling point timestamp t t Finally, we get the intersection from the starting timestamp t start To the end timestamp t end N between t intersection features to form an intersection feature set

[0050] Step S3: inputting the intersection features into a three-dimensional multi-target tracking network model to perform target tracking to obtain a multi-target tracking result for the intersection, and then generating a plurality of four-dimensional trajectories of vehicles at the intersection based on the multi-target tracking result for the intersection;

[0051] Preferably, Figure 3 As shown, the method of inputting the intersection features into a three-dimensional multi-target tracking network model for target tracking to obtain a multi-target tracking result for the intersection also includes:

[0052] Get the learnable detection query vector Q corresponding to the current sampling point timestamp t t The tracking query vector T corresponding to the last sampling point timestamp t-1 t-1 ;

[0053] The tracking query vector T corresponding to the last sampling point timestamp t-1 t-1 , the detection query vector Q corresponding to the current sampling point timestamp t t The intersection feature f corresponding to the current sampling point timestamp t tAll of them are input into the three-dimensional multi-target tracking network model based on Transformer decoder for target tracking, so as to output the tracking query vector T corresponding to the current sampling point timestamp t t Multi-target tracking results corresponding to the current sampling point timestamp t Finally, the intersection is obtained from the starting timestamp t start To the end timestamp t end N between t Multi-target tracking results to form the multi-target tracking result set for the intersection

[0054] It should be noted that the detection query vector and the tracking query vector are both high-dimensional feature representation vectors, and their parameters will be updated as the model is trained.

[0055] Specifically, the multi-target tracking result corresponding to the current sampling point timestamp t It includes the three-dimensional position information, length, width and height information, tilt angle information and speed information of all vehicles tracked at the current sampling point timestamp t.

[0056] Preferably, the step of generating a plurality of four-dimensional vehicle trajectories at the intersection based on the multi-target tracking results at the intersection further includes:

[0057] According to the multi-target tracking result set at the intersection Generate multiple four-dimensional trajectories of vehicles at the intersection.

[0058] Step S4: inputting the intersection features into the intersection boundary prediction network model to perform road boundary prediction to obtain the intersection road boundary;

[0059] Preferably, Figure 4 As shown, the step of inputting the intersection features into the intersection boundary prediction network model to perform road boundary prediction to obtain the intersection road boundary also includes:

[0060] The intersection is limited to a square of length E, and then the intersection feature set F r Inputting the data into a Transformer-based intersection position prediction network model for position prediction to predict the position information of the intersection, and establishing an intersection coordinate system based on the position information of the intersection; wherein the position information of the intersection includes the center point O of the intersection and the road deviation angle ω of the intersection;

[0061] The intersection feature set F rThe input is input into the intersection boundary prediction network model based on the convolutional neural network to perform road boundary prediction to predict the intersection road boundary.

[0062] Step S5: Figure 4 As shown, the vehicle four-dimensional trajectories outside the intersection road boundary and the vehicle four-dimensional trajectories with tracking time less than a preset value are discarded from the multiple vehicle four-dimensional trajectories to obtain the final vehicle four-dimensional trajectory of the intersection.

[0063] Specifically, if Figure 5 As shown, the road-end feature information extraction network model and the three-dimensional multi-target tracking network model constitute a road-end vehicle tracking network model, which also includes:

[0064] A training set is obtained, and the road-end vehicle tracking network model is trained based on the training set; wherein the training set includes a first data set, a second data set, and a third data set, the first data set includes images of the intersection captured by some cameras at the intersection during a historical time period, the second data set includes images of the intersection captured by all cameras at the intersection during a historical time period, and the third data set includes images of the intersection after noise processing is performed on the intersection images in the second data set.

[0065] It should be noted that the three training datasets were randomly shuffled and then fed into the roadside vehicle tracking network model for training. For the first dataset, the model was trained to generate 4D vehicle trajectories using partial data. For the second dataset, the model was trained normally. For the third dataset, the model was trained to generate 4D vehicle trajectories even when some information was missing.

[0066] The above data enhancement training method for cross-view images can make the road-side vehicle tracking network model adapt to the number of cameras N at the intersection. c Uncertainty and uneven distribution. Since the number of cameras on real roads varies, there is also the possibility of camera failure. In addition, camera image quality may be affected by weather or other factors. To address these situations, the roadside vehicle tracking network model must be able to stably generate vehicle 4D trajectories.

[0067] The present invention provides a method for generating a four-dimensional vehicle trajectory based on roadside camera images. Based on multi-viewpoint images taken by multiple roadside cameras at an intersection, a continuous four-dimensional trajectory diagram of vehicles across the intersection area is generated. The four dimensions include the coordinates x, y, and z in three-dimensional space, as well as time t. In addition, it also includes vehicle information such as the size of the vehicle and its speed at each moment. Since the roadside cameras cover a wide range of road surfaces and have large inclination angles, and multiple roadside cameras jointly capture the same intersection area, they can better detect and track vehicles traveling on the road, thereby generating accurate four-dimensional vehicle trajectories, and discarding the four-dimensional trajectories of vehicles outside the road boundary and the four-dimensional trajectories of vehicles with too short tracking time to obtain the final four-dimensional vehicle trajectory. The present invention can help vehicles understand the traffic information of the road section more accurately, making the spatial modeling of the road section more precise, which is of great value in the decision-making planning of autonomous vehicles, the generation of road elements and topological information, and other aspects.

[0068] It will be understood that the above embodiments are merely exemplary embodiments for illustrating the principles of the present invention, and the present invention is not limited thereto. Those skilled in the art will appreciate that various modifications and improvements can be made without departing from the spirit and substance of the present invention, and such modifications and improvements are also considered to be within the scope of protection of the present invention.

Claims

1. A method for generating a vehicle four-dimensional trajectory based on roadside camera images, characterized in that: The method for generating a four-dimensional vehicle trajectory based on roadside camera images includes: Step S1: Acquire intersection images captured by multiple cameras at the intersection, and perform synchronization processing on the intersection images captured by the multiple cameras to obtain multiple synchronized intersection images; Step S2: inputting the synchronized multiple intersection images into a road-end feature information extraction network model to perform feature extraction to obtain intersection features; Step S3: inputting the intersection features into a three-dimensional multi-target tracking network model to perform target tracking to obtain a multi-target tracking result for the intersection, and then generating a plurality of four-dimensional trajectories of vehicles at the intersection based on the multi-target tracking result for the intersection; Step S4: inputting the intersection features into the intersection boundary prediction network model to perform road boundary prediction to obtain the intersection road boundary; Step S5: discarding the vehicle four-dimensional trajectories outside the intersection road boundary and the vehicle four-dimensional trajectories with tracking time less than a preset value from the multiple vehicle four-dimensional trajectories to obtain the final vehicle four-dimensional trajectory of the intersection.

2. The method for generating a vehicle four-dimensional trajectory based on roadside camera images according to claim 1, characterized in that: The method of acquiring intersection images captured by multiple cameras at the intersection and synchronously processing the intersection images captured by the multiple cameras to obtain multiple synchronized intersection images further includes: N through the intersection c A camera captures N images in a period of time. c Data streams from different angles to form a data stream set Among them, each data stream S i Contains T i The intersection image at each time stamp, i.e. Select the latest first frame timestamp in the data stream set S as the starting timestamp t start , and select the earliest last frame timestamp in the data stream set S as the end timestamp t end , at the starting timestamp t start To the end timestamp t end Take a sampling point every Δt in the time interval between sampling points; For each sampling point, in each data stream S i The time stamp difference between the sampling point and the sampling point is less than the threshold θ t And the data corresponding to the timestamp with the smallest difference value is taken as the data stream S i The data at the sampling point timestamp is used to achieve clock synchronization; If the data stream S i There is no value less than the threshold θ t timestamp, the data stream S i The data at the sampling point timestamp defaults to null values; discard the data streams with the number of null values ​​greater than the threshold α, and for the remaining data streams, select the frame interpolation method or the frame replication method to fill the null values, or choose to retain the null values, so as to obtain the synchronized data stream set Among them, each synchronized data stream S i ' contains N c images of the intersection from different angles.

3. The method for generating a vehicle four-dimensional trajectory based on roadside camera images according to claim 2, characterized in that: The step of inputting the synchronized multiple intersection images into a road-end feature information extraction network model to extract features to obtain intersection features further includes: For each synchronized data stream S i ′, that is, N at each sampling point timestamp c The intersection image is N under the current sampling point timestamp t. c The intersection image is input into the road-end feature information extraction network model for feature extraction to obtain the intersection feature f corresponding to the current sampling point timestamp t t Finally, we get the intersection from the starting timestamp t start To the end timestamp t end N between t intersection features to form an intersection feature set 4. The method for generating a vehicle four-dimensional trajectory based on roadside camera images according to claim 3, characterized in that: The step of inputting the intersection features into a three-dimensional multi-target tracking network model to perform target tracking to obtain a multi-target tracking result for the intersection further includes: Get the detection query vector Q corresponding to the current sampling point timestamp t t The tracking query vector T corresponding to the last sampling point timestamp t-1 t-1 ; The tracking query vector T corresponding to the last sampling point timestamp t-1 t-1 , the detection query vector Q corresponding to the current sampling point timestamp t t The intersection feature f corresponding to the current sampling point timestamp t t are input into the three-dimensional multi-target tracking network model for target tracking, so as to output the tracking query vector T corresponding to the current sampling point timestamp t t Multi-target tracking results corresponding to the current sampling point timestamp t Finally, the intersection is obtained from the starting timestamp t start To the end timestamp t end N between t Multi-target tracking results to form the multi-target tracking result set for the intersection 5. The method for generating a vehicle four-dimensional trajectory based on roadside camera images according to claim 4, characterized in that: The multi-target tracking result corresponding to the current sampling point timestamp t It includes the three-dimensional position information, length, width and height information, tilt angle information and speed information of all vehicles tracked at the current sampling point timestamp t.

6. The method for generating a vehicle four-dimensional trajectory based on roadside camera images according to claim 4, characterized in that: The step of generating a plurality of vehicle four-dimensional trajectories at the intersection according to the multi-target tracking results at the intersection further includes: According to the multi-target tracking result set at the intersection Generate multiple four-dimensional trajectories of vehicles at the intersection.

7. The method for generating a vehicle four-dimensional trajectory based on roadside camera images according to claim 3, characterized in that: The step of inputting the intersection features into an intersection boundary prediction network model to perform road boundary prediction to obtain the intersection road boundary further includes: The intersection is limited to a square of length E, and then the intersection feature set F r Inputting the information into the intersection position prediction network model for position prediction to predict the position information of the intersection, and establishing an intersection coordinate system based on the position information of the intersection; The intersection feature set F r The data is input into the intersection boundary prediction network model to perform road boundary prediction, so as to predict the intersection road boundary.

8. The method for generating a vehicle four-dimensional trajectory based on roadside camera images according to claim 1, characterized in that: The road-end feature information extraction network model and the three-dimensional multi-target tracking network model constitute a road-end vehicle tracking network model, further comprising: Obtaining a training set, and training the road-end vehicle tracking network model according to the training set; Among them, the training set includes a first data set, a second data set and a third data set. The first data set includes intersection images taken by some cameras at the intersection during a historical time period. The second data set includes intersection images taken by all cameras at the intersection during a historical time period. The third data set includes intersection images after noise processing is performed on the intersection images in the second data set.

Images