Vehicle track prediction system and method based on roadside aerial view angle

Abstract

The invention relates to the technical field of vehicle track prediction, in particular to a vehicle track prediction system and method based on a roadside aerial view angle, and the system comprises a roadside multi-sensor data collection module which is used for collecting the environment data of a traffic environment; the roadside aerial view angle feature generation module is used for obtaining a two-dimensional image and performing geometric transformation on the two-dimensional image to generate three-dimensional representation of an aerial view angle; the space-time continuous vehicle track prediction module is used for predicting the motion track of the vehicle in the target future time period according to the three-dimensional representation; and the vehicle track truth value labeling module is used for generating labeling data of the real motion track of the vehicle so as to train or update the deep learning model. Therefore, the problems that a traditional prediction mode is limited by sufficiency and representativeness of data and separate processing of prediction, perception and planning modules, consequently, complex and dynamic traffic environments cannot be effectively handled, and the real-time performance and accuracy of decision making are affected are solved.

Description

Technical Field

[0001] This application relates to the field of vehicle trajectory prediction technology, and in particular to a vehicle trajectory prediction system and method based on a roadside bird's-eye view. Background Technology

[0002] In related technologies, existing vehicle trajectory prediction methods can be mainly classified into the following categories:

[0003] (1) Historical data-based methods: These methods typically rely on historical trajectory data and statistical models to predict the future movement of traffic participants by analyzing their past behavior. Common techniques include time series analysis and Markov models.

[0004] (2) Deep Learning-Based Methods: In recent years, deep learning technology has been widely used in vehicle trajectory prediction. These methods utilize neural network models, such as recurrent neural networks (RNNs) and convolutional neural networks (CNNs), to capture complex movement patterns through training on large amounts of data. Although deep learning methods have improved accuracy, they still face problems of insufficient data sufficiency and model generalization ability, especially in new or unseen traffic scenarios.

[0005] (3) Sensor-based methods: Some methods rely on onboard sensors (such as LiDAR, cameras, etc.) to acquire real-time environmental information. These methods obtain the status of surrounding vehicles and pedestrians through perception modules and combine them with prediction models to predict trajectories. However, the accuracy of this method is limited by the sensor range and environmental conditions, which may lead to information loss and inaccurate predictions.

[0006] (4) Modular autonomous driving systems: In many autonomous driving systems, the prediction module is separate from the perception, planning, and control modules. Although this structure is easy to implement, delays and error propagation during information processing may lead to untimely and inaccurate prediction results.

[0007] However, in related technologies, prediction methods based on historical data are limited by their weak adaptability to environmental changes and cannot handle complex traffic scenarios; prediction methods based on deep learning require training on large amounts of data to capture complex movement patterns, facing problems of insufficient data sufficiency and model generalization ability, especially in new or unseen traffic scenarios; the accuracy of prediction methods based on sensor input is limited by sensor range and environmental conditions, which may lead to information loss and inaccurate predictions; in modular autonomous driving systems, because the prediction module is often separate from the perception, planning and control modules, delays and error propagation occur during information processing, resulting in untimely and inaccurate prediction results, which urgently need improvement. Summary of the Invention

[0008] This application provides a vehicle trajectory prediction system and method based on a roadside bird's-eye view to solve the problems in related technologies, such as the inability to effectively cope with complex and dynamic traffic environments and the impact on the real-time performance and accuracy of decision-making due to the limitations of traditional prediction methods in terms of data sufficiency and representativeness and the separate processing of prediction, perception and planning modules.

[0009] The first aspect of this application provides a vehicle trajectory prediction system based on a roadside bird's-eye view, comprising: a roadside multi-sensor data acquisition module for acquiring environmental data of the traffic environment; a roadside bird's-eye view feature generation module for obtaining a two-dimensional image based on the environmental data and performing geometric transformations on the two-dimensional image to generate a three-dimensional representation of the bird's-eye view; a spatiotemporally continuous vehicle trajectory prediction module for predicting the vehicle's motion trajectory in a target future time period based on the three-dimensional representation using a pre-trained deep learning model; and a vehicle trajectory ground truth annotation module for generating labeled data of the vehicle's actual motion trajectory to train or update the deep learning model.

[0010] Through the aforementioned technical means, this application embodiment can integrate multi-camera data with advanced bird's-eye view algorithms, utilizing perspective information from multiple roadside cameras to construct a comprehensive bird's-eye view representation of the vehicle's surrounding environment. This integrates perception, motion prediction, and trajectory analysis into a unified framework. Furthermore, by directly analyzing real-time image data, the system can quickly predict vehicle trajectories, thereby optimizing traffic flow management and improving safety. Compared to existing trajectory prediction systems, the roadside bird's-eye view-based vehicle trajectory prediction system can significantly reduce traffic accident risks through comprehensive environmental information capture and enhance adaptability to complex traffic scenarios, making decision-making more accurate and flexible.

[0011] Optionally, in one embodiment of this application, the roadside bird's-eye view feature generation module includes: a conversion unit for converting the RGB image and lidar point cloud in the environmental data into a BEV image; a first generation unit for generating a spatial context feature map based on the BEV image as the two-dimensional image; an extraction unit for extracting at least one global feature from the two-dimensional image based on a pre-built Transformer network; and a second generation unit for performing feature stitching and fusion on the at least one global feature to obtain a three-dimensional representation of the bird's-eye view.

[0012] Through the aforementioned technical means, this application embodiment can synthesize two-dimensional images and three-dimensional point clouds from a bird's-eye view using BEV technology to generate two-dimensional images, thereby integrating data from multiple sensors and providing richer environmental information. Furthermore, it utilizes multiple attention heads of the Transformer network to focus on features from different angles to extract global features, thereby enhancing the model's expressive power and its ability to handle complex traffic scenarios. Moreover, it further stitches and fuses these global features, merging features obtained from different Transformer layers to enhance the model's comprehensive feature representation capabilities, forming richer feature representations. This significantly improves the accuracy of vehicle trajectory prediction, reduces the false positive rate, and provides reliable data support for intelligent traffic management.

[0013] Optionally, in one embodiment of this application, the roadside multi-sensor data acquisition module includes: at least one RGB camera and at least one fisheye camera installed on the traffic lights at the intersection for acquiring the RGB images; and at least one lidar installed on the traffic lights at the intersection for acquiring the lidar point cloud.

[0014] Through the above-mentioned technical means, the embodiments of this application can use an RGB camera to acquire color image data and further identify and classify traffic participants; a fisheye camera can be used to provide a wide-angle field of view, thereby covering a larger monitoring area and reducing blind spots; and a lidar laser scanning can be used to acquire high-precision three-dimensional spatial data, providing distance and depth information; thereby realizing comprehensive environmental perception and real-time monitoring and analysis of traffic conditions.

[0015] Optionally, in one embodiment of this application, the spatiotemporally continuous vehicle trajectory prediction module is further used to combine deep learning algorithms and physical models, comprehensively consider time and space factors, and use the deep learning model to extract features from the three-dimensional representation data to identify at least one of traffic participants, road information, and traffic light status, and generate the motion trajectory in the target future time period.

[0016] Through the above-mentioned technical means, the embodiments of this application can combine deep learning algorithms and physical models, comprehensively consider time and space factors, and then identify traffic participants, road information, and traffic light status to generate the movement trajectory within the target future time period. This can effectively predict the behavior of traffic participants and take corresponding control measures in a timely manner, reduce the risk of traffic accidents, and improve road safety.

[0017] Optionally, in one embodiment of this application, the vehicle trajectory truth labeling module is further used to extract the vehicle's motion trajectory from the environmental data, wherein the motion trajectory includes at least one of the trajectory data: timestamp and spatial coordinates.

[0018] Through the aforementioned technical means, the embodiments of this application can accurately extract the true value of the spatiotemporal trajectory of vehicle motion, providing high-quality training samples for deep learning models, thereby ensuring that the models can learn the real laws of vehicle motion. At the same time, its multi-dimensional and multi-source adaptability, as well as its ability to support model updates, can further enable the models to better adapt to complex and dynamic real-world traffic scenarios and respond to environmental changes in real time, ensuring that they maintain high efficiency under diverse traffic conditions, thereby improving the accuracy of tasks such as vehicle trajectory prediction and motion analysis.

[0019] A second aspect of this application provides a method for predicting vehicle trajectories based on a roadside bird's-eye view, comprising the following steps: collecting environmental data of the traffic environment; obtaining a two-dimensional image based on the environmental data, and performing a geometric transformation on the two-dimensional image to generate a three-dimensional representation from a bird's-eye view; predicting the vehicle's motion trajectory within a target future time period using a pre-trained deep learning model based on the three-dimensional representation; and generating labeled data of the vehicle's actual motion trajectory to train or update the deep learning model.

[0020] Through the aforementioned technical means, this application embodiment can integrate multi-camera data with advanced bird's-eye view algorithms, utilizing perspective information from multiple roadside cameras to construct a comprehensive bird's-eye view representation of the vehicle's surrounding environment. This integrates perception, motion prediction, and trajectory analysis into a unified framework. Furthermore, by directly analyzing real-time image data, the system can quickly predict vehicle trajectories, thereby optimizing traffic flow management and improving safety. Compared to existing trajectory prediction systems, the roadside bird's-eye view-based vehicle trajectory prediction system can significantly reduce traffic accident risks through comprehensive environmental information capture and enhance adaptability to complex traffic scenarios, making decision-making more accurate and flexible.

[0021] Optionally, in one embodiment of this application, generating a three-dimensional representation of the bird's-eye view includes: converting the RGB image and LiDAR point cloud in the environmental data into a BEV image; generating a spatial context feature map based on the BEV image as the two-dimensional image; extracting at least one global feature from the two-dimensional image based on a pre-built Transformer network; and performing feature stitching and fusion on the at least one global feature to obtain a three-dimensional representation of the bird's-eye view.

[0022] Through the aforementioned technical means, this application embodiment can synthesize two-dimensional images and three-dimensional point clouds from a bird's-eye view using BEV technology to generate two-dimensional images, thereby integrating data from multiple sensors and providing richer environmental information. Furthermore, it utilizes multiple attention heads of the Transformer network to focus on features from different angles to extract global features, thereby enhancing the model's expressive power and its ability to handle complex traffic scenarios. Moreover, it further stitches and fuses these global features, merging features obtained from different Transformer layers to enhance the model's comprehensive feature representation capabilities, forming richer feature representations. This significantly improves the accuracy of vehicle trajectory prediction, reduces the false positive rate, and provides reliable data support for intelligent traffic management.

[0023] Optionally, in one embodiment of this application, the collection of traffic environment data includes: at least one RGB camera and at least one fisheye camera installed on the traffic lights at the intersection to collect the RGB image; and at least one lidar installed on the traffic lights at the intersection to acquire the lidar point cloud.

[0024] Through the above-mentioned technical means, the embodiments of this application can use an RGB camera to acquire color image data and further identify and classify traffic participants; a fisheye camera can be used to provide a wide-angle field of view, thereby covering a larger monitoring area and reducing blind spots; and a lidar laser scanning can be used to acquire high-precision three-dimensional spatial data, providing distance and depth information; thereby realizing comprehensive environmental perception and real-time monitoring and analysis of traffic conditions.

[0025] Optionally, in one embodiment of this application, the step of predicting the vehicle's trajectory within a target future time period using a pre-trained deep learning model based on the three-dimensional representation includes: combining a deep learning algorithm and a physical model, comprehensively considering time and space factors, and using the deep learning model to extract features from the three-dimensional representation data to identify at least one of traffic participants, road information, and traffic light status, thereby generating the trajectory within the target future time period.

[0026] Through the above-mentioned technical means, the embodiments of this application can combine deep learning algorithms and physical models, comprehensively consider time and space factors, and then identify traffic participants, road information, and traffic light status to generate the movement trajectory within the target future time period. This can effectively predict the behavior of traffic participants and take corresponding control measures in a timely manner, reduce the risk of traffic accidents, and improve road safety.

[0027] Optionally, in one embodiment of this application, generating labeled data of the vehicle's actual motion trajectory to train or update the deep learning model includes: extracting the vehicle's motion trajectory from the environmental data, wherein the motion trajectory includes at least one of trajectory data: timestamp and spatial coordinates.

[0028] Through the aforementioned technical means, the embodiments of this application can accurately extract the true value of the spatiotemporal trajectory of vehicle motion, providing high-quality training samples for deep learning models, thereby ensuring that the models can learn the real laws of vehicle motion. At the same time, its multi-dimensional and multi-source adaptability, as well as its ability to support model updates, can further enable the models to better adapt to complex and dynamic real-world traffic scenarios and respond to environmental changes in real time, ensuring that they maintain high efficiency under diverse traffic conditions, thereby improving the accuracy of tasks such as vehicle trajectory prediction and motion analysis.

[0029] A third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the vehicle trajectory prediction method based on roadside bird's-eye view as described in the above embodiments.

[0030] A fourth aspect of this application provides a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the vehicle trajectory prediction method based on a roadside bird's-eye view as described above.

[0031] A fifth aspect of this application provides a computer program product that stores a computer program that, when executed by a processor, implements the vehicle trajectory prediction method based on a roadside bird's-eye view as described above.

[0032] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0033] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0034] Figure 1 This is a schematic diagram of a vehicle trajectory prediction system based on a roadside bird's-eye view, according to an embodiment of this application.

[0035] Figure 2 This is a schematic diagram of a vehicle trajectory prediction system based on a roadside bird's-eye view according to a specific embodiment of this application;

[0036] Figure 3This is a flowchart of a vehicle trajectory prediction method based on a roadside bird's-eye view according to an embodiment of this application;

[0037] Figure 4 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of this application. Detailed Implementation

[0038] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0039] The following description, with reference to the accompanying drawings, describes a vehicle trajectory prediction system and method based on a roadside bird's-eye view, according to embodiments of this application. Addressing the problems mentioned in the background section, where traditional prediction methods are limited by the sufficiency and representativeness of data and the separate processing of prediction, perception, and planning modules, thus failing to effectively cope with complex and dynamic traffic environments and affecting the real-time performance and accuracy of decision-making, this application provides a vehicle trajectory prediction system based on a roadside bird's-eye view. In this system, by integrating multi-camera data and advanced BEV algorithms, a comprehensive BEV representation of the vehicle's surrounding environment is constructed using perspective information from multiple roadside cameras, integrating perception, motion prediction, and trajectory analysis into a unified framework. Furthermore, a deep learning model is used to analyze real-time BEV image data to quickly predict the vehicle's trajectory, thereby optimizing traffic flow management and improving safety. This system significantly reduces the risk of traffic accidents and enhances adaptability to complex traffic scenarios through comprehensive environmental information capture, making decision-making more accurate and flexible. Thus, it solves the problems caused by traditional prediction methods being limited by the sufficiency and representativeness of data and the separate processing of prediction, perception, and planning modules, resulting in an inability to effectively cope with complex and dynamic traffic environments and affecting the real-time performance and accuracy of decision-making.

[0040] Specifically, Figure 1 This is a block diagram illustrating a vehicle trajectory prediction system based on a roadside bird's-eye view, as provided in an embodiment of this application.

[0041] like Figure 1 As shown, the vehicle trajectory prediction system 10 based on roadside bird's-eye view includes: a roadside multi-sensor data acquisition module 100, a roadside bird's-eye view feature generation module 200, a spatiotemporally continuous vehicle trajectory prediction module 300, and a vehicle trajectory ground truth labeling module 400.

[0042] The roadside multi-sensor data acquisition module 100 is used to collect environmental data of the traffic environment. The roadside multi-sensors include, but are not limited to, RGB cameras, fisheye cameras and lidar installed on traffic lights at intersections.

[0043] Optionally, in one embodiment of this application, the roadside multi-sensor data acquisition module 100 includes: at least one RGB camera and at least one fisheye camera installed on the traffic lights at the intersection for acquiring RGB images; and at least one lidar installed on the traffic lights at the intersection for acquiring lidar point clouds.

[0044] Specifically, the embodiments of this application can use RGB cameras, fisheye cameras and lidar installed on traffic lights at intersections to collect data. The hardware configuration includes RGB cameras for acquiring color image data, identifying and classifying traffic participants (such as vehicles, pedestrians, traffic signals, etc.), fisheye cameras for providing a wide field of view, which can cover a larger monitoring area and reduce blind spots, and lidar for acquiring high-precision three-dimensional spatial data through laser scanning, providing distance and depth information.

[0045] Furthermore, in this embodiment, the collected data can be processed using a network protocol (such as IEEE 1588PTP) to ensure consistent data acquisition times from different sensors and to ensure time alignment of multi-source data. During the data processing stage, each data packet can be timestamped to accurately correlate data from different sensors in subsequent analysis.

[0046] The roadside bird's-eye view feature generation module 200 is used to obtain a two-dimensional image based on environmental data and perform geometric transformations on the two-dimensional image to generate a three-dimensional representation of the bird's-eye view.

[0047] In actual execution, the Bird's-Eye View (BEV) feature generation module can perform geometric transformations on the captured two-dimensional image based on the physical position and orientation of the camera to generate a three-dimensional representation of the bird's-eye view.

[0048] Furthermore, the process of generating a three-dimensional representation of the bird's-eye view can specifically include: coordinate system transformation to convert the camera coordinate system into a unified BEV coordinate system; view synthesis: synthesizing the views of multiple cameras through image stitching technology to form a complete BEV image; BEV feature mapping: converting the bird's-eye view image into a grid-like BEV feature map to facilitate subsequent trajectory prediction.

[0049] Optionally, in one embodiment of this application, the roadside bird's-eye view feature generation module 200 includes: a conversion unit for converting RGB images and LiDAR point clouds in environmental data into BEV images; a first generation unit for generating a spatial context feature map based on the BEV image as a two-dimensional image; an extraction unit for extracting at least one global feature from the two-dimensional image based on a pre-built Transformer network; and a second generation unit for performing feature stitching and fusion on the at least one global feature to obtain a three-dimensional representation of the bird's-eye view.

[0050] Among them, BEV image can be understood as an image representation of an object or scene viewed from above. As a top view of the scene from the air, it can intuitively present the position, outline and relative layout of elements such as vehicles, pedestrians and roads (e.g., "the vehicle is in the center of the lane and 5 meters away from the pedestrian in front").

[0051] As one possible approach, embodiments of this application can construct a roadside feature extraction network based on BEV+Transformer. This roadside feature extraction network combines the spatial information represented by BEV with the attention mechanism of Transformer, and has powerful feature extraction capabilities. By using this roadside feature extraction network, key features can be extracted from data acquired by multiple sensors to support vehicle trajectory prediction and environmental understanding.

[0052] Specifically, such as Figure 2As shown, in this embodiment, the RGB image and LiDAR point cloud can first be converted into a BEV representation using the BEV algorithm. Through geometric transformation and mapping techniques, the two-dimensional image and three-dimensional point cloud are synthesized from a bird's-eye view to generate a feature map with spatial context. The BEV feature map can contain multiple channels, such as information about vehicles, pedestrians, roads, and traffic signals. Further, convolutional layers in a CNN can be used for preliminary feature extraction from the BEV feature map. Convolutional layers help extract local features, such as edges, shapes, and textures. Using several convolutional and pooling layers, higher-level features are gradually extracted, and the size of the feature map is reduced. Finally, a Transformer module can be added to extract global features based on the features extracted by the CNN. In terms of features, the Transformer employs a self-attention mechanism, which can capture long-distance dependencies and is suitable for handling complex traffic scenarios. By using multiple attention heads for parallel computation, features can be focused on from different angles, enhancing the model's expressive power. Since the Transformer itself does not possess location information, location encoding needs to be added to the input features to preserve spatial information. Feature fusion is performed between each layer of the Transformer, merging features obtained from different Transformer layers to enhance the model's comprehensive feature representation capability. Finally, weighted summation or concatenation methods can be used to combine features from different layers to form richer feature representations. Techniques such as Dropout and Layer Normalization can be introduced to enhance the model's robustness and prevent overfitting. Residual connections can also be used to preserve original feature information, helping the model better learn complex feature relationships.

[0053] Through the aforementioned technical means, this application embodiment can synthesize two-dimensional images and three-dimensional point clouds from a bird's-eye view using BEV technology to generate two-dimensional images, thereby integrating data from multiple sensors and providing richer environmental information. Furthermore, it utilizes multiple attention heads of the Transformer network to focus on features from different angles to extract global features, thereby enhancing the model's expressive power and its ability to handle complex traffic scenarios. Moreover, it further stitches and fuses these global features, merging features obtained from different Transformer layers to enhance the model's comprehensive feature representation capabilities, forming richer feature representations. This significantly improves the accuracy of vehicle trajectory prediction, reduces the false positive rate, and provides reliable data support for intelligent traffic management.

[0054] The spatiotemporally continuous vehicle trajectory prediction module 300 is used to predict the vehicle's trajectory in a target future time period based on a three-dimensional representation using a pre-trained deep learning model.

[0055] Optionally, in one embodiment of this application, the spatiotemporally continuous vehicle trajectory prediction module 300 is further used to combine deep learning algorithms and physical models, comprehensively consider time and space factors, and use deep learning models to extract features from three-dimensional representation data to identify at least one of traffic participants, road information, and traffic light status, and generate a motion trajectory in the target future time period.

[0056] The aforementioned modules aim to accurately predict vehicle trajectories over future time periods based on real-time collected data. Specifically, this involves multi-source data fusion, receiving input data from multiple roadside sensors (such as RGB cameras, fisheye cameras, and LiDAR), extracting features from the input data using deep learning models (such as convolutional neural networks), identifying key elements such as traffic participants, road information, and traffic light status, and optimizing traffic light control and traffic flow through real-time monitoring and analysis of traffic conditions. This improves the overall efficiency of urban traffic and reduces congestion.

[0057] Specifically, as one possible approach, embodiments of this application can construct a spatiotemporal continuous trajectory prediction network based on Long Short-Term Memory (LSTM) networks to accurately predict vehicle trajectories, especially in complex and dynamic traffic environments; LSTM networks can effectively capture long-term dependencies in time-series data and are suitable for handling trajectory prediction tasks.

[0058] In this embodiment, the collected data can first be formatted into a structure suitable for LSTM input, typically a three-dimensional tensor with a shape of (number of samples, time step, number of features), and the data can be normalized to improve the training efficiency and convergence speed of the model. Further, a multi-layer LSTM structure (i.e., stacking multiple LSTM layers) can be used to enhance the model's ability to learn complex temporal relationships. LSTM units can effectively capture long-term dependencies and prevent gradient vanishing by maintaining unit states and hidden states. Dropout layers are added between LSTM layers to reduce the risk of overfitting and improve the model's generalization ability. A fully connected layer is added after the LSTM layers to map the LSTM output to the prediction result, outputting the trajectory for future time steps. The output layer uses a linear activation function to output the predicted trajectory coordinates, typically (time step, position coordinates).

[0059] Through the above-mentioned technical means, the embodiments of this application can combine deep learning algorithms and physical models, comprehensively consider time and space factors, and then identify traffic participants, road information, and traffic light status to generate movement trajectories in the target future time period. This can effectively predict the behavior of traffic participants and take corresponding control measures in a timely manner, reduce the risk of traffic accidents, and improve road safety.

[0060] The vehicle trajectory ground truth annotation module 400 is used to generate annotated data of the vehicle's real motion trajectory for training or updating deep learning models.

[0061] Optionally, in one embodiment of this application, the vehicle trajectory truth labeling module 400 is further used to extract the vehicle's motion trajectory from environmental data, wherein the motion trajectory includes at least one of the trajectory data: timestamp and spatial coordinates.

[0062] Specifically, the vehicle trajectory ground truth annotation module 400 can extract the vehicle's motion trajectory from sensor data and generate trajectory data containing information such as timestamps and spatial coordinates (such as latitude and longitude, relative coordinates). This module can collect, process and annotate the vehicle's motion trajectory in real traffic environments to provide high-quality ground truth data for model training and evaluation, aiming to provide accurate annotation data for vehicle trajectory prediction.

[0063] Through the aforementioned technical means, the embodiments of this application can accurately extract the true value of the spatiotemporal trajectory of vehicle motion, providing high-quality training samples for deep learning models, thereby ensuring that the models can learn the real laws of vehicle motion. At the same time, its multi-dimensional and multi-source adaptability, as well as its ability to support model updates, can further enable the models to better adapt to complex and dynamic real-world traffic scenarios and respond to environmental changes in real time, ensuring that they maintain high efficiency under diverse traffic conditions, thereby improving the accuracy of tasks such as vehicle trajectory prediction and motion analysis.

[0064] The vehicle trajectory prediction system based on a roadside bird's-eye view proposed in this application integrates multi-camera data with advanced BEV algorithms. Utilizing perspective information from multiple roadside cameras, it constructs a comprehensive BEV representation of the vehicle's surrounding environment, integrating perception, motion prediction, and trajectory analysis into a unified framework. Furthermore, it uses a deep learning model to analyze real-time BEV image data, rapidly predicting the vehicle's trajectory, thereby optimizing traffic flow management and improving safety. This system significantly reduces the risk of traffic accidents through comprehensive environmental information capture and enhances adaptability to complex traffic scenarios, making decision-making more accurate and flexible. Thus, it solves the problems of traditional prediction methods being limited by the sufficiency and representativeness of data and the separate processing of prediction, perception, and planning modules, which leads to an inability to effectively cope with complex and dynamic traffic environments, affecting the real-time performance and accuracy of decision-making.

[0065] Next, refer to the appendix. Figure 3 This application describes a vehicle trajectory prediction method based on a roadside bird's-eye view, according to embodiments of the present application.

[0066] Figure 3 This is a flowchart illustrating the vehicle trajectory prediction method based on a roadside bird's-eye view, according to an embodiment of this application.

[0067] like Figure 3 As shown, the vehicle trajectory prediction method based on roadside bird's-eye view includes the following steps:

[0068] In step S301, environmental data of the traffic environment are collected.

[0069] In step S302, a two-dimensional image is obtained based on environmental data, and a geometric transformation is performed on the two-dimensional image to generate a three-dimensional representation from a bird's-eye view.

[0070] In step S303, based on the three-dimensional representation, a pre-trained deep learning model is used to predict the vehicle's trajectory within the target future time period.

[0071] In step S304, labeled data of the vehicle's actual motion trajectory is generated to train or update the deep learning model.

[0072] Optionally, in one embodiment of this application, generating a three-dimensional representation of a bird's-eye view includes: converting RGB images and LiDAR point clouds in environmental data into BEV images; generating a spatial context feature map from the BEV images as a two-dimensional image; extracting at least one global feature from the two-dimensional image based on a pre-built Transformer network; and performing feature stitching and fusion on the at least one global feature to obtain a three-dimensional representation of a bird's-eye view.

[0073] Optionally, in one embodiment of this application, collecting environmental data of the traffic environment includes: at least one RGB camera and at least one fisheye camera installed on the traffic lights at the intersection to collect RGB images; and at least one lidar installed on the traffic lights at the intersection to acquire lidar point clouds.

[0074] Optionally, in one embodiment of this application, predicting the vehicle's trajectory within a target future time period using a pre-trained deep learning model based on the three-dimensional representation includes: combining a deep learning algorithm and a physical model, comprehensively considering time and space factors, using the deep learning model to extract features from the three-dimensional representation data to identify at least one of traffic participants, road information, and traffic light status, and generating the trajectory within the target future time period.

[0075] Optionally, in one embodiment of this application, generating labeled data of the vehicle's actual motion trajectory to train or update a deep learning model includes: extracting the vehicle's motion trajectory from environmental data, wherein the motion trajectory includes at least one of trajectory data: timestamp and spatial coordinates.

[0076] It should be noted that the foregoing explanation of the vehicle trajectory prediction system embodiment based on roadside bird's-eye view also applies to the vehicle trajectory prediction method based on roadside bird's-eye view in this embodiment, and will not be repeated here.

[0077] The vehicle trajectory prediction method based on a roadside bird's-eye view proposed in this application integrates multi-camera data with advanced BEV algorithms. Utilizing perspective information from multiple roadside cameras, it constructs a comprehensive BEV representation of the vehicle's surrounding environment, integrating perception, motion prediction, and trajectory analysis into a unified framework. Furthermore, it uses a deep learning model to analyze real-time BEV image data, rapidly predicting the vehicle's trajectory, thereby optimizing traffic flow management and improving safety. This method significantly reduces traffic accident risks through comprehensive environmental information capture and enhances adaptability to complex traffic scenarios, making decision-making more accurate and flexible. Thus, it solves the problems in related technologies where traditional prediction methods are limited by the sufficiency and representativeness of data and the separate processing of prediction, perception, and planning modules, leading to an inability to effectively cope with complex and dynamic traffic environments and affecting the real-time performance and accuracy of decision-making.

[0078] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include:

[0079] The memory 401, the processor 402, and the computer program stored on the memory 401 and capable of running on the processor 402.

[0080] When the processor 402 executes the program, it implements the vehicle trajectory prediction method based on the roadside bird's-eye view provided in the above embodiments.

[0081] Furthermore, electronic devices also include:

[0082] Communication interface 403 is used for communication between memory 401 and processor 402.

[0083] The memory 401 is used to store computer programs that can run on the processor 402.

[0084] The memory 401 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0085] If the memory 401, processor 402, and communication interface 403 are implemented independently, then the communication interface 403, memory 401, and processor 402 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 4 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0086] Optionally, in a specific implementation, if the memory 401, processor 402, and communication interface 403 are integrated on a single chip, then the memory 401, processor 402, and communication interface 403 can communicate with each other through an internal interface.

[0087] Processor 402 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0088] This application also provides a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described vehicle trajectory prediction method based on a roadside bird's-eye view.

[0089] This application also provides a computer program product storing a computer program that, when executed by a processor, implements the above-described vehicle trajectory prediction method based on a roadside bird's-eye view.

[0090] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0091] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0092] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0093] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0094] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, it can be implemented using any one or more of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0095] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it includes one or a combination of the steps of the method embodiments.

[0096] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0097] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A vehicle trajectory prediction system based on a roadside bird's-eye view, characterized in that, include: The roadside multi-sensor data acquisition module is used to collect environmental data related to the traffic environment. The roadside bird's-eye view feature generation module is used to obtain a two-dimensional image based on the environmental data and perform geometric transformations on the two-dimensional image to generate a three-dimensional representation of the bird's-eye view. A spatiotemporally continuous vehicle trajectory prediction module is used to predict the vehicle's trajectory in a target future time period based on the three-dimensional representation using a pre-trained deep learning model. The vehicle trajectory ground truth annotation module is used to generate annotated data of the vehicle's actual motion trajectory to train or update the deep learning model.

2. The system according to claim 1, characterized in that, The roadside bird's-eye view feature generation module includes: The conversion unit is used to convert the RGB image and lidar point cloud in the environmental data into a BEV image; A generation unit is configured to generate a spatial context feature map based on the BEV image, which serves as the two-dimensional image. An extraction unit is used to extract at least one global feature from the two-dimensional image based on a pre-built Transformer network; A generation unit is used to perform feature splicing and fusion on the at least one global feature to obtain a three-dimensional representation of the bird's-eye view.

3. The system according to claim 2, characterized in that, The roadside multi-sensor data acquisition module includes: At least one RGB camera and at least one fisheye camera are installed on the traffic lights at the intersection for acquiring the RGB images; At least one lidar is installed on the traffic lights at the intersection to acquire lidar point clouds.

4. The system according to claim 1, characterized in that, The spatiotemporally continuous vehicle trajectory prediction module is further used to combine deep learning algorithms and physical models, comprehensively consider time and space factors, and use the deep learning model to extract features from the three-dimensional representation data to identify at least one of traffic participants, road information, and traffic light status, and generate the motion trajectory in the target future time period.

5. The system according to claim 1, characterized in that, The vehicle trajectory truth labeling module is further used to extract the vehicle's motion trajectory from the environmental data, wherein the motion trajectory includes at least one of the trajectory data: timestamp and spatial coordinates.

6. A vehicle trajectory prediction method based on a roadside bird's-eye view, characterized in that, Includes the following steps: Collect environmental data on the traffic environment; A two-dimensional image is obtained based on the environmental data, and a geometric transformation is performed on the two-dimensional image to generate a three-dimensional representation from a bird's-eye view. Based on the three-dimensional representation, a pre-trained deep learning model is used to predict the vehicle's trajectory within the target future time period; Annotated data of the vehicle's actual motion trajectory is generated to train or update the deep learning model.

7. The method according to claim 6, characterized in that, The generated three-dimensional representation of the bird's-eye view includes: Convert the RGB image and LiDAR point cloud in the environmental data into a BEV image; A spatial context feature map is generated from the BEV image, which serves as the two-dimensional image. Based on a pre-built Transformer network, at least one global feature is extracted from the two-dimensional image; The at least one global feature is spliced ​​and fused to obtain a three-dimensional representation of the bird's-eye view.

8. An electronic device, characterized in that, include: The memory, the processor, and the computer program stored in the memory and executable on the processor, the processor executing the program to implement the vehicle trajectory prediction method based on roadside bird's-eye view as described in any one of claims 6-7.

9. A non-volatile computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the vehicle trajectory prediction method based on roadside bird's-eye view as described in any one of claims 6-7.

10. A computer program product, comprising a computer program, characterized in that, The computer program is executed to implement the vehicle trajectory prediction method based on roadside bird's-eye view as described in any one of claims 6-7.

Images