Track prediction method and device, storage medium and program product
By generating and filtering predicted environment images of vehicles, the problem of low reliability in vehicle trajectory prediction is solved, achieving higher accuracy in trajectory prediction and improving the safety and performance of autonomous driving.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- XIAOMI EV TECH CO LTD
- Filing Date
- 2024-10-29
- Publication Date
- 2026-05-01
AI Technical Summary
In existing technologies, the reliability of vehicle trajectory prediction is low, especially when obstacle interaction behavior is difficult to predict.
By acquiring the current environmental image and location information of the target vehicle, multiple candidate trajectories are generated using a pre-trained trajectory prediction model. The target trajectory with higher reliability is then selected by generating a predicted environment image. This includes using a trajectory prediction model with variable temperature parameters and a predicted environment generation model, combined with collision prediction and passenger experience evaluation.
This improves the accuracy and reliability of trajectory prediction results, thereby enhancing the performance and safety of autonomous driving technology.
Smart Images
Figure CN121947542A_ABST
Abstract
Description
Trajectory prediction methods, devices, storage media, and software products Technical Field
[0001] This disclosure relates to the field of vehicles, and more particularly to a trajectory prediction method, apparatus, storage medium, and program product. Background Technology
[0002] Vehicle trajectory prediction is a crucial part of autonomous driving and a prerequisite for achieving autonomous driving planning and decision-making. Therefore, accurately predicting vehicle trajectories is a significant challenge in the transportation field.
[0003] Currently, most methods involve finding scenarios similar to the current scenario from a data pool and predicting trajectories using foreground obstacles and road structures (such as lane lines) labeled in similar scenarios. However, the interaction behavior between obstacles is often difficult to predict, which leads to low reliability of the predicted trajectory. Summary of the Invention
[0004] To overcome the problems existing in related technologies, this disclosure provides a trajectory prediction method, apparatus, storage medium, and program product.
[0005] According to a first aspect of the present disclosure, a trajectory prediction method is provided, comprising: acquiring a current environmental image and current location information of a target vehicle; obtaining at least one candidate trajectory based on the current environmental image and the current location information; determining a predicted environmental image for each candidate trajectory; and determining a target trajectory from the candidate trajectories based on the predicted environmental image.
[0006] Optionally, obtaining at least one candidate trajectory based on the current environment image and the current location information includes: inputting the current environment image and the current location information into a pre-trained trajectory prediction model with variable temperature parameters to obtain multiple reference trajectories, wherein different temperature parameters correspond to different reference trajectories; for each reference trajectory, determining a driving prediction result for the reference trajectory, the driving prediction result including a collision prediction result and / or a ride experience prediction result; and determining the candidate trajectory from the multiple reference trajectories based on the driving prediction result.
[0007] Optionally, determining the candidate trajectory from multiple reference trajectories based on the driving prediction result includes: if the collision prediction result of the reference trajectory indicates that the target vehicle will not collide when driving along the reference trajectory, and / or the ride experience prediction result indicates that the speed change of the target vehicle during the driving process along the reference trajectory is less than a speed change threshold, the maximum acceleration is less than an acceleration threshold, and the acceleration change in adjacent time periods is less than an acceleration change threshold, then the reference trajectory is determined as the candidate trajectory.
[0008] Optionally, the trajectory prediction model is trained as follows: multiple first training sample data of the target vehicle are acquired, each first training sample data includes a first sample historical image, first sample historical location information, and a first sample trajectory, wherein the first sample historical image and the first sample historical location information are used as input data of the trajectory prediction model, the first sample trajectory is the target output data of the trajectory prediction model, and the first sample trajectory includes multiple grid markers, the grid markers being used to indicate the positions of discrete points constituting the trajectory; the trajectory prediction model is then trained.
[0009] Optionally, the method further includes: acquiring multiple test sample data of the target vehicle, each test sample data including a second sample historical image, a second sample historical position, and a second sample future image; inputting the second sample historical image and the second sample historical position into the pre-trained trajectory prediction model with variable temperature parameters to obtain multiple second sample predicted trajectories; determining the predicted image of each second sample predicted trajectory; and evaluating the trajectory prediction model based on the predicted image of each second sample predicted trajectory and the corresponding second sample future image.
[0010] Optionally, determining the predicted environment image for each candidate trajectory includes: for each candidate trajectory, inputting the candidate trajectory and the current environment image into a pre-trained predicted environment generation model to obtain the predicted environment image.
[0011] Optionally, the predictive environment generation model is trained in the following manner: multiple second training sample data of the target vehicle are acquired, each second training sample data includes a first sample historical image, a first sample future image and a first sample trajectory, the first sample historical image and the first sample trajectory are the input data of the predictive environment generation model, and the first sample future image is the target output data of the predictive environment generation model; the predictive environment generation model is trained.
[0012] Optionally, determining the target trajectory from the candidate trajectories based on the predicted environment image includes: if the predicted environment image represents the target position inside the target vehicle from a perspective that does not deviate from the road centerline and has a clear field of vision, then the candidate trajectory is determined as the target trajectory.
[0013] According to a second aspect of the present disclosure, a trajectory prediction device is provided, comprising: an acquisition module for acquiring a current environmental image and current location information of a target vehicle; a first determination module for obtaining at least one candidate trajectory based on the current environmental image and the current location information; a second determination module for determining a predicted environmental image for each candidate trajectory; and a third determination module for determining a target trajectory from the candidate trajectories based on the predicted environmental image.
[0014] According to a third aspect of the present disclosure, a trajectory prediction apparatus is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the computer program in the memory to implement the steps of the trajectory prediction method provided in the first aspect of the present disclosure.
[0015] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided that stores computer program instructions thereon, which, when executed by a processor, implement the steps of the trajectory prediction method provided in the first aspect of the present disclosure.
[0016] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the trajectory prediction method provided in the first aspect of the present disclosure.
[0017] The technical solutions provided by the embodiments of this disclosure can include the following beneficial effects: During the driving process of the target vehicle, firstly, at least one candidate trajectory is obtained based on the acquired current environmental image and current location information of the target vehicle. Then, a predicted environmental image for each candidate trajectory is determined. Finally, based on the predicted environmental image of each candidate trajectory, the target trajectory is determined from the candidate trajectories. In this way, by generating a predicted environmental image for each candidate trajectory, a deeper understanding of the road and environmental conditions the vehicle may face in the future can be achieved. By using the predicted environmental image as a reference to compare and analyze multiple candidate trajectories, a more reliable trajectory can be selected as the target trajectory, thus improving the accuracy of the trajectory prediction results.
[0018] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description
[0019] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.
[0020] Figure 1 is a flowchart illustrating a trajectory prediction method according to an exemplary embodiment.
[0021] Figure 2 is a schematic diagram illustrating a trajectory discrete point quantization according to an exemplary embodiment.
[0022] Figure 3 is a schematic diagram illustrating a collision calculation according to an exemplary embodiment.
[0023] Figure 4 is a schematic diagram illustrating a trajectory prediction method according to an exemplary embodiment.
[0024] Figure 5 is a block diagram illustrating a trajectory prediction device according to an exemplary embodiment.
[0025] Figure 6 is a block diagram illustrating a trajectory prediction device according to an exemplary embodiment.
[0026] Figure 7 is a block diagram of a chip system according to an exemplary embodiment. Detailed Implementation
[0027] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.
[0028] It should be noted that all actions involving the acquisition of signals, information, or data in this disclosure are carried out in compliance with the relevant data protection laws and policies of the country where the location is situated, and with authorization from the owner of the relevant device.
[0029] Figure 1 is a flowchart illustrating a trajectory prediction method according to an exemplary embodiment. This method can be applied to a vehicle terminal. As shown in Figure 1, the trajectory prediction method includes steps S101 to S104.
[0030] In step S101, the current environmental image and current location information of the target vehicle are acquired.
[0031] For example, the current environment image can be acquired based on an image acquisition device (such as a camera) pre-installed on the target vehicle. The current location information of the target vehicle can be determined based on the Global Navigation Satellite System, or it can be determined by combining signals collected by multiple sensors such as inertial measurement units, image acquisition devices, and radar pre-installed on the vehicle.
[0032] In step S102, at least one candidate trajectory is obtained based on the current environment image and the current location information.
[0033] For example, a scene with a similarity greater than a preset similarity to the current environment image and current location information can be identified from a pre-set historical database, or a preset number of scenes with the highest similarity to the current environment image and current location information can be identified; the trajectory corresponding to the identified scene can be identified as a candidate trajectory.
[0034] In step S103, a predicted environment image for each candidate trajectory is determined.
[0035] For example, a correspondence between a trajectory and a predicted environment image can be constructed based on trajectories in a historical database and video footage of the target vehicle traveling along those trajectories. The predicted environment image for a candidate trajectory can include multiple consecutive frames. Based on this correspondence, a predicted environment image corresponding to the current candidate trajectory can be determined. Generating a predicted environment image for each candidate trajectory allows the system to gain a deeper understanding of the road and environmental conditions the vehicle may encounter in the future, enabling the feasibility of the candidate trajectory to be evaluated based on the predicted environment image.
[0036] In step S104, the target trajectory is determined from the candidate trajectories based on the predicted environment image.
[0037] For example, based on a predicted environment image, it can be determined whether there is a risk of lane departure if the candidate trajectory corresponding to the predicted environment image is followed. If there is no risk of lane departure, the candidate trajectory can be designated as the target trajectory. For instance, the risk of lane departure can be determined using methods such as Time to Line Crossing (TLC), Future Offset Difference (FOD), and the angle between lane lines; these will not be elaborated upon here. Alternatively, the feasibility of the target vehicle following the candidate trajectory corresponding to the predicted environment image can be determined using other methods, and the feasible candidate trajectory can be designated as the target trajectory. Thus, using the predicted environment image as a reference allows for a more accurate assessment of the feasibility of each candidate trajectory. Selecting the more reliable trajectory from the candidate trajectories as the target trajectory improves the accuracy of trajectory prediction results, thereby further enhancing the performance and safety standards of autonomous driving technology.
[0038] In the above technical solution, firstly, at least one candidate trajectory is obtained based on the acquired current environmental image and current location information of the target vehicle. Next, a predicted environmental image is determined for each candidate trajectory. Then, based on the predicted environmental image of each candidate trajectory, the target trajectory is determined from the candidate trajectories. In this way, by generating a predicted environmental image for each candidate trajectory, a deeper understanding of the road and environmental conditions the vehicle may face in the future can be achieved. By using the predicted environmental image as a reference to compare and analyze multiple candidate trajectories, a more reliable trajectory can be selected as the target trajectory, improving the accuracy of the trajectory prediction results.
[0039] In an optional embodiment, in step S102, at least one candidate trajectory is obtained based on the current environment image and the current location information, including: inputting the current environment image and the current location information into a pre-trained trajectory prediction model with variable temperature parameters to obtain multiple reference trajectories, wherein different temperature parameters correspond to different reference trajectories; and determining each reference trajectory as a candidate trajectory.
[0040] For example, the trajectory prediction model can be a VLA (Vision Language Action) model. After inputting the current environment image into the trajectory prediction model, corresponding image feature information can be obtained. Then, using Q-Former (QueryingTransformer, a lightweight Transformer architecture), the image feature information can be converted into language feature information. This language feature information is then concatenated with the feature information obtained based on the current location information to obtain concatenated feature information. The location information of the discrete points constituting the trajectory can then be obtained based on the concatenated feature information. During the operation of the trajectory prediction model, the probability of predicted words in the softmax output layer can be adjusted by controlling the temperature parameter, thereby controlling the randomness and creativity of the generated reference trajectories. That is, multiple reference trajectories can be obtained by changing the temperature parameter of the trajectory prediction model.
[0041] In one embodiment, the trajectory prediction model can be trained as follows: acquiring multiple first training sample data of the target vehicle, each first training sample data including a first sample historical image, first sample historical location information and a first sample trajectory, wherein the first sample historical image and the first sample historical location information are used as input data of the trajectory prediction model, the first sample trajectory is the target output data of the trajectory prediction model, and the first sample trajectory includes multiple grid markers, the grid markers are used to indicate the positions of discrete points constituting the trajectory; and training the trajectory prediction model.
[0042] For example, typically, the positional information of discrete points constituting a trajectory, obtained from splicing feature information, is composed of floating-point numbers. These floating-point numbers lack logical relationships, and their precision affects the accuracy of token encoding. To address this issue, these floating-point numbers can be discretized and tokenized, and the trajectory coordinates in the vehicle coordinate system can be converted into regular bin indices. The resulting tokens are then used as network identifiers to make the trajectory point sequence more reliable. Therefore, to make the trajectory point sequence output by the trajectory prediction model more reliable, a trajectory point sequence composed of multiple grid identifiers can be selected as the first sample trajectory. During training, the bin-indexed tokens can be added to the vocabulary of the trajectory prediction model as trajectory tokens. For example, the grid identifier token_id can be obtained based on the following formula: token_id = round({(x,y)} / bin_size), where bin_size is the grid size in the bird's-eye view, and {(x,y)} are the trajectory coordinates in the vehicle coordinate system. A schematic diagram of this conversion is shown in Figure 2.
[0043] In an optional embodiment, the trajectory prediction model can be tested as follows: multiple test sample data of the target vehicle are acquired, each test sample data including a second sample historical image, a second sample historical position, and a second sample future image; the second sample historical image and the second sample historical position are input into a pre-trained trajectory prediction model with variable temperature parameters to obtain multiple second sample predicted trajectories; a predicted image for each second sample predicted trajectory is determined; and the trajectory prediction model is evaluated based on the predicted image of each second sample predicted trajectory and the corresponding second sample future image.
[0044] For example, the second sample historical image and the second sample future image can come from the same video. The trajectory prediction model can be evaluated based on the predicted image of each second sample predicted trajectory and the corresponding second sample future image in the following way: determine the similarity between the predicted image of each second sample predicted trajectory and the corresponding second sample future image; if the ratio of the number of test sample data with similarity greater than a similarity threshold to the number of test sample data reaches a proportion threshold, then the trajectory prediction model is deemed to have passed the evaluation. The similarity threshold and proportion threshold can be preset based on actual needs. For example, the predicted image of each second sample predicted trajectory can be determined as follows: for each second sample predicted trajectory, the second sample predicted trajectory and the second sample historical image are input into a pre-trained prediction environment generation model to obtain the predicted image of the second sample predicted trajectory.
[0045] The test sample data can include image and location information from scenarios where autonomous driving fails. The reliability of the second-sample predicted trajectory can be verified by using the predicted image of the second-sample predicted trajectory, i.e., verifying whether the trajectory prediction model passes the evaluation. This makes a large amount of data from autonomous driving failure scenarios available, effectively improving data utilization.
[0046] In an optional embodiment, in step S102, obtaining at least one candidate trajectory based on the current environmental image and current location information includes: inputting the current environmental image and current location information into a pre-trained trajectory prediction model with variable temperature parameters to obtain multiple reference trajectories, wherein different temperature parameters correspond to different reference trajectories; for each reference trajectory, determining the driving prediction result of the reference trajectory, the driving prediction result including collision prediction result and / or, ride experience prediction result; and determining the candidate trajectory from the multiple reference trajectories based on the driving prediction result.
[0047] The specific implementation of the trajectory prediction model has been described in detail above, and will not be repeated here.
[0048] For example, using techniques from related technologies to determine whether a vehicle will collide, a collision prediction result can be obtained by determining whether the target vehicle will collide with other vehicles or road edge lines when traveling along a reference trajectory. The other vehicles or road edge lines can be obtained based on an offline object detection / lane detection model. As shown in Figure 3, if the collision prediction result of the reference trajectory indicates that the target vehicle will collide when traveling along the reference trajectory, then the safety of the reference trajectory is deemed insufficient. To avoid collisions in actual operation, the reference trajectory can be deleted, i.e., it will not be identified as a candidate trajectory.
[0049] For example, the predicted riding experience can be determined based on the speed and acceleration of the target vehicle traveling along the reference trajectory. Specifically, the smoothness of the target vehicle's travel along the reference trajectory can be determined based on speed, and the comfort can be determined based on acceleration. For instance, if the speed change of the target vehicle during its travel along the reference trajectory exceeds a speed change threshold, the smoothness of the corresponding reference trajectory can be determined to be insufficient; if the maximum acceleration of the target vehicle during its travel along the reference trajectory exceeds an acceleration threshold, or if the acceleration change in adjacent time periods exceeds an acceleration change threshold, the comfort of the corresponding reference trajectory can be determined to be insufficient. The speed change threshold, acceleration threshold, and acceleration change threshold can be preset based on actual needs. To ensure the user's riding experience, reference trajectories that do not meet the smoothness or comfort requirements can be deleted, i.e., they are not identified as candidate trajectories.
[0050] In one embodiment, a candidate trajectory can be determined from multiple reference trajectories based on the driving prediction results in the following manner: if the collision prediction result of the reference trajectory indicates that the target vehicle will not collide when driving along the reference trajectory, and / or, the ride experience prediction result indicates that the speed change of the target vehicle during the driving process along the reference trajectory is less than the speed change threshold, the maximum acceleration is less than the acceleration threshold, and the acceleration change in adjacent time periods is less than the acceleration change threshold, then the reference trajectory is determined as a candidate trajectory.
[0051] For example, the speed variance during the process of traveling along a reference trajectory can be determined based on the speed at each time interval, and the variance characterizes the speed change. If the collision prediction result of the reference trajectory indicates that the target vehicle will not collide while traveling along the reference trajectory, then the reference trajectory can be determined to have sufficient safety. If the ride experience prediction result indicates that the speed change of the target vehicle during the process of traveling along the reference trajectory is less than a speed change threshold, the maximum acceleration is less than an acceleration threshold, and the acceleration change in adjacent time intervals is less than an acceleration change threshold, then the reference trajectory can be determined to have sufficient smoothness and comfort, providing a good ride experience for the user. Thus, by filtering the reference trajectory based on the collision prediction result and / or the ride experience prediction result, some unreasonable trajectories can be deleted, reducing the amount of data processing in subsequent processes and improving the efficiency of trajectory prediction.
[0052] In an optional embodiment, in step S103, determining the predicted environment image for each candidate trajectory includes: for each candidate trajectory, inputting the candidate trajectory and the current environment image into a pre-trained predicted environment generation model to obtain the predicted environment image.
[0053] Among them, the environment generation model can be the DiT (Diffusion Transformer) model. The DiT model is a diffusion model that combines the Transformer architecture and can be used to complete image and video generation tasks.
[0054] In one embodiment, the predictive environment generation model can be trained as follows: multiple second training sample data of the target vehicle are acquired, each second training sample data includes a first sample historical image, a first sample future image and a first sample trajectory, the first sample historical image and the first sample trajectory are the input data of the predictive environment generation model, and the first sample future image is the target output data of the predictive environment generation model; the predictive environment generation model is trained.
[0055] The first sample historical image and the first sample future image can both come from the same video. The first sample historical image appears in the video before the first sample future image appears in the video.
[0056] In one embodiment, after obtaining the predicted environment image, the trajectory prediction method provided in this disclosure may further include: fusing target information into the predicted environment image, wherein the target information may be at least one of 3D bounding box, 2D bounding box, depth map, and semantic map information; and outputting the fused predicted environment image.
[0057] For example, information can be fused through layer overlay. This can enhance the visual appeal of an image and provide useful auxiliary information for subsequent image analysis, understanding, and processing.
[0058] In an optional embodiment, in step S104, determining the target trajectory from the candidate trajectories based on the predicted environment image includes: if the viewpoint representing the target position inside the target vehicle in the predicted environment image does not deviate from the road centerline and the field of view is clear, then the candidate trajectory is determined as the target trajectory.
[0059] For example, the target location can be the driver's position. If the predicted environmental image has a clear field of view, and the viewing angle of the target position inside the target vehicle does not deviate from the road centerline, then the candidate trajectory corresponding to the predicted environmental image can be determined to be highly reasonable and can be identified as the target trajectory. If the predicted environmental image has an unclear field of view, such as the presence of objects like tree branches that, while not posing a collision risk, would partially obstruct the view, then for driving safety, the candidate trajectory corresponding to the predicted environmental image can be determined to be insufficiently reasonable and will not be identified as the target trajectory. If the predicted environmental image has a clear field of view, but the viewing angle of the target position inside the target vehicle deviates from the road centerline, then for driving safety, the candidate trajectory corresponding to the predicted environmental image can be determined to be insufficiently reasonable and will not be identified as the target trajectory.
[0060] Thus, based on the predicted environment image, the feasibility of each candidate trajectory can be evaluated more accurately, improving the accuracy of trajectory prediction results, which can further enhance the performance and safety standards of autonomous driving technology.
[0061] Figure 4 is a schematic diagram illustrating a trajectory prediction method according to an exemplary embodiment. As shown in Figure 4, the current environment image and current location information can be input into the trajectory prediction model to obtain multiple reference trajectories. Then, a filtering unit filters the reference trajectories, which determines candidate trajectories from the reference trajectories based on the collision prediction results and ride experience prediction results of the reference trajectories. Then, a prediction environment image for each candidate trajectory can be obtained through a prediction environment generation model. Afterward, the prediction environment image can be input into a judgment unit, which judges the rationality of the candidate trajectory corresponding to the prediction environment image and determines the candidate trajectory with higher rationality as the target trajectory.
[0062] Thus, by filtering the reference trajectory to obtain candidate trajectories, some unreasonable trajectories can be deleted, reducing the amount of data processing in subsequent processes and improving trajectory prediction efficiency. By generating a prediction environment image for each candidate trajectory, a deeper understanding of the road and environmental conditions that the vehicle may encounter in the future can be achieved. By using the prediction environment image as a reference to compare and analyze multiple candidate trajectories, a more reliable trajectory can be selected as the target trajectory, improving the reliability and accuracy of the trajectory prediction results.
[0063] Figure 5 is a block diagram illustrating a trajectory prediction device 500 according to an exemplary embodiment. Referring to Figure 5, the device 500 includes an acquisition module 501, a first determination module 502, a second determination module 503, and a fourth determination module 504.
[0064] The acquisition module 501 is used to acquire the current environment image and current location information of the target vehicle; the first determination module 502 is used to obtain at least one candidate trajectory based on the current environment image and the current location information; the second determination module 503 is used to determine the predicted environment image of each candidate trajectory; and the third determination module 504 is used to determine the target trajectory from the candidate trajectories based on the predicted environment image.
[0065] In this way, by generating a predicted environment image for each candidate trajectory, we can gain a deeper understanding of the road and environmental conditions that the vehicle may face in the future. By using the predicted environment image as a reference to compare and analyze multiple candidate trajectories, we can select the more reliable trajectory as the target trajectory and improve the accuracy of trajectory prediction results.
[0066] Optionally, the first determining module 502 includes: a first determining sub-model, used to input the current environment image and the current location information into a pre-trained trajectory prediction model with variable temperature parameters to obtain multiple reference trajectories, wherein different temperature parameters correspond to different reference trajectories; a second determining sub-model, used to determine the driving prediction result of each reference trajectory, the driving prediction result including collision prediction result and / or, ride experience prediction result; and a third determining sub-model, used to determine the candidate trajectory from the multiple reference trajectories based on the driving prediction result.
[0067] Optionally, the third determining sub-model is used to determine the candidate trajectory in the following manner: if the collision prediction result of the reference trajectory indicates that the target vehicle will not collide when traveling along the reference trajectory, and / or, the ride experience prediction result indicates that the speed change of the target vehicle during the travel of the reference trajectory is less than the speed change threshold, the maximum acceleration is less than the acceleration threshold, and the acceleration change in adjacent time periods is less than the acceleration change threshold, then the reference trajectory is determined as the candidate trajectory.
[0068] Optionally, the trajectory prediction model is trained as follows: multiple first training sample data of the target vehicle are acquired, each first training sample data includes a first sample historical image, first sample historical location information, and a first sample trajectory, wherein the first sample historical image and the first sample historical location information are used as input data of the trajectory prediction model, the first sample trajectory is the target output data of the trajectory prediction model, and the first sample trajectory includes multiple grid markers, the grid markers being used to indicate the positions of discrete points constituting the trajectory; the trajectory prediction model is then trained.
[0069] Optionally, the trajectory prediction device 500 may further include: a model evaluation module, configured to acquire multiple test sample data of the target vehicle, each test sample data including a second sample historical image, a second sample historical position, and a second sample future image; input the second sample historical image and the second sample historical position into the pre-trained trajectory prediction model with variable temperature parameters to obtain multiple second sample predicted trajectories; determine the predicted image of each second sample predicted trajectory; and evaluate the trajectory prediction model based on the predicted image of each second sample predicted trajectory and the corresponding second sample future image.
[0070] Optionally, the second determining module 503 is used to determine the predicted environment image of each candidate trajectory by inputting the candidate trajectory and the current environment image into a pre-trained predicted environment generation model for each candidate trajectory to obtain the predicted environment image.
[0071] Optionally, the predictive environment generation model is trained in the following manner: multiple second training sample data of the target vehicle are acquired, each second training sample data includes a first sample historical image, a first sample future image and a first sample trajectory, the first sample historical image and the first sample trajectory are the input data of the predictive environment generation model, and the first sample future image is the target output data of the predictive environment generation model; the predictive environment generation model is trained.
[0072] Optionally, the third determining module 504 is used to determine the target trajectory from the candidate trajectories in the following manner: if the view of the predicted environment image representing the target position inside the target vehicle does not deviate from the road centerline and the field of vision is clear, then the candidate trajectory is determined as the target trajectory.
[0073] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.
[0074] This disclosure also provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the steps of the trajectory prediction method provided in this disclosure.
[0075] Figure 6 is a block diagram illustrating a trajectory prediction device 800 according to an exemplary embodiment. For example, the device 800 may be a vehicle controller, etc.
[0076] Referring to FIG6, the device 800 may include one or more of the following components: processing component 802, memory 804, power supply component 806, multimedia component 808, audio component 810, input / output interface 812, sensor component 814, and communication component 816.
[0077] Processing component 802 typically controls the overall operation of device 800, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the trajectory prediction method described above. Furthermore, processing component 802 may include one or more modules to facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802.
[0078] Memory 804 is configured to store various types of data to support the operation of device 800. Examples of this data include instructions for any application or method operating on device 800, contact data, phonebook data, messages, pictures, videos, etc. Memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.
[0079] Power supply component 806 provides power to various components of device 800. Power supply component 806 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to device 800.
[0080] Multimedia component 808 includes a screen that provides an output interface between the device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 808 includes a front-facing camera and / or a rear-facing camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.
[0081] Audio component 810 is configured to output and / or input audio signals. For example, audio component 810 includes a microphone (MIC) configured to receive external audio signals when device 800 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 804 or transmitted via communication component 816. In some embodiments, audio component 810 also includes a speaker for outputting audio signals.
[0082] Input / output interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.
[0083] Sensor assembly 814 includes one or more sensors for providing status assessments of various aspects of device 800. For example, sensor assembly 814 may detect the on / off state of device 800, the relative positioning of components such as the display and keypad of device 800, changes in the position of device 800 or a component of device 800, the presence or absence of user contact with device 800, the orientation or acceleration / deceleration of device 800, and temperature changes of device 800. Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 814 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.
[0084] Communication component 816 is configured to facilitate wired or wireless communication between device 800 and other devices. Device 800 can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 816 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 816 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.
[0085] In an exemplary embodiment, the device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the trajectory prediction method described above.
[0086] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, which can be executed by a processor 820 of the device 800 to complete the trajectory prediction method described above. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.
[0087] In another exemplary embodiment, a computer program product is also provided, the computer program product comprising a computer program executable by a programmable device, the computer program having a code portion for performing the trajectory prediction method described above when executed by the programmable device.
[0088] Some embodiments of this disclosure also provide a chip system, as shown in FIG7, which includes at least one processor 1301 and at least one interface circuit 1302. The processor 1301 and the interface circuit 1302 are interconnected via lines. For example, the interface circuit 1302 can be used to receive signals from other devices (e.g., the memory of an electronic device). As another example, the interface circuit 1302 can be used to send signals to other devices (e.g., the processor 1301). Exemplarily, the interface circuit 1302 can read instructions stored in memory and send those instructions to the processor 1301. When the instructions are executed by the processor 1301, the trajectory prediction device can perform the steps in the above embodiments. Of course, the chip system may also include other discrete devices, and some embodiments of this disclosure do not specifically limit this.
[0089] In some embodiments of this disclosure, the interface circuit 1302 can acquire data, program instructions, and / or information from the internal storage area of the chip system; it can also acquire data, program instructions, and / or information from outside the chip system.
[0090] Optionally, the chip system also includes a memory 1303 for storing necessary computer programs and data.
[0091] Those skilled in the art will also understand that the various illustrative logical blocks and steps listed in the embodiments of this application can be implemented by electronic hardware, computer software, or a combination of both. Whether such functionality is implemented through hardware or software depends on the specific application and the overall system design requirements. Those skilled in the art can implement the described functionality using various methods for each specific application, but such implementation should not be construed as exceeding the scope of protection of the embodiments of this application.
[0092] It should be understood that, unless otherwise specifically indicated, features of various embodiments of this disclosure described herein can be combined with each other. As used herein, the term “and / or” includes any one of the relevant listed items and any combination of any two or more; similarly, “at least one of…” includes any one of the relevant listed items and any combination of any two or more.
[0093] Although terms such as “first,” “second,” and “third” may be used herein to describe various components, parts, regions, layers, or sections, these components, parts, regions, layers, or sections are not limited to these terms. Rather, these terms are used only to distinguish one component, part, region, layer, or section from another. Therefore, without departing from the teachings of the examples described herein, the first component, part, region, layer, or section mentioned in the examples may also be referred to as the second component, part, region, layer, or section. Furthermore, the terms “first” and “second” are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as “first” or “second” may explicitly or implicitly include at least one of that feature. In the description herein, “a plurality” means at least two, such as two, three, etc., unless otherwise explicitly specified.
[0094] It should be understood that spatial relative terms, such as “above,” “upper,” “below,” and “lower,” are used herein to describe the relationship between one element and another shown in the figures. In addition to the orientation depicted in the figures, these spatial relative terms are also intended to encompass different orientations of the device in use or operation. For example, if the device in the figures is flipped, an element described as “above” or “upper” relative to another element would be “below” or “lower” relative to that other element. Thus, depending on the spatial orientation of the device, the term “above” encompasses both above and below orientations. Devices may have other orientations (e.g., rotated 90 degrees or in other orientations), and the spatial relative terms used herein should be interpreted accordingly.
[0095] Furthermore, the term “exemplary” is used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as “exemplary” is not necessarily to be construed as advantageous compared to other aspects or designs. Rather, the use of the term “exemplary” is intended to present the concept in a concrete manner. As used herein, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or.” That is, unless otherwise specified or clear from the context, “X applies A or B” is intended to mean any of the natural inclusive arrangements. That is, “X applies A or B” satisfies any of the foregoing instances if X applies A; X applies B; or both X applies A and B. Additionally, unless otherwise specified or clear from the context to refer to the singular form, the articles “a” and “an” as used in this application and the appended claims are generally understood to mean “one or more.”
[0096] Similarly, although this disclosure has been shown and described with respect to one or more implementations, equivalent variations and modifications will occur to those skilled in the art upon reading and understanding this specification and the accompanying drawings. This disclosure includes all such modifications and variations and is limited only by the scope of the claims. In particular, with respect to the various functions performed by the components described above (e.g., elements, resources, etc.), unless otherwise indicated, the terminology used to describe such components is intended to correspond to any component (functionally equivalent) that performs the specific function of the described component, even if structurally not equivalent to the disclosed structure. Furthermore, although specific features of this disclosure may have been disclosed with respect to only one of several implementations, such features may be combined with one or more other features of other implementations, as may be desired and advantageous to any given or particular application. Moreover, with regard to the terms “comprising,” “owning,” “having,” “having,” or variations thereof as used in the detailed description or claims, such terms are intended to be inclusive in a manner similar to the term “including.”
[0097] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.
[0098] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.
Claims
1. A trajectory prediction method, characterized in that, include: Acquire the target vehicle's current environment image and current location information; Based on the current environment image and the current location information, at least one candidate trajectory is obtained; Determine a predicted environment image for each candidate trajectory; determine the target trajectory from the candidate trajectories based on the predicted environment image.
2. The method according to claim 1, characterized in that, The step of obtaining at least one candidate trajectory based on the current environment image and the current location information includes: inputting the current environment image and the current location information into a pre-trained trajectory prediction model with variable temperature parameters to obtain multiple reference trajectories, wherein different temperature parameters correspond to different reference trajectories; for each reference trajectory, determining the driving prediction result of the reference trajectory, the driving prediction result including collision prediction result and / or, ride experience prediction result; and determining the candidate trajectory from the multiple reference trajectories based on the driving prediction result.
3. The method according to claim 2, characterized in that, The step of determining the candidate trajectory from multiple reference trajectories based on the driving prediction result includes: if the collision prediction result of the reference trajectory indicates that the target vehicle will not collide when driving along the reference trajectory, and / or the ride experience prediction result indicates that the speed change of the target vehicle during the driving process along the reference trajectory is less than a speed change threshold, the maximum acceleration is less than an acceleration threshold, and the acceleration change in adjacent time periods is less than an acceleration change threshold, then the reference trajectory is determined as the candidate trajectory.
4. The method according to claim 2, characterized in that, The trajectory prediction model is trained as follows: multiple first training sample data of the target vehicle are acquired, each first training sample data includes a first sample historical image, first sample historical location information, and a first sample trajectory, wherein the first sample historical image and the first sample historical location information are used as input data of the trajectory prediction model, the first sample trajectory is the target output data of the trajectory prediction model, and the first sample trajectory includes multiple grid markers, the grid markers are used to indicate the positions of discrete points constituting the trajectory; the trajectory prediction model is then trained.
5. The method according to claim 4, characterized in that, The method further includes: acquiring multiple test sample data of the target vehicle, each test sample data including a second sample historical image, a second sample historical position, and a second sample future image; inputting the second sample historical image and the second sample historical position into the pre-trained trajectory prediction model with variable temperature parameters to obtain multiple second sample prediction trajectories; determining the prediction image of each second sample prediction trajectory; and evaluating the trajectory prediction model based on the prediction image of each second sample prediction trajectory and the corresponding second sample future image.
6. The method according to claim 1, characterized in that, Determining the predicted environment image for each candidate trajectory includes: for each candidate trajectory, inputting the candidate trajectory and the current environment image into a pre-trained predicted environment generation model to obtain the predicted environment image.
7. The method according to claim 6, characterized in that, The predictive environment generation model is trained as follows: multiple second training sample data of the target vehicle are acquired, each second training sample data includes a first sample historical image, a first sample future image and a first sample trajectory, the first sample historical image and the first sample trajectory are the input data of the predictive environment generation model, and the first sample future image is the target output data of the predictive environment generation model; the predictive environment generation model is trained.
8. The method according to claim 1, characterized in that, The step of determining the target trajectory from the candidate trajectories based on the predicted environment image includes: if the predicted environment image represents the target position inside the target vehicle from a perspective that does not deviate from the road centerline and has a clear field of vision, then the candidate trajectory is determined as the target trajectory.
9. A trajectory prediction device, characterized in that, include: The acquisition module is used to acquire the current environmental image and current location information of the target vehicle; The first determining module is used to obtain at least one candidate trajectory based on the current environment image and the current location information; The second determining module is used to determine the predicted environment image for each of the candidate trajectories; The third determining module is used to determine the target trajectory from the candidate trajectories based on the predicted environment image.
10. A trajectory prediction device, characterized in that, include: processor; A memory for storing processor-executable instructions; wherein the processor is configured to execute the computer program in the memory to implement the steps of the method as claimed in any one of claims 1-8.
11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1-8.
12. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1-8.