Trajectory prediction method, device, storage medium and program product for autonomous driving

Abstract

The present application discloses a trajectory prediction method, device, storage medium and program product for autonomous driving, which relate to the field of autonomous driving technology. Based on obtaining vehicle collection information and driving navigation information of a target vehicle, a pre-trained visual language model is used to process the vehicle collection information and driving navigation information to output multimodal joint features; then, an original trajectory prediction model is used to output a first predicted trajectory according to the multimodal joint features; then, based on the first predicted trajectory, a target time series trajectory prediction model is determined; a second predicted trajectory is output by the target time series trajectory prediction model according to the multimodal joint features and the first predicted trajectory; then, the multimodal joint features and the second predicted trajectory are input into a generative strategy optimization model to output the target predicted trajectory; and the problem of inaccurate trajectory prediction in the related art is solved.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to an autonomous driving trajectory prediction method, device, storage medium, and program product. Background Art

[0002] Autonomous driving technology aims to ensure safe driving of vehicles in complex and dynamic environments, and trajectory prediction is the key link in connecting perception and decision-making planning to achieve autonomous driving.

[0003] In related technologies, image information acquired by a vehicle is often input into a large model, which directly generates a predicted trajectory. However, solutions in related technologies suffer from the problem of inaccurate trajectory prediction. Summary of the Invention

[0004] The present application provides a trajectory prediction method, device, storage medium and program product for autonomous driving to at least solve the problem of inaccurate trajectory prediction in related technologies.

[0005] The present application provides a trajectory prediction method for autonomous driving, comprising: obtaining vehicle acquisition information and driving navigation information of a target vehicle; inputting the vehicle acquisition information and driving navigation information into a pre-trained visual language model, and outputting multimodal joint features; inputting the multimodal joint features into an original trajectory prediction model, and outputting a first predicted trajectory; determining a target temporal trajectory prediction model from a plurality of temporal trajectory prediction models based on the first predicted trajectory; inputting the multimodal joint features and the first predicted trajectory into a target temporal trajectory prediction model, and outputting a second predicted trajectory; inputting the multimodal joint features and the second predicted trajectory into a generative strategy optimization model, and outputting a target predicted trajectory.

[0006] This application also provides a trajectory prediction device for autonomous driving, comprising:

[0007] An acquisition module is used to obtain vehicle collection information and driving navigation information of the target vehicle;

[0008] a processing module configured to input vehicle acquisition information and driving navigation information into a pre-trained visual language model and output a multimodal joint feature; input the multimodal joint feature into an original trajectory prediction model and output a first predicted trajectory; determine a target temporal trajectory prediction model from a plurality of temporal trajectory prediction models based on the first predicted trajectory; input the multimodal joint feature and the first predicted trajectory into the target temporal trajectory prediction model and output a second predicted trajectory;

[0009] The output module is used to input the multimodal joint features and the second predicted trajectory into the generative policy optimization model and output the target predicted trajectory.

[0010] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-mentioned autonomous driving trajectory prediction methods when executing the computer program.

[0011] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned autonomous driving trajectory prediction methods are implemented.

[0012] The present application also provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned autonomous driving trajectory prediction methods when executed by a processor.

[0013] The trajectory prediction method, device, storage medium and program product for autonomous driving provided by the present application, based on obtaining vehicle collection information and driving navigation information of the target vehicle, output multimodal joint features by processing the vehicle collection information and driving navigation information with a pre-trained visual language model; then, the original trajectory prediction model outputs a first predicted trajectory based on the multimodal joint features; then, based on the first predicted trajectory, the target time series trajectory prediction model is determined; the target time series trajectory prediction model outputs a second predicted trajectory based on the multimodal joint features and the first predicted trajectory; then, the multimodal joint features and the second predicted trajectory are input into a generative strategy optimization model to output the target predicted trajectory; thereby solving the problem of inaccurate trajectory prediction in related technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0015] Figure 1 Schematic diagram of the process of the automatic driving trajectory prediction method provided in the embodiment of the application Figure 1 ;

[0016] Figure 2 A schematic diagram of the process of processing data based on the target time series trajectory prediction model provided in an embodiment of the present application;

[0017] Figure 3 Schematic diagram of the process of the automatic driving trajectory prediction method provided in the embodiment of the application Figure 2 ;

[0018] Figure 4 A schematic diagram of the structure of a trajectory prediction device for autonomous driving provided in an embodiment of the present application;

[0019] Figure 5 This is a schematic diagram of the structure of the electronic device provided in this application. DETAILED DESCRIPTION

[0020] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0021] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0022] Autonomous driving technology aims to ensure safe driving of vehicles in complex and dynamic environments, and trajectory prediction is the key link in connecting perception and decision-making planning to achieve autonomous driving.

[0023] In related technologies, image information acquired by a vehicle is often input into a large model, which directly generates a predicted trajectory. However, solutions in related technologies suffer from the problem of inaccurate trajectory prediction.

[0024] In order to solve the above technical problems, the embodiments of the present application propose the following technical concepts:

[0025] On the basis of obtaining the vehicle collection information and driving navigation information of the vehicle, the vehicle collection information and driving navigation information are input into a pre-trained visual language model to output a multimodal joint feature; then, the multimodal joint feature is input into the original trajectory prediction model to output a first predicted trajectory; wherein, the original trajectory prediction model is built and trained based on historical multimodal joint features; then, based on the first predicted trajectory, a target time series trajectory prediction model is determined from multiple time series trajectory prediction models; the multimodal joint feature and the first predicted trajectory are input into the target time series trajectory prediction model to output a second predicted trajectory; further, the multimodal joint feature and the second predicted trajectory are input into a generative strategy optimization model to output a target predicted trajectory. By processing the vehicle collection information and driving navigation information through the above-mentioned multiple pre-trained models, an accurate predicted trajectory can be generated, which solves the problem of inaccurate trajectory prediction in the solutions of the related art.

[0026] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0027] Figure 1 Schematic diagram of the process of the automatic driving trajectory prediction method provided in the embodiment of the application Figure 1 ,like Figure 1 As shown, the embodiment of the present application provides a trajectory prediction method for autonomous driving, which is applied to an electronic control unit, a vehicle control unit or a vehicle. The method is described in detail as follows:

[0028] Step S101: Acquire vehicle collection information and driving navigation information of the target vehicle.

[0029] Exemplarily, the target vehicle's vehicle-collected information is based on information collected by various sensors of the vehicle while the vehicle is in motion and / or stopped. For example, image acquisition information is information collected by at least one of the vehicle's visual sensors. Driving navigation information is based on information received by the vehicle's information receiving module. For example, vehicle location information is based on information received by the vehicle's satellite data receiving module, and wind information is based on information received by the vehicle's communication module, where the communication module establishes a data connection with a meteorological data platform. Alternatively, wind information is information collected by at least one of the vehicle's wind speed sensors, such as a mechanical wind speed sensor, an ultrasonic wind speed sensor, or a thermal film wind speed sensor.

[0030] Step S102: input the vehicle collection information and driving navigation information into the pre-trained visual language model, and output multimodal joint features.

[0031] Exemplarily, a pre-trained visual language model can determine corresponding preset text prompts based on the model of the vehicle being used. Furthermore, the pre-trained visual language model processes the vehicle collection information and driving navigation information based on the preset text prompts, thereby unifying the vehicle collection information and driving navigation information in terms of language (semantic) and feature representation, and outputting multimodal joint features. Specifically, for example, the pre-trained visual language model vectorizes the vehicle collection information and driving navigation information based on the preset text prompts, and outputs multimodal joint features. The multimodal joint features include a first feature vector and a second feature vector, wherein the vector dimensions of the first feature vector and the second feature vector are consistent.

[0032] Specifically, the specific implementation steps of step S102 include:

[0033] Step S1021, determining image acquisition information, road friction, and vehicle speed based on vehicle acquisition information.

[0034] For example, by parsing vehicle-collected information, various types of information collected by various sensors of the vehicle while the vehicle is in motion and / or stopped, namely, image acquisition information, road friction, and vehicle speed, are determined. Specifically, the image acquisition information includes traffic sign information, electronic traffic signal information, manual traffic signal information, the type of other vehicles, and other vehicle lighting information. Different types of other vehicles have different impacts on the predicted trajectory. For example, if the other vehicle is a heavy truck, if the heavy truck is in front of the target vehicle, the predicted trajectory will consider allowing the target vehicle to overtake to avoid a continuous restriction in the visual sensor's field of view when collecting information. If the heavy truck is behind the target vehicle, the predicted trajectory will consider the heavy truck's safe braking distance, thereby causing the vehicle to speed up or change lanes to ensure the target vehicle's driving safety. However, if the other vehicle is a car, the trajectory prediction strategy will be different. If the car is in front of the target vehicle, the predicted trajectory may not allow the target vehicle to overtake because the car has a smaller impact on the visual sensor's field of view when collecting information than the heavy truck.

[0035] For another example, road friction is calculated based on measurements from the vehicle's wheel speed sensors and / or inertial measurement units. Specifically, the wheel speed is determined by the wheel speed sensors, and the road friction (or maximum friction) currently available from the road surface is calculated based on the difference between the target vehicle's speed and the wheel speed. Furthermore, the longitudinal / lateral acceleration and yaw rate of the target vehicle are measured by the inertial measurement unit, and the actual acceleration of the target vehicle is compared with the theoretical maximum acceleration in conjunction with the vehicle dynamics model to determine the road friction (or maximum friction) currently available from the road surface.

[0036] Step S1022: Determine vehicle position information, road width information, and wind force information based on driving navigation information.

[0037] For example, by parsing driving navigation information, various types of information received by the vehicle-based information receiving module are determined, namely, vehicle location information, road width information, and wind information. Specifically, road width information refers to the width of curved roads (including right-angle bends). Due to the curvature of the road, the vehicle's visual sensors cannot capture road width information. Therefore, the satellite data receiving module receives satellite data to obtain road width information. Wind information includes wind speed and direction information.

[0038] In step S1023, the image acquisition information, road friction, vehicle speed, vehicle position information, road width information, and wind information are input into a pre-trained visual language model, and multimodal joint features are output; wherein the multimodal joint features include vehicle position features, image features, road friction features, vehicle speed features, and road width features.

[0039] Exemplarily, since the data formats of image acquisition information, road friction, vehicle speed, vehicle position information, road width information and wind information are different, the image acquisition information, road friction, vehicle speed, vehicle position information, road width information and wind information are input into a pre-trained visual language model, so that the pre-trained visual language model processes the image acquisition information, road friction, vehicle speed, vehicle position information, road width information and wind information according to preset text prompts, so that the above information is unified in terms of language dimension (semantic dimension) and feature representation dimension, and then outputs multimodal joint features; wherein the multimodal joint features include vehicle position features, image features, road friction features, vehicle speed features and road width features.

[0040] Step S103: Input the multimodal joint features into the original trajectory prediction model and output a first predicted trajectory.

[0041] Illustratively, the original trajectory prediction model is constructed based on a neural network model and trained with historical multimodal joint features; furthermore, the original trajectory prediction model can output a first predicted trajectory according to the input multimodal joint features.

[0042] Specifically, based on the embodiment of steps S1021-S1023, it is determined that the multimodal joint features include vehicle position features, image features, road friction features, vehicle speed features, and road width features; furthermore, the specific implementation steps of step S103 include:

[0043] Step S1031: Obtain road condition complexity features based on image features and road width features.

[0044] For example, based on image features, obstacles (objects to be avoided) surrounding the target vehicle can be identified, and based on road width features, the width of the target vehicle's road can be determined. Therefore, based on the obstacles surrounding the target vehicle and the width of the target vehicle's road, a road complexity feature can be derived. Specifically, for example, if the obstacles surrounding the target vehicle are determined, the wider the road width of the target vehicle's road, the less complex the road complexity (road complexity feature), and the narrower the road width of the target vehicle's road, the more complex the road complexity (road complexity feature). If the width of the target vehicle's road is determined, the fewer obstacles surrounding the target vehicle, the less complex the road complexity (road complexity feature), and the more obstacles surrounding the target vehicle, the more complex the road complexity (road complexity feature).

[0045] Step S1032: Obtain braking distance characteristics based on vehicle speed characteristics, road friction characteristics, and wind characteristics.

[0046] Exemplarily, wind characteristics include wind speed characteristics and wind direction characteristics; braking distance characteristics are comprehensively determined based on vehicle speed characteristics, road friction characteristics, wind speed characteristics, and wind direction characteristics; specifically, for example, the greater the vehicle speed, the smaller the road friction, and when the wind direction is the same as the target vehicle's driving direction, the greater the wind speed, the longer the braking distance; conversely, the lower the vehicle speed, the greater the road friction, and when the wind direction is opposite to the target vehicle's driving direction, the greater the wind speed, the shorter the braking distance; therefore, the braking distance characteristics are obtained by comprehensively analyzing the correlation between vehicle speed characteristics, road friction characteristics, and wind characteristics.

[0047] Step S1033 , obtaining trajectory prediction duration and trajectory prediction frequency based on vehicle speed characteristics, road condition complexity characteristics, and braking distance characteristics.

[0048] For example, since the greater the vehicle speed (vehicle speed characteristic), the longer the distance traveled per unit time, the reference value of the predicted trajectory in the latter part of the trajectory prediction time is small, so the trajectory prediction time should be shortened. For example, the reference value of the predicted trajectory in the third second of a 3-second trajectory prediction time is small, so the original 3-second trajectory prediction time is re-determined as a 2-second trajectory prediction time; similarly, the more complex the road condition complexity (road condition complexity characteristic), the less reference value of the predicted trajectory in the latter part of the trajectory prediction time, so the trajectory prediction time should be shortened.

[0049] Furthermore, the greater the vehicle speed (vehicle speed characteristic), the longer the distance traveled per unit time, so the number of predicted trajectory points per unit time should be increased, that is, the trajectory prediction frequency should be increased; similarly, the more complex the road condition complexity (road condition complexity characteristic), the more predicted trajectory points should be required per unit time; the shorter the braking distance (braking distance characteristic), the more predicted trajectory points should be required per unit time to ensure braking safety.

[0050] Therefore, the corresponding trajectory prediction duration and trajectory prediction frequency are obtained by analyzing the vehicle speed characteristics, road condition complexity characteristics and braking distance characteristics.

[0051] Step S1034: output a first predicted trajectory based on the vehicle position characteristics, trajectory prediction duration, and trajectory prediction frequency.

[0052] For example, after determining the trajectory prediction duration and frequency of the target trajectory, a first predicted trajectory of the target vehicle can be output based on the vehicle's position characteristics. Specifically, for example, if the trajectory prediction duration is 3 seconds and the trajectory prediction frequency is 4 per second, then 12 predicted trajectory points are obtained. The vehicle position corresponding to the vehicle position characteristics is then determined as the origin, resulting in a first predicted trajectory that combines the 12 predicted trajectory points, starting from the origin.

[0053] In another possible implementation, step S1034 includes outputting an original predicted trajectory based on the vehicle position feature vector, the road width feature vector, the trajectory prediction duration, and the trajectory prediction frequency. Specifically, for example, if the trajectory prediction duration is 3 seconds and the trajectory prediction frequency is 4 per second, 12 corresponding predicted trajectory points are obtained. The vehicle position corresponding to the vehicle position feature is then determined as the origin. Based on the road width feature vector, the 12 predicted trajectory points are spatially constrained so that each predicted trajectory point is within the road width. This results in a first predicted trajectory that combines the 12 predicted trajectory points and starts at the origin.

[0054] Step S104: determining a target time series trajectory prediction model from a plurality of time series trajectory prediction models according to the first prediction trajectory.

[0055] Exemplarily, the time series trajectory prediction model is determined based on the gated recurrent unit, and then, by determining the gated recurrent unit according to the first predicted trajectory, the target time series trajectory prediction model is determined.

[0056] Specifically, the specific implementation steps of step S104 include:

[0057] Step S1041: Determine the number of predicted trajectory points according to the first predicted trajectory.

[0058] Step S1042 : determining the number of gated recurrent units according to the predicted number of trajectory points.

[0059] Step S1043 : determining a target time series trajectory prediction model from a plurality of time series trajectory prediction models according to the number of units.

[0060] For example, if the trajectory prediction duration is 3 seconds and the trajectory prediction frequency is 4 per second, 12 predicted trajectory points are obtained; furthermore, based on the 12 predicted trajectory points, the number of gated recurrent units is determined to be 12; further, the multiple time series trajectory prediction models include a first time series trajectory prediction model, a second time series trajectory prediction model, and a third time series trajectory prediction model, wherein the first time series trajectory prediction model is determined based on 8 gated recurrent units, the second time series trajectory prediction model is determined based on 10 gated recurrent units, and the third time series trajectory prediction model is determined based on 12 gated recurrent units, so the third time series trajectory prediction model is determined as the target time series trajectory prediction model.

[0061] Step S105: input the multimodal joint features and the first predicted trajectory into the target time series trajectory prediction model, and output the second predicted trajectory.

[0062] Exemplarily, the target time series trajectory prediction model further processes the first predicted trajectory based on the multimodal joint features and the time series to output a second predicted trajectory.

[0063] Specifically, taking the case where the first predicted trajectory includes the first first predicted trajectory point and the second first predicted trajectory point, and the target temporal trajectory prediction model includes the first gated recurrent unit and the second gated recurrent unit as an example, the specific implementation steps of step S105 include:

[0064] Step S1051 : Input the multimodal joint feature and the first first predicted trajectory point into the first gated recurrent unit to obtain the first second predicted trajectory point.

[0065] Step S1052 : Input the first second predicted trajectory point and the second first predicted trajectory point into a second gated recurrent unit to obtain a second second predicted trajectory point.

[0066] Step S1053: output the second predicted trajectory according to the first second predicted trajectory point and the second second predicted trajectory point.

[0067] For example, Figure 2 A schematic diagram of the processing process of processing data based on the target time series trajectory prediction model provided in the embodiment of the present application is shown as follows: Figure 2 As shown, the first predicted trajectory point is , the second first predicted trajectory point is , combine the multimodal joint features and the first predicted trajectory point Input to the first gated recurrent unit to obtain the first second predicted trajectory point ; Then, the first second predicted trajectory point and the second first predicted trajectory point Input to the second gated recurrent unit to obtain the second predicted trajectory point ; Then, according to the first second predicted trajectory point and the second predicted trajectory point , the second predicted trajectory can be output.

[0068] Step S106: Input the multimodal joint features and the second predicted trajectory into the generative strategy optimization model, and output the target predicted trajectory.

[0069] Specifically, the generative strategy optimization model includes a reference model, a reward model, and a grouping model; the second predicted trajectory includes multiple second predicted trajectory points; and the specific implementation steps of step S106 include:

[0070] Step S1061: Input the multimodal joint features into the reference model and output the historical reference trajectory.

[0071] Exemplarily, based on the embodiment shown in steps S1021-S1023, it is determined that the multimodal joint features include vehicle position features, image features, road friction features, vehicle speed features and road width features; then, the reference model determines a historical reference trajectory from a preset database based on the input vehicle position features, image features, road friction features, vehicle speed features and road width features, wherein the feature distance between the multimodal joint features of the historical reference trajectory and the multimodal joint features of the target vehicle is less than a preset feature distance threshold.

[0072] Step S1062: Input the multimodal joint features, the second predicted trajectory, and the historical reference trajectory into the reward model, and output the trajectory point reward value corresponding to each second predicted trajectory point.

[0073] Exemplarily, the second predicted trajectory includes multiple second predicted trajectory points, and the historical reference trajectory includes multiple historical reference trajectory points; the mean square error of the corresponding second predicted trajectory points and historical reference trajectory points at each moment is calculated to obtain the first trajectory point reward value corresponding to each second predicted trajectory point; it is determined whether each second predicted trajectory point interferes with or is too close to the obstacle (object to be avoided) corresponding to the multimodal joint feature, and the corresponding second trajectory point reward value is determined based on whether interference occurs or the distance is too close, wherein if interference occurs or the distance is too close, a first type of second trajectory point reward value is determined; if no interference occurs, a second type of second trajectory point reward value is determined, and the first type of second trajectory point reward value is less than the second type of second trajectory point reward value; further, based on the first trajectory point reward value and the second trajectory point reward value corresponding to each second predicted trajectory point, the trajectory point reward value corresponding to each second predicted trajectory point is obtained and output.

[0074] Step S1063: Input each second predicted trajectory point and the trajectory point reward value corresponding to each second predicted trajectory point into the grouping model, and output the target predicted trajectory.

[0075] Exemplarily, the grouping model groups the trajectory point reward values ​​corresponding to each second predicted trajectory point according to the trajectory point reward value threshold, and determines a group of trajectory point reward values ​​greater than or equal to the trajectory point reward value threshold as the target trajectory point reward value group; then, each second predicted trajectory point corresponding to each trajectory point reward value in the target trajectory point reward value group is determined as each target predicted trajectory point; then, based on each target predicted trajectory point, the target predicted trajectory is obtained, and then the target predicted trajectory is output.

[0076] In another possible implementation, after determining each second predicted trajectory point corresponding to each trajectory point reward value in the target trajectory point reward value group as each target predicted trajectory point, coordinate interpolation processing is performed according to the coordinates of each target predicted trajectory point to obtain at least one interpolated trajectory point; then, trajectory fitting is performed based on each target predicted trajectory point and at least one interpolated trajectory point to obtain a target predicted trajectory, and then the target predicted trajectory is output; the smoothness of the target predicted trajectory is improved, and the stability of the target vehicle when driving based on the target predicted trajectory is ensured.

[0077] Furthermore, after step S1063, the method provided in the embodiment of the present application further includes:

[0078] Step S1064: Obtain a KL divergence based on the historical reference trajectory and the second predicted trajectory. The KL divergence is used to adjust the model parameters of the pre-trained visual language model, and / or the model parameters of the original trajectory prediction model, and / or the model parameters of the target temporal trajectory prediction model.

[0079] For example, the second predicted trajectory includes multiple second predicted trajectory points, and the probability distribution corresponding to each second predicted trajectory point is: and the corresponding probability values The historical reference trajectory includes multiple historical reference trajectory points and the probability distribution corresponding to each historical reference trajectory point. and the corresponding probability values ;in, For the The probability value of the second predicted trajectory point, For the The probability value of a historical reference trajectory point.

[0080] Then, the probability value corresponding to each second predicted trajectory point is , the probability value corresponding to each historical reference trajectory point , substituting into formula (1), we can get the KL divergence ,

[0081] (1)

[0082] in, is the probability distribution corresponding to each second predicted trajectory point; is the probability distribution corresponding to each historical reference trajectory point; For the The probability value of the second predicted trajectory point; For the The probability value of a historical reference trajectory point; is the total number of trajectory points of the second predicted trajectory point, and is also the total number of trajectory points of the historical reference trajectory point.

[0083] Furthermore, based on the calculated KL divergence, the model parameters of the pre-trained visual language model, and / or the model parameters of the original trajectory prediction model, and / or the model parameters of the target temporal trajectory prediction model can be adjusted so that the second predicted trajectory output by the pre-trained visual language model, the original trajectory prediction model, and the target temporal trajectory prediction model is closer to the historical reference trajectory output by the reference model.

[0084] In this embodiment, on the basis of obtaining the vehicle collection information and driving navigation information of the target vehicle, the vehicle collection information and driving navigation information are processed by a pre-trained visual language model to output a multimodal joint feature; then, the original trajectory prediction model outputs a first predicted trajectory according to the multimodal joint feature; then, based on the first predicted trajectory, the target time series trajectory prediction model is determined; the target time series trajectory prediction model outputs a second predicted trajectory according to the multimodal joint feature and the first predicted trajectory; then, the multimodal joint feature and the second predicted trajectory are input into the generative strategy optimization model to output the target predicted trajectory; thus, the problem of inaccurate trajectory prediction in the related art is solved.

[0085] After step S106, the method provided in the embodiment of the present application further includes:

[0086] Step S107 , obtaining a mean square error based on the plurality of second predicted trajectory points and the plurality of target predicted trajectory points; the mean square error is used to adjust model parameters of the target time series trajectory prediction model.

[0087] Specifically, the second predicted trajectory includes a plurality of second predicted trajectory points; the target predicted trajectory includes a plurality of target predicted trajectory points; the coordinate values ​​of each second predicted trajectory point are , the coordinate values ​​of each target prediction trajectory point , substituting into formula (2), we can calculate the mean square error ,

[0088] (2)

[0089] in, is the total number of trajectory points of the second predicted trajectory point, and is also the total number of trajectory points of the target predicted trajectory point; For the The coordinate components of the second predicted trajectory point; For the The coordinate components of the target predicted trajectory points.

[0090] The mean square error is used to adjust the model parameters of the target time series trajectory prediction model so that the second predicted trajectory output based on the target time series trajectory prediction model is closer to the target predicted trajectory.

[0091] Figure 3 Schematic diagram of the process of the automatic driving trajectory prediction method provided in the embodiment of the application Figure 2 In the embodiment of the present application, Figure 1 Based on the embodiment provided, steps S102 and S103 are further refined. Figure 3 As shown, the method includes:

[0092] S201: Acquire vehicle collection information and driving navigation information of a target vehicle.

[0093] S202: Determine image acquisition information, road friction, and vehicle speed based on vehicle acquisition information.

[0094] S203: Determine vehicle position information, road width information, and wind force information based on driving navigation information.

[0095] S204: Obtaining a visibility value based on the image acquisition information.

[0096] S205: Determine the magnitude relationship between the visibility value and the visibility threshold.

[0097] Exemplarily, based on the image acquisition information, the visibility of the environment in which the target vehicle is located is determined, and then the relationship between the visibility value and the visibility threshold is judged. If the visibility value is less than the visibility threshold, it indicates that relying solely on the image acquisition information to determine the obstacles around the target vehicle (objects to be avoided) may result in missing obstacles around the target vehicle, so steps S206-S211 are executed to output the first predicted trajectory; if the visibility value is greater than or equal to the visibility threshold, it indicates that the obstacles around the target vehicle (objects to be avoided) can be determined based on the image acquisition information, so steps S212-S213 are executed to output the first predicted trajectory.

[0098] S206: If the visibility value is less than the visibility threshold, determine the vehicle-mounted radar point cloud information based on the vehicle collected information.

[0099] For example, when the visibility value is less than the visibility threshold, the on-board radar point cloud information can be determined based on the vehicle collection information, so as to achieve the subsequent steps of determining the obstacles around the target vehicle based on the image collection information and the on-board radar point cloud information.

[0100] S207: Inputting image acquisition information, vehicle-mounted radar point cloud information, road friction, vehicle speed, vehicle position information, road width information and wind information into a pre-trained visual language model, and outputting multimodal joint features; wherein the multimodal joint features include vehicle position features, image features, vehicle-mounted radar point cloud features, road friction features, vehicle speed features, road width features and wind features.

[0101] In the steps of this embodiment, the implementation method of step S207 is the same as that of the present application. Figure 1 The implementation of step S1023 in the illustrated embodiment is similar and will not be described again here.

[0102] S208: Obtaining road condition complexity features based on the image features, the vehicle-mounted radar point cloud features, and the road width features.

[0103] For example, based on image features and on-board radar point cloud features, obstacles around the target vehicle (objects that need to be avoided) can be determined, and based on road width features, the width of the road where the target vehicle is located can be determined. Therefore, based on the obstacles around the target vehicle and the width of the road where the target vehicle is located, the road condition complexity features can be obtained.

[0104] S209: Obtain braking distance characteristics based on vehicle speed characteristics, road friction characteristics, and wind characteristics.

[0105] In the steps of this embodiment, the implementation method of step S209 is the same as that of the present application. Figure 1 The implementation of step S1032 in the illustrated embodiment is the same and will not be repeated here.

[0106] S210: Obtaining trajectory prediction duration and trajectory prediction frequency based on vehicle speed characteristics, road condition complexity characteristics, and braking distance characteristics.

[0107] In the steps of this embodiment, the implementation method of step S210 is the same as that of the present application. Figure 1 The implementation method of step S1033 in the illustrated embodiment is the same and will not be repeated here.

[0108] S211: Output a first predicted trajectory based on the vehicle position characteristics, trajectory prediction duration, and trajectory prediction frequency.

[0109] In the steps of this embodiment, the implementation method of step S211 is the same as that of the present application. Figure 1 The implementation method of step S1034 in the illustrated embodiment is the same and will not be repeated here.

[0110] In the steps of this embodiment, by judging the size relationship between the visibility value and the visibility threshold, when the visibility value is less than the visibility threshold, the on-board radar point cloud information is determined based on the vehicle collection information, so as to achieve the subsequent steps of determining the obstacles around the target vehicle based on the image collection information and the on-board radar point cloud information; and then generating and outputting a first predicted trajectory, thereby ensuring the accuracy of the first predicted trajectory used in the subsequent steps, and the accuracy of the historical reference trajectory determined by the generative strategy optimization model based on the multimodal joint features, thereby improving the accuracy of the target predicted trajectory and ensuring the safety of the target vehicle when driving based on the target predicted trajectory.

[0111] S212: If the visibility value is greater than or equal to the visibility threshold, the image acquisition information, road friction, vehicle speed, vehicle position information, road width information and wind information are input into the pre-trained visual language model, and multimodal joint features are output, wherein the multimodal joint features include vehicle position features, image features, road friction features, vehicle speed features, road width features and wind features.

[0112] In the steps of this embodiment, the implementation method of step S212 is the same as that of the present application. Figure 1 The implementation method of step S1023 in the illustrated embodiment is the same and will not be repeated here.

[0113] S213: Obtain road condition complexity characteristics based on image features and road width characteristics; obtain braking distance characteristics based on vehicle speed characteristics, road friction characteristics, and wind characteristics; obtain trajectory prediction duration and trajectory prediction frequency based on vehicle speed characteristics, road condition complexity characteristics, and braking distance characteristics; output a first predicted trajectory based on vehicle position characteristics, trajectory prediction duration, and trajectory prediction frequency.

[0114] In the steps of this embodiment, the implementation method of step S213 is the same as that of the present application. Figure 1 The implementation of step S103 in the illustrated embodiment is the same and will not be described again here.

[0115] S214: Determine a target time series trajectory prediction model from a plurality of time series trajectory prediction models according to the first prediction trajectory.

[0116] S215: Input the multimodal joint features and the first predicted trajectory into a target time series trajectory prediction model, and output a second predicted trajectory.

[0117] S216: Input the multimodal joint features and the second predicted trajectory into the generative strategy optimization model, and output the target predicted trajectory.

[0118] In this embodiment, the implementation of step S201 is the same as that of the present application. Figure 1 The implementation of step S101 in the embodiment shown is the same as that of the present application. The implementation of steps S214 to S216 is the same as that of the present application. Figure 1 The implementation methods of steps S104 to S106 in the illustrated embodiment are the same and will not be described in detail here.

[0119] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0120] Figure 4This is a schematic diagram of the structure of the trajectory prediction device for autonomous driving provided in an embodiment of the present application. Figure 4 As shown, an embodiment of the present application further provides an autonomous driving trajectory prediction device 3, comprising:

[0121] An acquisition module 31 is used to acquire vehicle collection information and driving navigation information of a target vehicle;

[0122] The processing module 32 is configured to input the vehicle collection information and driving navigation information into a pre-trained visual language model and output a multimodal joint feature; input the multimodal joint feature into an original trajectory prediction model and output a first predicted trajectory; determine a target temporal trajectory prediction model from a plurality of temporal trajectory prediction models based on the first predicted trajectory; input the multimodal joint feature and the first predicted trajectory into the target temporal trajectory prediction model and output a second predicted trajectory;

[0123] The output module 33 is used to input the multimodal joint features and the second predicted trajectory into the generative strategy optimization model and output the target predicted trajectory.

[0124] In one possible implementation, when the processing module 32 inputs the vehicle acquisition information and driving navigation information into a pre-trained visual language model and outputs multimodal joint features, it is specifically used to: determine the image acquisition information, road friction, and vehicle driving speed based on the vehicle acquisition information; determine the vehicle position information, road width information, and wind information based on the driving navigation information; input the image acquisition information, road friction, vehicle driving speed, vehicle position information, road width information, and wind information into the pre-trained visual language model, and output multimodal joint features; wherein the multimodal joint features include vehicle position features, image features, road friction features, vehicle driving speed features, road width features, and wind features.

[0125] In one possible implementation, when the processing module 32 inputs the multimodal joint features into the original trajectory prediction model and outputs the first predicted trajectory, it is specifically used to: obtain road condition complexity features based on image features and road width features; obtain braking distance features based on vehicle speed features, road friction features, and wind characteristics; obtain trajectory prediction duration and trajectory prediction frequency based on vehicle speed features, road condition complexity features, and braking distance features; and output the first predicted trajectory based on vehicle position features, trajectory prediction duration, and trajectory prediction frequency.

[0126] In one possible implementation, when the processing module 32 determines the target time series trajectory prediction model from multiple time series trajectory prediction models based on the first prediction trajectory, it is specifically used to: determine the number of prediction trajectory points based on the first prediction trajectory; determine the number of units of the gated loop unit based on the number of prediction trajectory points; and determine the target time series trajectory prediction model from the multiple time series trajectory prediction models based on the number of units.

[0127] In one possible implementation, the first predicted trajectory includes a first first predicted trajectory point and a second first predicted trajectory point; the target time series trajectory prediction model includes a first gated recurrent unit and a second gated recurrent unit; when the processing module 32 inputs the multimodal joint feature and the first predicted trajectory into the target time series trajectory prediction model and outputs the second predicted trajectory, it is specifically used to: input the multimodal joint feature and the first first predicted trajectory point into the first gated recurrent unit to obtain a first second predicted trajectory point; input the first second predicted trajectory point and the second first predicted trajectory point into the second gated recurrent unit to obtain a second second predicted trajectory point; and output the second predicted trajectory based on the first second predicted trajectory point and the second second predicted trajectory point.

[0128] In one possible implementation, the generative strategy optimization model includes a reference model, a reward model, and a grouping model; the second predicted trajectory includes multiple second predicted trajectory points; when the output module 33 inputs the multimodal joint features and the second predicted trajectory into the generative strategy optimization model and outputs the target predicted trajectory, it is specifically used to: input the multimodal joint features into the reference model and output the historical reference trajectory; input the multimodal joint features, the second predicted trajectory, and the historical reference trajectory into the reward model and output the trajectory point reward value corresponding to each second predicted trajectory point; input each second predicted trajectory point and the trajectory point reward value corresponding to each second predicted trajectory point into the grouping model and output the target predicted trajectory.

[0129] In one possible implementation, after the multimodal joint features are input into the reference model and the historical reference trajectory is output, the processing module 32 is further used to: obtain the KL divergence based on the historical reference trajectory and the second predicted trajectory; the KL divergence is used to adjust the model parameters of the pre-trained visual language model, and / or the model parameters of the original trajectory prediction model, and / or the model parameters of the target time series trajectory prediction model.

[0130] In one possible implementation, the second predicted trajectory includes multiple second predicted trajectory points; the target predicted trajectory includes multiple target predicted trajectory points; after the multimodal joint features and the second predicted trajectory are input into the generative strategy optimization model and the target predicted trajectory is output, the trajectory prediction device 3 of the autonomous driving is further used to: obtain a mean square error based on the multiple second predicted trajectory points and the multiple target predicted trajectory points; the mean square error is used to adjust the model parameters of the target time series trajectory prediction model.

[0131] In one possible implementation, when the processing module 32 inputs the vehicle acquisition information and driving navigation information into the pre-trained visual language model and outputs the multimodal joint features, it is specifically used to: determine the image acquisition information, road friction, and vehicle driving speed based on the vehicle acquisition information; determine the vehicle position information, road width information, and wind information based on the driving navigation information; obtain the visibility value based on the image acquisition information; if the visibility value is less than the visibility threshold, determine the on-board radar point cloud information based on the vehicle acquisition information; input the image acquisition information, on-board radar point cloud information, road friction, vehicle driving speed, vehicle position information, road width information, and wind information into the pre-trained visual language model, and output the multimodal joint features; wherein the multimodal joint features include vehicle position features, image features, on-board radar point cloud features, road friction features, vehicle driving speed features, road width features, and wind features.

[0132] In one possible implementation, the processing module 32 is further used to: if the visibility value is greater than or equal to the visibility threshold, input the image acquisition information, road friction, vehicle speed, vehicle position information, road width information and wind information into the pre-trained visual language model, and output multimodal joint features, wherein the multimodal joint features include vehicle position features, image features, road friction features, vehicle speed features, road width features and wind features.

[0133] In one possible implementation, when the processing module 32 inputs the multimodal joint features into the original trajectory prediction model and outputs the first predicted trajectory, it is specifically used to: obtain road condition complexity features based on image features, vehicle-mounted radar point cloud features, and road width features; obtain braking distance features based on vehicle speed features, road friction features, and wind characteristics; obtain trajectory prediction duration and trajectory prediction frequency based on vehicle speed features, road condition complexity features, and braking distance features; and output the first predicted trajectory based on vehicle position features, trajectory prediction duration, and trajectory prediction frequency.

[0134] In a possible implementation, the image acquisition information includes traffic sign information, electronic traffic signal information, manual traffic signal information, vehicle types of other vehicles, and other vehicle lighting information.

[0135] For the description of the features in the embodiment corresponding to the trajectory prediction device 3 for autonomous driving, please refer to the relevant description of the embodiment corresponding to the trajectory prediction method for autonomous driving, and will not be repeated here.

[0136] Figure 5 This is a schematic diagram of the structure of the electronic device provided in this application. Figure 5 As shown, the electronic device 50 provided in this embodiment includes: at least one processor 501 and a memory 502. Optionally, the electronic device 50 further includes a communication component 503. The processor 501, the memory 502 and the communication component 503 are connected via a bus.

[0137] During the specific implementation process, at least one processor 501 executes the computer-executable instructions stored in the memory 502, so that at least one processor 501 executes the above-mentioned embodiment of the trajectory prediction method for autonomous driving.

[0138] The specific implementation process of the processor 501 can be found in the above method embodiment. Its implementation principle and technical effects are similar and will not be repeated here in this embodiment.

[0139] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the application may be directly executed by a hardware processor or by a combination of hardware and software modules within the processor.

[0140] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage.

[0141] A bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. Buses can be categorized as address buses, data buses, and control buses. For ease of illustration, the buses in the drawings of this application are not limited to just one bus or just one type of bus.

[0142] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program, wherein the computer program is configured to execute the steps of any of the above-mentioned embodiments of the autonomous driving trajectory prediction method when running.

[0143] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0144] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any of the above-mentioned embodiments of the autonomous driving trajectory prediction method are implemented.

[0145] An embodiment of the present application also provides another computer program product, including a non-volatile computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps in any of the above-mentioned embodiments of the autonomous driving trajectory prediction method.

[0146] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0147] The above is a detailed introduction to the trajectory prediction method, device, storage medium and program product for autonomous driving provided by this application. This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of this application, several improvements and modifications can be made to this application, and these improvements and modifications also fall within the scope of protection of the claims of this application.

Claims

1. A trajectory prediction method for autonomous driving, characterized in that: include: Obtain vehicle collection information and driving navigation information of the target vehicle; Inputting the vehicle collection information and the driving navigation information into a pre-trained visual language model to output multimodal joint features; Inputting the multimodal joint features into an original trajectory prediction model and outputting a first predicted trajectory; Determining a target time series trajectory prediction model from a plurality of time series trajectory prediction models according to the first prediction trajectory; Inputting the multimodal joint feature and the first predicted trajectory into the target time series trajectory prediction model, and outputting a second predicted trajectory; Inputting the multimodal joint feature and the second predicted trajectory into a generative strategy optimization model, and outputting a target predicted trajectory; The step of inputting the vehicle collection information and the driving navigation information into a pre-trained visual language model and outputting multimodal joint features includes: Determining image acquisition information, road friction, and vehicle speed based on the vehicle acquisition information; Determining vehicle position information, road width information, and wind force information based on the driving navigation information; Inputting the image acquisition information, the road friction, the vehicle speed, the vehicle position information, the road width information, and the wind force information into the pre-trained visual language model, and outputting the multimodal joint features; wherein the multimodal joint features include vehicle position features, image features, road friction features, vehicle speed features, road width features, and wind force features; Inputting the multimodal joint features into the original trajectory prediction model and outputting a first predicted trajectory includes: Obtaining a road condition complexity feature according to the image feature and the road width feature; Obtaining a braking distance characteristic according to the vehicle speed characteristic, the road friction characteristic, and the wind characteristic; Obtaining trajectory prediction duration and trajectory prediction frequency according to the vehicle speed characteristic, the road condition complexity characteristic, and the braking distance characteristic; Outputting the first predicted trajectory according to the vehicle position feature, the trajectory prediction duration, and the trajectory prediction frequency; The step of determining a target time series trajectory prediction model from a plurality of time series trajectory prediction models according to the first predicted trajectory includes: Determining the number of predicted trajectory points based on the first predicted trajectory; Determining the number of gated recurrent units according to the number of predicted trajectory points; The target time series trajectory prediction model is determined from a plurality of time series trajectory prediction models according to the number of units.

2. The automatic driving trajectory prediction method according to claim 1, characterized in that: The first predicted trajectory includes a first first predicted trajectory point and a second first predicted trajectory point; the target time series trajectory prediction model includes a first gated recurrent unit and a second gated recurrent unit; The step of inputting the multimodal joint feature and the first predicted trajectory into the target time series trajectory prediction model and outputting a second predicted trajectory includes: Inputting the multimodal joint feature and the first first predicted trajectory point into the first gated recurrent unit to obtain a first second predicted trajectory point; Inputting the first second predicted trajectory point and the second first predicted trajectory point into the second gated recurrent unit to obtain a second second predicted trajectory point; Outputting the second predicted trajectory according to the first second predicted trajectory point and the second second predicted trajectory point.

3. The automatic driving trajectory prediction method according to claim 1, characterized in that: The generative strategy optimization model includes a reference model, a reward model, and a grouping model; the second predicted trajectory includes a plurality of second predicted trajectory points; Inputting the multimodal joint feature and the second predicted trajectory into a generative strategy optimization model and outputting a target predicted trajectory includes: Inputting the multimodal joint features into the reference model and outputting a historical reference trajectory; Inputting the multimodal joint feature, the second predicted trajectory, and the historical reference trajectory into the reward model, and outputting a trajectory point reward value corresponding to each second predicted trajectory point; The second predicted trajectory points and the trajectory point reward values ​​corresponding to the second predicted trajectory points are input into the grouping model, and the target predicted trajectory is output.

4. The automatic driving trajectory prediction method according to claim 3, characterized in that: After inputting the multimodal joint features into the reference model and outputting the historical reference trajectory, the method further includes: A KL divergence is obtained based on the historical reference trajectory and the second predicted trajectory; the KL divergence is used to adjust model parameters of the pre-trained visual language model, and / or model parameters of the original trajectory prediction model, and / or model parameters of the target time series trajectory prediction model.

5. The automatic driving trajectory prediction method according to claim 1, characterized in that: The second predicted trajectory includes a plurality of second predicted trajectory points; the target predicted trajectory includes a plurality of target predicted trajectory points; After inputting the multimodal joint feature and the second predicted trajectory into the generative strategy optimization model and outputting the target predicted trajectory, the method further includes: A mean square error is obtained based on the multiple second predicted trajectory points and the multiple target predicted trajectory points; the mean square error is used to adjust model parameters of the target time series trajectory prediction model.

6. The automatic driving trajectory prediction method according to claim 1, characterized in that: The step of inputting the vehicle collection information and the driving navigation information into a pre-trained visual language model and outputting multimodal joint features includes: Determining image acquisition information, road friction, and vehicle speed based on the vehicle acquisition information; Determining vehicle position information, road width information, and wind force information based on the driving navigation information; Obtaining a visibility value according to the image acquisition information; If the visibility value is less than the visibility threshold, determining the vehicle-mounted radar point cloud information based on the vehicle collected information; The image acquisition information, the on-board radar point cloud information, the road friction, the vehicle speed, the vehicle position information, the road width information and the wind information are input into the pre-trained visual language model, and the multimodal joint features are output; wherein the multimodal joint features include vehicle position features, image features, on-board radar point cloud features, road friction features, vehicle speed features, road width features and wind features.

7. The automatic driving trajectory prediction method according to claim 6, characterized in that: Also includes: If the visibility value is greater than or equal to the visibility threshold, the image acquisition information, the road friction, the vehicle speed, the vehicle position information, the road width information and the wind information are input into a pre-trained visual language model, and the multimodal joint features are output, wherein the multimodal joint features include vehicle position features, image features, road friction features, vehicle speed features, road width features and wind features.

8. The automatic driving trajectory prediction method according to claim 6, characterized in that: Inputting the multimodal joint features into the original trajectory prediction model and outputting a first predicted trajectory includes: Obtaining a road condition complexity feature according to the image feature, the vehicle-mounted radar point cloud feature, and the road width feature; Obtaining a braking distance characteristic according to the vehicle speed characteristic, the road friction characteristic, and the wind characteristic; Obtaining trajectory prediction duration and trajectory prediction frequency according to the vehicle speed characteristic, the road condition complexity characteristic, and the braking distance characteristic; The first predicted trajectory is output according to the vehicle position feature, the trajectory prediction duration, and the trajectory prediction frequency.

9. The method for trajectory prediction of autonomous driving according to any one of claims 1 and 6-8, characterized in that: The image acquisition information includes traffic sign information, electronic traffic signal information, manual traffic signal information, vehicle types of other vehicles, and other vehicle lighting information.

10. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the autonomous driving trajectory prediction method according to any one of claims 1 to 9 when executing the computer program.

11. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the trajectory prediction method for autonomous driving are implemented.

12. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the trajectory prediction method for autonomous driving are implemented.

Images