Vehicle trajectory planning method and device, electronic equipment and storage medium

Abstract

The present disclosure provides a vehicle trajectory planning method, device, electronic equipment and storage medium. In the embodiment of the present disclosure, the vehicle trajectory planning method comprises: generating a trajectory vector of the vehicle by a first time sequence network coding using historical trajectory data of the vehicle; generating an image vector of the vehicle by a feature extraction network coding using current front view image data of the vehicle; generating a scene classification vector by a compression module and a second time sequence network coding using historical front view image data and current front view image data of the vehicle, the scene classification vector containing feature information of a current scene in which the vehicle is located; and obtaining future trajectory planning data of the vehicle by a third time sequence network and a multi-task head using the image vector, the trajectory vector and the scene classification vector. The present disclosure can realize scene-adaptive vehicle trajectory planning, does not require high-cost high-precision maps and complex manually designed vectorization rules, has good robustness in different scenes, is highly accurate and has good real-time performance.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to a vehicle trajectory planning method and device, an electronic device, and a storage medium. BACKGROUND

[0002] In an autonomous driving scenario, trajectory planning of a vehicle is crucial for various tasks related to the vehicle in the autonomous driving scenario, and the trajectory planning of the vehicle is mainly achieved by training a model such as a neural network. Although there are many technologies related to trajectory planning at present, there are still problems such as low model training efficiency, poor robustness of the model in different scenarios, low model execution efficiency leading to poor real-time trajectory prediction, the model not meeting the actual driving conditions of the vehicle leading to poor trajectory prediction accuracy, and model training relying on high-cost high-precision maps in the vehicle trajectory planning method suitable for the autonomous driving scenario. SUMMARY

[0003] To solve at least one of the above technical problems, the present disclosure provides a vehicle trajectory planning method, device, electronic device, and storage medium.

[0004] According to a first aspect of the present disclosure, a vehicle trajectory planning method is provided, comprising:

[0005] generating a trajectory vector of the vehicle by encoding historical trajectory data of the vehicle through a first time sequence network;

[0006] generating an image vector of the vehicle by encoding current front-view image data of the vehicle through a feature extraction network;

[0007] generating a scene classification vector by encoding historical front-view image data and current front-view image data of the vehicle through a compression module and a second time sequence network, the scene classification vector containing feature information of a current scene in which the vehicle is located;

[0008] obtaining future trajectory planning data of the vehicle by using the image vector, the trajectory vector, and the scene classification vector through a third time sequence network and a multi-task head.

[0009] In some embodiments, the generating a trajectory vector of the vehicle by encoding historical trajectory data of the vehicle through a first time sequence network comprises:

[0010] obtaining historical trajectory data of the vehicle in a predetermined period of time in the past, extracting n historical positions in the historical trajectory data, and converting the n historical positions to a vehicle body coordinate system at a current time to obtain a historical position sequence of the vehicle in the vehicle body coordinate system at the current time;

[0011] The historical position sequence of the vehicle in the current vehicle body coordinate system is input into a first time-series network of length n. The first time-series network collects and combines information from the earliest historical position to the latest historical position in the historical position sequence to generate a trajectory vector.

[0012] In some implementations, the trajectory vector includes one or more of the following kinematic information: acceleration change, velocity change, position change, and path change.

[0013] In some implementations, the image vector includes road information, other vehicle information, and obstacle information for the vehicle's current location.

[0014] In some implementations, the current front view image data of the vehicle is used to generate an image vector of the vehicle through a feature extraction network, including:

[0015] Perform uniform scaling on each frame of the current front view image data;

[0016] Normalize the current front view image of each scaled frame in the RGB channel direction;

[0017] The normalized current foreground images of each frame are processed by a feature extraction network to generate image vectors of a predetermined length.

[0018] In some implementations, the scene classification vector includes one or more of the following information used to describe specific scene features:

[0019] This indicates whether the weather will be rainy or foggy.

[0020] Indicates whether the location is in an urban area or in a mountainous area;

[0021] Information indicating the current level of congestion;

[0022] Information indicating whether to drive at night;

[0023] The indicator shows whether it is at an intersection or in the straight lane;

[0024] Information indicating whether the vehicle in front should change lanes;

[0025] Information indicating whether the vehicle in front should pull over.

[0026] In some implementations, scene classification vectors are generated using the vehicle's historical and current front view image data through a compression module and a second temporal network encoding, including:

[0027] The compression module compresses historical front view images and current front view images for a predetermined time period into a compressed vector of a predetermined length.

[0028] The compressed vectors are input into corresponding time sequence units in the second time sequence network in time sequence, and information of the compressed vectors from the earliest time to the latest time is collected and combined through the second time sequence network to generate the scene classification vector of the vehicle.

[0029] In some embodiments, the future trajectory planning data of the vehicle is obtained by the third time sequence network and the multi-task head using the image vector, the trajectory vector and the scene classification vector, including:

[0030] The third time sequence network is used to iteratively generate a latent state vector sequence by combining the image vector and the trajectory vector.

[0031] The scene adaptive relative displacement of the vehicle is parsed from the latent state vector sequence by the multi-task head using the scene classification vector to generate the future trajectory planning data of the vehicle.

[0032] In some embodiments, the future trajectory planning data of the vehicle is obtained by the third time sequence network and the multi-task head using the image vector, the trajectory vector and the scene classification vector, including:

[0033] The third time sequence network is used to iteratively generate a latent state vector sequence by combining the image vector and the trajectory vector, and the latent state vector sequence contains m latent state vectors, where m is an integer greater than 1.

[0034] The relative displacement, target detection information and semantic segmentation information are parsed from each latent state vector by the multi-task head using the scene classification vector.

[0035] The future trajectory planning data of the vehicle is obtained by combining the relative displacement, target detection information and semantic segmentation information of each latent state vector in time sequence.

[0036] In some embodiments, the first time sequence network, the feature extraction network, the compression module, the second time sequence network, the third time sequence network and the multi-task head form a vehicle trajectory planning model, the shared parameters of the vehicle trajectory planning model include the parameters of the first time sequence network, the parameters of the feature extraction network, the parameters of the compression module, the parameters of the second time sequence network and the parameters of the third time sequence network, the task head parameters of the vehicle trajectory planning model include the parameters of each task head in the multi-task head, and the vehicle trajectory planning model is obtained by the following steps: performing first stage training on the vehicle trajectory planning model to update the shared parameters and the task head parameters of the vehicle trajectory planning model at the same time; fixing the task head parameters obtained in the first stage training, performing second stage training on the vehicle trajectory planning model to update the shared parameters obtained in the first stage training; fixing the shared parameters obtained in the second stage training, performing third stage training on the vehicle trajectory planning model to update the task head parameters obtained in the first stage training.

[0037] In some embodiments, the convergence condition of the first stage training is that the average value of the first-order loss of the multi-task head is less than a preset simultaneous update threshold; and / or, the convergence condition of the second stage training is that the average value of the first-order loss of the multi-task head is less than a preset shared update threshold; and / or, the first-order loss of each task head in the multi-task head is less than a preset corresponding task update threshold.

[0038] According to a second aspect of the present disclosure, a vehicle trajectory planning device is provided, comprising:

[0039] a trajectory vector encoding unit configured to generate a trajectory vector of the ego vehicle by first temporal network encoding using historical trajectory data of the ego vehicle;

[0040] an image vector encoding unit configured to generate an image vector of the ego vehicle by feature extraction network encoding using current front-view image data of the ego vehicle;

[0041] a scene classification vector encoding unit configured to generate a scene classification vector by compression module and second temporal network encoding using historical front-view image data and current front-view image data of the ego vehicle, the scene classification vector containing feature information of a current scene in which the ego vehicle is located;

[0042] a trajectory determination unit configured to obtain future trajectory planning data of the ego vehicle by third temporal network and multi-task head using the image vector, the trajectory vector and the scene classification vector.

[0043] According to a third aspect of the present disclosure, an electronic device is provided, comprising:

[0044] a memory storing execution instructions; and

[0045] a processor executing the execution instructions stored in the memory, so that the processor executes the vehicle trajectory planning method described above.

[0046] According to a fourth aspect of the present disclosure, a readable storage medium is provided, the readable storage medium stores execution instructions, the execution instructions are executed by a processor to implement the vehicle trajectory planning method described above.

[0047] The embodiments of the present disclosure can realize scene-adaptive vehicle trajectory planning, without high-cost high-precision maps and complex manually designed vectorization rules, and have good robustness, high accuracy and good real-time performance in different scenes. BRIEF DESCRIPTION OF DRAWINGS

[0048] The accompanying drawings illustrate exemplary embodiments of the present disclosure and together with the general description of the present disclosure given above, and the detailed description of the embodiments given below, serve to explain the principles of the present disclosure.

[0049] Figure 1 is a structural schematic diagram of a vehicle trajectory planning model according to some embodiments of the present disclosure.

[0050] Figure 2 is a flow schematic diagram of a vehicle trajectory planning method according to some embodiments of the present disclosure.

[0051] Figure 3 is a process schematic diagram of generating a trajectory vector through a first temporal network according to some embodiments of the present disclosure.

[0052] Figure 4 is a process schematic diagram of generating an image vector through encoding according to some embodiments of the present disclosure.

[0053] Figure 5 is a process schematic diagram of generating a scene classification vector through encoding according to some embodiments of the present disclosure.

[0054] Figure 6 is a process schematic diagram of generating a latent state vector sequence through a third temporal network according to some embodiments of the present disclosure.

[0055] Figure 7 is a process schematic diagram of obtaining future trajectory planning data through a multi-task head combining a scene classification vector and a latent state vector sequence according to some embodiments of the present disclosure.

[0056] Figures 8a-8c is a visualization example diagram of future trajectory planning data, target detection information and semantic segmentation information according to some embodiments of the present disclosure.

[0057] Figure 9 is a training flow schematic diagram of a vehicle trajectory planning model according to some embodiments of the present disclosure.

[0058] Figure 10 is a structural schematic block diagram of a vehicle trajectory planning apparatus with a hardware implementation of a processing system according to one embodiment of the present disclosure. DETAILED DESCRIPTION

[0059] The present disclosure will be described in further detail below with reference to the drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the related content, and not to limit the present disclosure. In addition, it should be noted that only parts related to the present disclosure are shown in the drawings for ease of description.

[0060] It should be noted that the embodiments and features in the embodiments of the present disclosure can be combined with each other without conflict. The technical solutions of the present disclosure will be described in detail below with reference to the drawings and in combination with the embodiments.

[0061] Unless otherwise stated, the exemplary implementations / examples shown are to be understood as exemplary features illustrating various details of some ways in which the inventive concepts of the present disclosure can be implemented in practice. Thus, unless otherwise stated, features of the various implementations / examples can additionally be combined, separated, interchanged, and / or rearranged, without departing from the inventive concepts of the present disclosure.

[0062] The use of cross-hatching and / or shading in the drawings is generally used to make the boundaries and regions of adjacent components more clearly understood. As such, unless specifically stated otherwise, the presence of cross-hatching or shading in a drawing is not a requirement for the presence of the particular material, material property, size, proportion, commonality of the illustrated components, and / or any other feature, attribute, property, etc. of the components between adjacent components. Moreover, the dimensions and the relative dimensions of the various components shown in the drawings can be exaggerated, for clarity and / or descriptive purposes and are presented to provide some scale to the illustrations. When an exemplary embodiment is described as implementing a process in a certain order, the process can be implemented in a different order without departing from the inventive concepts of the present disclosure. For example, two processes described in succession can be performed at substantially the same time or in reverse order. Furthermore, identical reference numerals can be used to designate identical components that are common through the figures.

[0063] When a component is referred to as being “on” or “over” another component, “connected to” or “coupled to” another component, it can be directly on, directly connected to, or directly coupled to the other component, or one or more intervening components can be present. However, when a component is referred to as being “directly on,” “directly connected to,” or “directly coupled to” another component, there are no intervening components present. To that extent, the term “connected” can refer to a physical or electrical connection, whether or not with intervening components.

[0064] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. Further, when the terms “comprises,” “comprising,” “includes,” “including,” and / or “contains,” “containing,” are used, it is understood that “comprises,” “comprising,” “includes,” “including,” and / or “contains,” “containing,” are open-ended terms that do not exclude additional, unrecited elements or method steps. It is also noted that as used herein, the terms “substantially,” “approximately,” and other like terms are used as terms of approximation and not as terms of degree, such that they are utilized to account for inherent variation in measurement, calculation, and / or provision of a value that would be recognized by those of ordinary skill in the art.

[0065] The related art is briefly described as follows:

[0066] Technical solution 1: The Chinese patent with publication number CN110083160 discloses a robot trajectory planning method based on deep learning. It first establishes a kinematic model of the robot and gives the robot a basic planning trajectory, and then uses Q-learning reinforcement learning to obtain the optimal planning trajectory. This technical solution is aimed at the robot motion scene, not the autonomous driving scene, and needs to establish a kinematic model, which is complex. The neural network used is only a basic deep neural network, not a time series neural network, and does not specifically analyze the information of vehicle position changes at different time points in historical trajectories from the time dimension.

[0067] Technical solution 2: The Chinese patent with publication number CN113033902 discloses an automatic driving lane-changing trajectory planning method based on improved deep learning. It first trains several BP neural networks to replace the activation function in the classic BP network, then simulates the trajectory curve of vehicle lane-changing through a polynomial curve, and executes an obstacle avoidance algorithm to obtain the optimal trajectory curve. This technical solution has limited application scenarios, only for vehicle lane-changing scenarios, cannot be applied to lane-keeping and no-lane-line scenarios, and needs to train several BP networks, which is complex and resource-intensive.

[0068] Technical solution 3: The Chinese patent with publication number CN109213147 discloses a robot obstacle avoidance trajectory planning method and system based on deep learning. It first takes images from multiple angles and inputs them into a convolutional neural network simultaneously, then outputs the q value corresponding to each action through the convolutional network, selects the maximum q value to update the pose in the simulation environment, and finally iterates until the target point is reached. This technical solution uses an industrial robot scenario and cannot be applied to an autonomous driving scenario. It also needs multiple angle images input and updates the pose in the simulation environment, which cannot directly output results based on single angle camera image input.

[0069] Technical solution 4: The paper J. Gao, C. Sun, H. Zhao, Y. Shen, D. Anguelov, C. Li, and C. Schmid. Vectornet, Encoding hd maps and agent dynamics from vectorized representation. In IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR) discloses a method of generating target object behavior prediction by representing high-definition maps and object trajectories by vectors and processing vector information through graph neural networks. This method first represents different geometric instances of high-definition maps (such as pedestrian crossings, curbs, etc.) by a series of polylines through designed rules, each segment of the polyline is a node vector, and the polyline is a collection of node vectors, and the node vector contains information such as the starting point of the line segment, the end point of the line segment, the direction, and the attribute of the line segment. The object trajectory is represented by a series of consecutive displacement vectors within a certain time range. Then the vectors are aggregated into subgraphs through graph neural networks, and the interaction information between the vectors is extracted through graph neural networks. Finally, the trajectories related to the target object are decoded through a fully connected layer to generate behavior prediction trajectories.

[0070] Technical solution 5: H. Zhao, J. Gao, T. Lan, C. Sun, B. Sapp, B. Varadarajan, Y. Shen, Y. Shen, Y. Chai, C. Schmid, C. Li, and D. Anguelov. Tnt: Target-driven trajectory prediction. In Conference on Robot Learning (CoRL) discloses a target-driven trajectory prediction method. This method first predicts the potential future targets of objects by encoding the interaction information between the environment map and the objects in the environment, then generates a series of trajectory state sequences based on the predicted series of targets, and finally estimates the probability of each trajectory and selects a certain number of high-probability trajectories to generate a trajectory prediction set by fusing similar trajectories.

[0071] The use scenarios of the above technical solutions 1, 2, and 3 do not include the general automatic driving scenario, and cannot directly provide trajectory planning results for different scenarios under automatic driving. Moreover, in addition to using a neural network, technical solutions 1, 2, and 3 also need additional steps such as Q-learning reinforcement learning, polynomial curve simulation, or simulation environment posture updating to obtain results, which is complex and low in efficiency. Technical solutions 1 and 2 do not consider simultaneously combining camera image information and historical trajectory information as input, and have poor adaptability to different environments and low robustness.

[0072] Technical solution 4 needs to first perform vectorization representation on the high-precision map based on manually designed rules, and cannot directly generate vectors through a neural network receiving sensor input. The high-precision map has high production cost, and in areas where no high-precision map data has been collected or in areas with poor signals, the vehicle cannot use the high-precision map, and the manually designed vectorization rules are complex and low in efficiency.

[0073] Technical solution 5 simultaneously outputs the entire trajectory state in the prediction time range through a fully connected layer in the step of generating a series of trajectory state sequences based on a series of predicted targets, which assumes that the displacement of an object at each future time is independent, that is, the displacement at a certain future time has no relationship with the displacement at the previous time, but this does not conform to the actual situation because the object at a certain future time has kinematic relationship with the object at the previous time, and the scheme outputs multi-modal trajectory prediction, which belongs to a different field from the single output vehicle trajectory planning.

[0074] In view of this, the embodiments of the present disclosure provide a vehicle trajectory planning method, device, electronic equipment, and storage medium as follows, and the embodiments of the present disclosure provide a vehicle trajectory planning model that is high in efficiency and accuracy and can automatically distinguish and adapt to different automatic driving scenarios. The embodiments of the present disclosure only need to directly input the foreground image collected by a camera and the historical trajectory data of the vehicle into the vehicle trajectory planning model to obtain an image vector, a scene classification vector, and a trajectory vector, wherein the scene classification vector contains scene-related information of the vehicle such as whether it is raining and foggy, whether it is night driving, whether it is at a crossroads or a straight lane, and the like, and then processes the image vector, the scene classification vector, and the trajectory vector through the vehicle trajectory planning model to generate future trajectory planning data of the vehicle, thereby efficiently and accurately completing vehicle trajectory planning in various different automatic driving scenarios without high-cost high-precision maps and complex manually designed vectorization rules.

[0075] Figure 1 An exemplary structure diagram of the vehicle trajectory planning model of the embodiments of the present disclosure is shown. As shown in FIG. 1, the vehicle trajectory planning model includes an image vector generation module 101, a scene classification vector generation module 102, a trajectory vector generation module 103, and a trajectory planning module 104. Figure 1As shown, the vehicle trajectory planning model of the embodiment of the present disclosure can include a first time sequence network, a feature extraction network, a compression module, a second time sequence network, a third time sequence network, and a multi-task head. The input data of the first time sequence network is the historical trajectory data of the ego vehicle, the input data of the feature extraction network is the current front view image data of the ego vehicle, and the input data of the compression module includes the historical front view image data and the current front view image data of the ego vehicle. The compression vector output by the compression module is the input data of the second time sequence network. The trajectory vector output by the first time sequence network and the image vector output by the feature extraction network are the input data of the third time sequence network. The latent state vector sequence output by the third time sequence network and the scene classification vector output by the second time sequence network are the input data of the multi-task head. The output data of the multi-task head includes the future trajectory planning data of the ego vehicle. In addition, the output data of the multi-task head can also include detection information of targets such as other vehicles and obstacles, semantic segmentation information such as lane lines and road edges, and the like.

[0076] In some embodiments, the first time sequence network, the second time sequence network, and the third time sequence network can be, but are not limited to, a simple recurrent network, a Long Short-Term Memory (LSTM) network, a Gated Recurrent Unit (GRU) network, a recurrent neural network, a ClockworkRNN, or other types of time sequence networks.

[0077] In some embodiments, the compression module can be implemented through a fully connected layer. Those skilled in the art should understand that other neural network modules with data compression functions can also be used as the compression module, and the model structure of the compression module and its specific implementation manner are not limited in the embodiment of the present disclosure.

[0078] In some embodiments, the feature extraction network can be, but is not limited to, a convolutional neural network or other neural networks with feature extraction functions.

[0079] Figure 2 A flowchart of a vehicle trajectory prediction method according to an embodiment of the present disclosure is shown. As shown, the vehicle trajectory prediction method of the embodiment of the present disclosure can include the following steps S202-S208: Figure 2

[0080] In step S202, the trajectory vector of the ego vehicle is generated by encoding the historical trajectory data of the ego vehicle through the first time sequence network.

[0081] ​Here, the historical trajectory data of the ego vehicle refers to real historical trajectory data of the ego vehicle in a predetermined period in the past. Specifically, the historical trajectory data of the ego vehicle can include trajectory data of a real historical trajectory of the ego vehicle in a predetermined period in the past (for example, in the past 3 seconds of the current time), at least including historical position information of the ego vehicle in the predetermined period (for example, in the past 3 seconds of the current time), which can be represented by, but is not limited to, coordinates of a pre-agreed world coordinate system. The world coordinate system usually takes the direction perpendicular to the ground as the Z axis, and the plane parallel to the ground as the X-Y plane.

[0082] In step S202, real historical trajectory data of the ego vehicle in a predetermined period in the past is collected as input data of the first time sequence network, and a trajectory vector is generated by processing the input data through the first time sequence network. The trajectory vector can be used to indicate the motion state of the ego vehicle.

[0083] In some embodiments, the trajectory vector can include kinematic information of the vehicle, which can include but is not limited to acceleration change, speed change, position change, path change, etc.

[0084] The trajectory vector corresponds to the real historical trajectory one-to-one. If trajectory prediction is performed for M vehicles at the same time, the real historical trajectory data in the predetermined period can include trajectory data of M historical trajectories in the predetermined period in the past, and step S202 will obtain corresponding M trajectory vectors. Wherein, M is an integer greater than or equal to 1.

[0085] In some embodiments, an exemplary specific implementation process of step S202 can include the following steps a1-a2:

[0086] In step a1, the historical trajectory data of the ego vehicle in a predetermined period in the past (for example, in the past 3 seconds of the current time) is obtained, n historical positions in the historical trajectory data are extracted, and the n historical positions are converted to the body coordinate system at the current time, thereby obtaining a historical position sequence of the ego vehicle in the body coordinate system at the current time.

[0087] The time interval between adjacent historical positions in the n historical positions is the same. Specifically, the interval time between adjacent historical positions in the n historical positions extracted from the trajectory data can be a pre-set fixed value.

[0088] If only the motion of the vehicle on the ground is considered, only the planar coordinates (for example, the horizontal coordinates and the vertical coordinates) of the n historical positions can be converted, which can indicate the position of the vehicle on the ground. The historical position sequence of the ego vehicle in the body coordinate system at the current time obtained in this way can be represented as a two-dimensional coordinate sequence as shown in the following formula (1):

[0089] {(x1,y1),(x2,y2),(x3,y3),…,(x n ,y n )}(1)

[0090] wherein n is the length of the first time sequence network (i.e., the number of time sequence units in the first time sequence network), n is an integer greater than 1, and x i represents the horizontal coordinate of the i-th historical position in the current time body coordinate system, and y i represents the vertical coordinate of the i-th historical position in the current time coordinate system.

[0091] The coordinate information in the historical position sequence can be arranged in time sequence, i.e., (x1,y1) represents the planar coordinates of the earliest time historical position in the n historical positions in the current time body coordinate system, and (x n ,y n ) represents the planar coordinates of the latest time historical position in the n historical positions in the current time body coordinate system.

[0092] In step a2, the historical position sequence of the host vehicle in the current time body coordinate system is input into the first time sequence network with a length of n, and the information of the earliest time historical position to the latest time historical position in the historical position sequence is collected and combined through the first time sequence network, thereby generating a trajectory vector with a predetermined length (e.g., length of 256), which contains kinematic information such as acceleration change, speed change, position change, and path change of the host vehicle.

[0093] Figure 3 A process diagram for generating a trajectory vector through a first time sequence network is shown. The first time sequence network can include n time sequence units (time sequence unit 1-time sequence unit n) connected in sequence, the input data of the first time sequence unit (time sequence unit 1) is the coordinate information (x1,y1) of the earliest time historical position in the historical position sequence, the output data of the first time sequence unit (time sequence unit 1) is hidden information h1, the input data of the i-th time sequence unit (time sequence unit i) (i=2,3,…,n) includes the coordinate information (x i ,y i ) of the i-th historical position in the historical position sequence and the hidden information h i-1 output by the previous time sequence unit i-1, the hidden information h i-1 output by the previous time sequence unit (time sequence unit i-1) contains the previous historical position (i.e., the i-1-th historical position (x i-1 ,y i-1 )) and all historical positions {(x1,y1),…,(x i-2 ,y i-2the hidden information h output by the last time sequence unit (time sequence unit n) in the first time sequence network n i.e. the trajectory vector. In this way, the earliest historical position to the latest historical position in the historical position sequence can be collected and combined by the n time sequence units connected in sequence in the first time sequence network, so that the trajectory vector of the vehicle can be generated by combining the specific historical position of the vehicle and the association between the historical position and the previous historical position.

[0094] In step S204, the current front view image data of the ego vehicle is used to generate an image vector of the ego vehicle through a feature extraction network.

[0095] In some embodiments, the image vector can include road information, other vehicle information and obstacle information at the current position of the ego vehicle. For example, the road information at the current position of the ego vehicle can include, but is not limited to, the extension direction and curvature of the road at the current position of the ego vehicle (i.e. the extension direction, curvature, etc. of the lane line), the ground slope, etc. information; the other vehicle information can include, but is not limited to, the position of the front vehicle, etc. information; other obstacle information can include, but is not limited to, the category, size, position, etc. information of other obstacles other than the front vehicle.

[0096] Figure 4 An example diagram of the processing procedure of step S204 is shown. As shown in Figure 4 Step S204 can include the following steps S402-S408.

[0097] In step S402, each frame of the current front view image in the entire data set of the current front view image data is uniformly scaled to a predetermined size (e.g. 224*480 pixels).

[0098] In a specific application, the current front view image data can include data of multiple frames of current front view images, and the data of the multiple frames of current front view images constitutes the entire data set of the current front view image data.

[0099] In step S404, each frame of the scaled current front view image is normalized in the RGB channel direction.

[0100] Specifically, the average value and the standard deviation of each frame of the front view image in the entire data set can be calculated, and the following processing is performed for each pixel in each frame of the front view image: the original RGB value of the pixel is subtracted from the average value and then divided by the standard deviation to obtain a new RGB value of the pixel, thereby obtaining the normalized front view image.

[0101] In step S406, the normalized current front view image is processed by the feature extraction network to generate an image vector of a predetermined length (e.g. 256).

[0102] It should be noted that, Figure 4For an exemplary implementation of step S204, the implementation of step S204 is not limited to this in actual application.

[0103] At step S206, the historical front-view image data and the current front-view image data of the vehicle are used to generate a scene classification vector through a compression module and a second time sequence network coding.

[0104] The scene classification vector can contain feature information of the current scene of the vehicle. For example, the scene classification vector of the vehicle can contain information of whether it is raining and foggy, whether it is night driving, whether it is at a crossroad or a straight lane, whether the preceding vehicle is changing lanes or parking by the roadside, and the like. The scene classification vectors of the same vehicle in different scenes are different, and the scene classification vectors of different vehicles in the same scene are also different. Therefore, the scene classification vector can be used to identify the specific scene of the vehicle.

[0105] In some embodiments, the scene classification vector can include, but is not limited to, one or more of the following information for describing specific scene features: weather information indicating whether it is raining and foggy, geographical information indicating whether it is in an urban area or a mountainous area, information indicating the current congestion level, information indicating whether it is night driving, information indicating whether it is at a crossroad or a straight lane, information indicating whether the preceding vehicle is changing lanes, and information indicating whether the preceding vehicle is parking by the roadside.

[0106] The scene classification vector can help the vehicle trajectory planning model to understand and adapt to the specific scene in which the vehicle is located. For different scenes, even if the image vector and the trajectory vector can be the same, the final predicted trajectory data will be different because the scene classification vectors are necessarily different. Therefore, the vehicle trajectory planning with scene adaptation can be realized by the embodiments of the present disclosure.

[0107] Figure 5 An example diagram of the processing procedure of step S206 is shown. As shown in Figure 5 Step S206 can include:

[0108] At step b1, the historical front-view image and the current front-view image of a predetermined period (for example, within 3s before the current time) are compressed into a vector of a predetermined length (for example, the length can be 256) through a compression module, and the vector is the compressed vector in Figure 4 , thereby generating a compressed vector set containing the compressed vector of the historical front-view image and the compressed vector of the current front-view image.

[0109] As shown in Figure 5As shown, the compression module can include p compression units (p is an integer greater than 1, and takes a preset value), and the p compression units are connected in one-to-one correspondence with the p time sequence units in the second time sequence network. The historical forward-looking images to be compressed and the current forward-looking image can be input into the corresponding compression units in time sequence for compression. As shown Figure 5 As shown, the earliest forward-looking image frame 1 can be compressed into a compressed vector 1 by the compression unit 1, the forward-looking image frame 2 after the forward-looking image frame 1 in time sequence can be compressed into a compressed vector 2 by the compression unit 2, the forward-looking image frame 3 after the forward-looking image frame 2 in time sequence can be compressed into a compressed vector 3 by the compression unit 3, and so on, and the latest forward-looking image frame p (i.e. the current forward-looking image frame) can be compressed into a compressed vector p by the compression unit p. In this way, the generated compressed vector set can include the compressed vector 1 to the compressed vector p arranged in time sequence according to the forward-looking image frames.

[0110] Step b2, inputting the compressed vectors in the compressed vector set into the corresponding time sequence units in the second time sequence network in time sequence, collecting and combining the information of the compressed vector at the earliest time to the compressed vector at the latest time in the compressed vector set by the second time sequence network, so as to generate the scene classification vector of the vehicle.

[0111] As shown Figure 5 As shown, the second time sequence network includes p time sequence units, the input data of the first time sequence unit (time sequence unit 1) is the compressed vector 1 output by the compression module 1, the output data of the first time sequence unit (time sequence unit 1) is transmitted to the second time sequence unit (time sequence unit 2), the input data of the i-th time sequence unit (time sequence unit i) (i = 2, 3, …, p) includes the compressed vector i and the output data of the previous time sequence unit (time sequence unit i-1), the output data of the previous time sequence unit implicitly includes the information of the previous compressed vector i-1 and all the compressed vectors before it {compressed vector 1, …, compressed vector i-2}, and the input data of the last time sequence unit (time sequence unit p) includes the compressed vector p and the output data of the previous time sequence unit (time sequence unit p-1), and the output data of the last time sequence unit (time sequence unit p) is the scene classification vector of the vehicle. In this way, the information of the compressed vector at the earliest time to the compressed vector at the latest time in the compressed vector set can be collected and combined by the p time sequence units connected in sequence in the second time sequence network, so that the scene classification vector of the vehicle can be generated by combining the information of the current image frame, the information of the historical forward-looking image frame and the time sequence correlation between the forward-looking image frames.

[0112] Step S208, obtaining the future trajectory planning data of the vehicle by the third time sequence network and the multi-task head using the image vector, the trajectory vector and the scene classification vector.

[0113] In some embodiments, step S208 can include steps c1-c2 as follows:

[0114] Step c1: The latent state vector sequence is generated iteratively by combining the image vector and the trajectory vector through a third temporal network;

[0115] A potential state vector sequence is used to indicate possible state changes of the vehicle within a predetermined future time period. For example, a potential state vector sequence can indicate changes in the vehicle's motion, obstacles, and / or road conditions within a predetermined future time period (e.g., 5 seconds after the current time).

[0116] The potential state vector sequence can contain various possible outcomes of trajectory planning for the vehicle within a predetermined future time period. In some implementations, the potential state vector sequence can contain m potential state vectors, each indicating a possible state of the vehicle at a specific moment within the predetermined future time period; that is, one potential state vector corresponds to a specific moment within the predetermined future time period.

[0117] Specifically, a potential state vector may include possible position changes of the vehicle at a specific future time (e.g., the vehicle's displacement relative to the previous time), information on potential obstacles (e.g., the position and size of detected obstacles such as vehicles and pedestrians), and possible road information (e.g., information related to lane lines and curbs). For example, the length of each potential state vector may be a predetermined value, such as 256.

[0118] Since image vectors contain information such as the direction of the road ahead and obstacle information, while trajectory vectors contain the vehicle's kinematic information, combining image vectors and trajectory vectors allows the vehicle to travel safely along the road and avoid obstacles while minimizing changes in vehicle motion (i.e., ensuring passenger comfort). Thus, a safe and comfortable vehicle trajectory planning result is contained within the latent state vector sequence. Therefore, the latent state vector sequence can not only predict future state changes of the vehicle during its journey, but also predict the positions of constantly changing obstacles and other targets, as well as constantly changing lane lines and road edges.

[0119] Figure 6 An example diagram illustrating the process of generating a sequence of latent state vectors using a third-order temporal network is shown. Figure 6 As shown, the third time-series network comprises m sequentially connected time-series units (m is an integer greater than 1, and m is a preset value). Figure 6The first time sequence unit (i.e., time sequence unit 1) includes the track vector and the image vector as input data, and the first time sequence unit outputs the latent state vector 1 and the hidden information 1. The i-th time sequence unit (i.e., time sequence unit i, i = 2, 3, …, m) includes the hidden information 1 output by the previous time sequence unit (i.e., time sequence unit i-1) and the image vector as input data, and the i-th time sequence unit outputs the latent state vector 2 and the hidden information 2. Similarly, the m-th time sequence unit includes the image vector and the hidden information m-1 output by the previous time sequence unit (i.e., time sequence unit m-1) as input data, and the m-th time sequence unit outputs the latent state vector m. Thus, through the processing of the image vector and the track vector by each time sequence unit in the third time sequence network, the m latent state vectors arranged in time sequence can be obtained, and the m latent state vectors constitute the latent state vector sequence. It can be seen that, by combining the kinematic information of the ego vehicle in the track vector and the information of the lane line, the curb, the obstacle and the like in the image vector, each time sequence unit in the third time sequence network can generate the latent state vector of each future specific moment, and each latent state vector represents the possible state of the ego vehicle at the corresponding future specific moment.

[0120] In step c2, the scene-adaptive relative displacement of the ego vehicle is parsed from the latent state vector sequence by the multi-task head using the scene classification vector, so as to generate the future trajectory planning data of the ego vehicle.

[0121] The relative displacement prediction task head in the multi-task head processes the latent state vector and the scene classification vector, and the scene-adaptive determined relative displacement of each moment in the future is parsed from all possible results contained in the latent state vector by the scene classification vector. The future trajectory planning data of the ego vehicle can be obtained through the relative displacement of each moment in the future.

[0122] In addition to the relative displacement, other information can also be parsed by other task heads (e.g., the target detection task head, the semantic segmentation task head and the like) in the multi-task head. The other information can include but is not limited to the detection information of the other vehicle, the road sign, the obstacle and the like, the semantic segmentation information of the lane line, the curb and the like, and the like. Thus, the future trajectory planning data of the ego vehicle can be generated in combination with the other information and the relative displacement. In specific applications, the content of the other information can be deleted or added by setting different types of task heads. Through the multi-task head, the other information can be obtained, and the understanding of the scene in the future trajectory planning can be further improved. The mutual relationship between the ego vehicle and the environment, other objects and the like is introduced in the future trajectory planning, so that the future trajectory planning data that is more consistent with the actual scene and the driving condition can be obtained.

[0123] Since one potential state vector represents all possible results of the future time (the displacement of the vehicle relative to the last time, the target detection of the vehicle and pedestrian obstacles, the segmentation results of the lane line and the curb), and the scene classification vector contains the information corresponding to the current scene, the combination of the two can analyze the relative displacement, target detection information and semantic segmentation information corresponding to the current scene at a certain future time from all possible results, and the accurate future trajectory planning data can be obtained by combining the relative displacement of each future time.

[0124] Figure 7 A processing process diagram for obtaining future trajectory planning data by combining the scene classification vector and the potential state vector sequence through the multi-task head is shown. Figure 7 In the example, the multi-task head includes three task heads, namely: a relative displacement prediction task head, a target detection task head and a semantic segmentation task head. The relative displacement prediction task head is used to predict the relative displacement of the vehicle according to the potential state vector and the scene classification vector. The target detection task head is used to perform target detection according to the potential state vector and the scene classification vector to obtain the target detection information of the vehicle. The semantic segmentation task head is used to obtain the corresponding semantic segmentation information through the potential state vector and the scene classification vector. Each potential state vector in the potential state vector sequence is connected with the scene classification vector in the channel direction of the vector to form input data and enter each task head for processing, so as to obtain the relative displacement, target detection information and semantic segmentation information corresponding to each potential state vector through each task head. Finally, the relative displacement, target detection information and semantic segmentation information corresponding to each potential state vector are combined in time sequence to obtain the future trajectory planning data of the vehicle. Figure 8a A visualization diagram of the future trajectory planning data is shown, Figure 8b A visualization diagram of the target detection information is shown, Figure 8c A visualization diagram of the semantic segmentation information is shown.

[0125] In specific applications, each task head in the multi-task head can be implemented by, for example, a fully connected layer and the like. Those skilled in the art should understand that each task head in the multi-task head can also be implemented by other neural network modules, and the present disclosure embodiments do not limit this.

[0126] The training method of the vehicle trajectory planning model provided by the present disclosure embodiments will be described in detail below.

[0127] The vehicle trajectory planning model of the present disclosure embodiments can be trained using public datasets in the field of autonomous driving, such as the nuScenes dataset.

[0128] In some embodiments, the vehicle planning model can be updated and optimized by a distributed model training method based on multi-task learning.

[0129] The parameters of the vehicle planning model that need to be updated are divided into two parts, i.e., shared parameters and multi-task head parameters. The shared parameters can include the parameters of the first time sequence network, the parameters of the feature extraction network, the parameters of the compression module, the parameters of the second time sequence network, and the parameters of the third time sequence network. The task head parameters can include the parameters of each task head in the multi-task head. In some embodiments, as shown in Figure 9 The vehicle trajectory planning model is trained through the following three stages:

[0130] In step S902, the first stage training of the vehicle trajectory planning model is performed to simultaneously update the shared parameters and the task head parameters of the vehicle trajectory planning model.

[0131] Specifically, in the first stage training of the vehicle trajectory planning model, the shared parameters and the multi-task head parameters can be simultaneously iteratively updated by the weighted sum of the cross-entropy losses of the real values and the multi-task outputs of the vehicle trajectory planning model on the selected data set until the convergence condition of the first stage is met. The convergence condition of the first stage training is that the average value of the first-order losses of the multi-task heads is less than a pre-set simultaneous update threshold.

[0132] The process of each round of parameter update in the first stage training can include: using the vehicle trajectory planning model to process the current sample in the selected data set to obtain the information of each task head corresponding to the current sample, calculating the cross-entropy loss of each task head based on the real value of the current sample and the information of each task head, calculating the weighted loss sum using the cross-entropy loss of each task head, calculating the gradient using the weighted loss sum to update the shared parameters and the task head parameters using the gradient descent method, judging whether the convergence condition of the first stage is met after the update, if the convergence condition of the first stage is met, ending the first stage training, if the convergence condition of the first stage is not met, continuing to use the next round of parameter update until the convergence condition of the first stage is met.

[0133] The weighted loss sum can be calculated by the loss function shown in the following formula (2):

[0134]

[0135] In formula (2), L total represents the weighted loss sum of the vehicle trajectory planning model, L i represents the cross-entropy loss of the i-th task head, a i represents the preset weight of the i-th task head, and k represents the number of task heads in the multi-task head. For example, Figure 7 k is 3.

[0136] For example, Figure 7For example, the information of each task head corresponding to the current sample can include: relative displacement corresponding to the current sample, detection information of targets such as other vehicles, road signs, obstacles, and the like, semantic segmentation information for road features such as lane lines and kerbs, and the like, the true value of the current sample includes the true relative position corresponding to the current sample, the true target information and the true road information, and the cross-entropy loss of each task head can include: the cross-entropy loss of the relative displacement prediction, the cross-entropy loss of the target detection, and the cross-entropy loss of the semantic segmentation.

[0137] Figure 7 In the example of the first stage training, the weighted loss sum can be calculated by a loss function shown in the following formula (3):

[0138] L total = α1×L1+ p2×L2+ α3×L3 (3)

[0139] Wherein, L total represents the weighted loss sum of the vehicle trajectory planning model, L1 represents the cross-entropy loss of the relative displacement prediction task head, i.e., the cross-entropy loss of the relative displacement prediction, L2 represents the cross-entropy loss of the target detection task head, i.e., the cross-entropy loss of the target detection, and L3 represents the cross-entropy loss of the semantic segmentation task head, i.e., the cross-entropy loss of the semantic segmentation, and α1, α2 and α3 are preset weights of the above three task heads, respectively.

[0140] Specifically, in the process of each round of parameter update in the first stage training, the first-order loss average value of the multi-task head can be calculated by the following formula (4) whether the first-order loss average value is less than the preset simultaneous update threshold value, if yes, the convergence is ended, and the training is ended, otherwise, the iteration is continued.

[0141]

[0142] Wherein, L j represents the first-order loss of the jth task head, represents the first-order loss average value of the multi-task head, and k represents the number of task heads in the multi-task head. L j can be obtained by subtracting the information output by the jth task head from the true value thereof and taking the absolute value.

[0143] Through the first stage training, the vehicle trajectory planning model can have a basic understanding of the process of generating vectors for extracting images and trajectory features to perform multi-task output. After this step is completed, the vehicle trajectory planning model can output rough trajectory planning results and other information.

[0144] In step S904, the parameters of the task heads obtained in the first stage training are fixed, and the second stage training is performed on the vehicle trajectory planning model to update the shared parameters obtained in the first stage training.​

[0145] Specifically, in the second stage training of the vehicle trajectory planning model, the task head parameters obtained in the first stage training are fixed, and the shared parameters are iteratively updated on the selected data set through the weighted sum of the cross-entropy losses of the true values and the multi-task outputs of the vehicle trajectory planning model until the convergence condition of the second stage is met. The convergence condition of the second stage training is that the average value of the first-order losses of the multi-task heads is less than the preset shared update threshold.

[0146] The update process of the shared parameters in each round of the second stage training is the same as the process of updating the parameters in each round in step S902, except that only the shared parameters are updated in the second stage, the task head parameters remain the values obtained in the first stage training, and whether the second stage training converges depends on whether the average value of the first-order losses of the multi-task heads is less than the preset shared update threshold. The average value of the first-order losses of the multi-task heads can still be calculated by formula (4), and the weighted loss sum in each round of parameter update can still be obtained by formula (2).

[0147] Through the second stage training, the feature extraction capability and the scene understanding and adaptation capability of the vehicle trajectory planning model can be enhanced. After this step is completed, the output effects of the vehicle trajectory planning model on different tasks are all good, and the robustness is high.

[0148] In step S906, the shared parameters obtained in the second stage training are fixed, and the third stage training is performed on the vehicle trajectory planning model to update the task head parameters obtained in the first stage training.

[0149] Specifically, in the second stage training of the vehicle trajectory planning model, the shared parameters obtained in the second stage training are fixed, and the task head parameters are iteratively updated on the selected data set through the weighted sum of the cross-entropy losses of the true values and each task head of the vehicle trajectory planning model until the convergence condition of the third stage is met. The convergence condition of the third stage training can be that the first-order loss of each task head in the multi-task head is less than the preset corresponding task update threshold.

[0150] The update process of the shared parameters in each round of the second stage training is the same as the process of updating the parameters in each round in step S902, and the weighted loss sum in each round of parameter update can still be obtained by formula (2), except that only the task head parameters are updated in the third stage, the shared parameters are fixed as the values obtained in the second stage training, and whether the third stage training converges depends on whether the first-order loss of each task head is less than the preset corresponding task update threshold, that is, the first-order loss of each task head in the multi-task head is less than the preset task update threshold of the task head.

[0151] Through the third stage training, the performance of the multi-task head can be further refined, and after the step is completed, the accuracy of the outputs of different tasks of the vehicle trajectory planning model is high, and a very accurate trajectory planning result can be obtained.

[0152] The vehicle trajectory planning method of the embodiment of the present disclosure takes the front view image collected by the front view camera loaded on the vehicle and the historical trajectory of the vehicle as input, does not need a high-precision map, automatically encodes to generate an image vector, a scene classification vector and a trajectory vector, the scene classification vector can be used to identify the current scene of the vehicle (such as whether it is rainy and foggy, whether it is night driving, a crossroad or a straight lane, whether the front vehicle changes lanes or parks on the side, etc.), and then the image vector and the trajectory vector are combined to iteratively generate a series of potential state vectors in a predetermined period of time in the future, and finally the relative displacement and other information at each time in the future are obtained based on the series of potential state vectors and the scene classification vector, and finally the future trajectory planning data of the vehicle is obtained through the relative displacement and other information at each time in the future.

[0153] The vehicle trajectory planning method of the embodiment of the present disclosure can achieve the following beneficial effects:

[0154] 1) can automatically adapt to different automatic driving scenes, obtain scene-adaptive vehicle trajectory planning results, has high accuracy and strong robustness;

[0155] 2) vectorization of input information and vector calculation can improve operation efficiency, speed is fast, real-time trajectory planning can be realized under high vehicle speed;

[0156] 3) without high-cost high-precision maps, can also be used in areas with weak signals and no signals;

[0157] 4) while obtaining future trajectory planning data, other information such as target detection information and semantic segmentation information can also be obtained, which deepens the understanding of the scene and makes the model understand the mutual relationship between the vehicle and the environment and other objects, the other information not only makes the vehicle trajectory planning more accurate, but also the other information can be applied to subsequent other automatic driving tasks.

[0158] Figure 10 is a structural schematic block diagram of a vehicle trajectory planning device of an embodiment of the present disclosure in a hardware implementation of a processing system.

[0159] The apparatus can include respective means for performing each or several of the steps of the above-described flowcharts. Hence, each step or several of the steps of the above-described flowcharts can be performed by a respective means, and the apparatus can include one or more of these means. A means can be one or more hardware modules specially configured to carry out a respective step, or implemented by a processor configured to carry out the respective step, or stored within a computer-readable medium for implementation by a processor, or by some combination.

[0160] The hardware structure can be implemented using a bus architecture. The bus architecture can include any number of interconnecting buses and bridges, depending on the specific application of the hardware and overall design constraints. Bus 1100 connects various circuits including one or more processors 1200, memory 1300, and / or hardware modules together. Bus 1100 can also connect various other circuits 1400 such as peripheral devices, voltage regulators, power management circuits, external antennas, etc.

[0161] Bus 1100 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one connection line is shown in the figure, but it does not mean that there is only one bus or one type of bus.

[0162] Any process or method described in flowcharts or otherwise described herein can be understood as representing code, for example, executable instructions, of one or more modules, segments, or portions for implementing the specific logic functions or steps of the process, and the scope of preferred embodiments of the present disclosure includes additional implementation in which the steps are performed in an order different from that shown or discussed, including substantially simultaneously, or in reverse order, according to the functions involved, which should be understood by those skilled in the art to which the embodiments of the present disclosure belong. The processor performs the various methods and processes described above. For example, the method embodiments in the present disclosure can be implemented as a software program tangibly embodied in a machine-readable medium, such as a memory. In some embodiments, part or all of the software program can be loaded and / or installed via the memory and / or communication interface. When the software program is loaded into the memory and executed by the processor, one or more steps of the above-described methods can be performed. Alternatively, in other embodiments, the processor can be configured to perform one of the above-described methods by any other appropriate means, for example, by means of firmware.

[0163] The logic and / or steps represented in the flowcharts and / or described herein can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor- containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions.

[0164] For purposes of this specification, a "computer-readable medium" can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The computer-readable medium can specifically include the following, which are non- exhaustive list: an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system or apparatus; a portable computer diskette (magnetic); a random access memory (RAM); a read-only memory (ROM); an erasable programmable read-only memory (EPROM or Flash memory); optical fiber; and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can even be paper or other suitable medium upon which the program is printed, as the program can be electronically captured, for example via an optical scanner, then compiled, interpreted, or otherwise processed in the electronic manner into a useable form.

[0165] It should be understood that portions of the present disclosure can be implemented with hardware, software, or a combination thereof. In the embodiments described above, multiple steps or methods can be implemented in software stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any of the following technologies, known in the art, or a combination thereof, can be used: discrete logic circuitry having logic gates for implementing logic functions upon an application of data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), and the like.

[0166] Those of ordinary skill in the art can understand that all or part of the steps of the above-mentioned embodiment methods can be completed by programs instructing relevant hardware, and the programs can be stored in a readable storage medium. When the programs are executed, they include one of the steps of the method embodiments or a combination thereof.

[0167] In addition, each functional unit in various embodiments of the present disclosure can be integrated in one processing module, or each unit can be physically present separately, or two or more units can be integrated in one module. The integrated module can be realized in the form of hardware or in the form of a software functional module. When the integrated module is realized in the form of a software functional module and sold or used as an independent product, it can also be stored in a readable storage medium. The storage medium can be a read-only memory, a magnetic disk or an optical disk, etc.

[0168] Figure 10 is a schematic structural diagram of a vehicle trajectory planning device according to an embodiment of the present disclosure. As shown in Figure 10 , the vehicle trajectory planning device 1000 according to the present disclosure can include:

[0169] a trajectory vector encoding unit 1002 configured to generate a trajectory vector of the ego vehicle by first time sequence network encoding using historical trajectory data of the ego vehicle;

[0170] an image vector encoding unit 1004 configured to generate an image vector of the ego vehicle by feature extraction network encoding using current front-view image data of the ego vehicle;

[0171] a scene classification vector encoding unit 1006 configured to generate a scene classification vector containing feature information of a current scene in which the ego vehicle is located by compression module and second time sequence network encoding using historical front-view image data and current front-view image data of the ego vehicle;

[0172] a trajectory determination unit 1008 configured to obtain future trajectory planning data of the ego vehicle by third time sequence network and multi-task head using the image vector, the trajectory vector and the scene classification vector.

[0173] In some embodiments, the trajectory vector encoding unit 1002 is configured to: obtain historical trajectory data of the ego vehicle in a predetermined time period in the past, extract n historical positions in the historical trajectory data, convert the n historical positions to a vehicle body coordinate system at a current time to obtain a historical position sequence of the ego vehicle in the vehicle body coordinate system at the current time, n being an integer greater than 1; and input the historical position sequence of the ego vehicle in the vehicle body coordinate system at the current time into a first time sequence network with a length of n, and collect and combine information from the earliest historical position to the latest historical position in the historical position sequence by the first time sequence network to generate the trajectory vector.

[0174] In some embodiments, the image vector encoding unit 1004 is configured to: uniformly scale each frame of the current front-view image data; normalize each frame of the scaled current front-view image in the RGB channel direction, respectively; and process each frame of the normalized current front-view image through a feature extraction network to generate an image vector of a predetermined length.

[0175] In some embodiments, the trajectory determination unit 1008 is configured to: iteratively generate a sequence of latent state vectors by combining the image vector and the trajectory vector through a third temporal network, the sequence of latent state vectors including m latent state vectors, m being an integer greater than 1; parse relative displacement, target detection information and semantic segmentation information from each latent state vector through a multi-task head using the scene classification vector; and combine the relative displacement, the target detection information and the semantic segmentation information of each latent state vector in time sequence to obtain the future trajectory planning data of the ego vehicle.

[0176] The present disclosure also provides an electronic device, including: a memory storing execution instructions; and a processor or other hardware module executing the execution instructions stored in the memory, so that the processor or other hardware module executes the vehicle trajectory planning method described above.

[0177] The present disclosure also provides a readable storage medium, which stores execution instructions, and the execution instructions are executed by a processor to implement the vehicle trajectory planning method described above.

[0178] In the description of the present specification, the description of the terms "one embodiment / way", "some embodiments / ways", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment / way or example are included in at least one embodiment / way or example of the present application. In the present specification, the illustrative description of the above terms is not necessarily the same embodiment / way or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in one or more embodiments / ways or examples. In addition, the person skilled in the art can combine and combine the different embodiments / ways or examples described in the present specification and the features of the different embodiments / ways or examples, without contradiction.

[0179] In addition, the terms "first", "second" are only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "a plurality of" is at least two, for example, two, three, etc., unless otherwise specifically limited.

[0180] Those skilled in the art will understand that the above-described embodiments are merely intended to clarify the present disclosure, and are not intended to limit the scope of the present disclosure. Other changes or modifications can be made by those skilled in the art based on the above disclosure, and the changes or modifications are still within the scope of the present disclosure.

Claims

1. A vehicle trajectory planning method, characterized by, The method comprises the following steps: generating a trajectory vector of the ego vehicle by a first time sequence network based on historical trajectory data of the ego vehicle; generating an image vector of the ego vehicle by a feature extraction network based on current front-view image data of the ego vehicle; generating a scene classification vector by a compression module and a second time sequence network based on historical front-view image data and current front-view image data of the ego vehicle, the scene classification vector containing feature information of a current scene in which the ego vehicle is located; obtaining future trajectory planning data of the ego vehicle by a third time sequence network and a multi-task head based on the image vector, the trajectory vector and the scene classification vector, wherein the future trajectory planning data comprises a sequence of potential state vectors generated by the third time sequence network based on the image vector and the trajectory vector, the sequence of potential state vectors containing m potential state vectors, m being an integer greater than 1; obtaining relative displacement, target detection information and semantic segmentation information from each potential state vector based on the scene classification vector by the multi-task head; and combining the relative displacement, the target detection information and the semantic segmentation information of each potential state vector in time sequence to obtain the future trajectory planning data of the ego vehicle.

2. The vehicle trajectory planning method of claim 1, wherein, The method of generating the trajectory vector of the ego vehicle by the first time sequence network based on the historical trajectory data of the ego vehicle comprises the following steps: obtaining historical trajectory data of the ego vehicle in a predetermined period of time in the past, extracting n historical positions in the historical trajectory data, and converting the n historical positions to a vehicle body coordinate system at a current time to obtain a sequence of historical positions of the ego vehicle in the vehicle body coordinate system at the current time, n being an integer greater than 1; inputting the sequence of historical positions of the ego vehicle in the vehicle body coordinate system at the current time into a first time sequence network with a length of n, and collecting and combining information of the earliest historical position to the latest historical position in the sequence of historical positions by the first time sequence network to generate the trajectory vector.

3. The vehicle trajectory planning method according to claim 1 or 2, characterized by, The trajectory vector comprises one or more of the following kinematic information: acceleration change, speed change, position change, path change.

4. The vehicle trajectory planning method of claim 1, wherein, The image vector comprises road information, other vehicle information and obstacle information at a current position of the ego vehicle.

5. The vehicle trajectory planning method of claim 1, wherein, The method of generating the image vector of the ego vehicle by the feature extraction network based on the current front-view image data of the ego vehicle comprises the following steps: uniformly scaling each frame of the current front-view image data; normalizing each frame of the scaled current front-view image data in the RGB channel direction; processing each frame of the normalized current front-view image data by the feature extraction network to generate an image vector with a predetermined length.

6. The vehicle trajectory planning method of claim 1, wherein, The scene classification vector comprises one or more of the following information for describing specific scene features: weather information indicating whether it is rainy and foggy; geographical information indicating whether it is located in an urban area or a mountainous area; information indicating the current congestion level; information indicating whether it is night driving; information indicating whether it is at a crossroads or a straight lane; information indicating whether the preceding vehicle changes lanes; information indicating whether the preceding vehicle is parked by the side of the road.

7. The vehicle trajectory planning method of claim 1, wherein, The method of generating the scene classification vector by the compression module and the second time sequence network based on the historical front-view image data and the current front-view image data of the ego vehicle comprises the following steps: compressing the historical front-view image data in a predetermined period of time in the past and the current front-view image data into a compression vector with a predetermined length by the compression module; The compressed vectors are input into corresponding time sequence units in the second time sequence network in time sequence, and information of the compressed vectors from the earliest time to the latest time is collected and combined through the second time sequence network to generate the scene classification vector of the vehicle.

8. The vehicle trajectory planning method of claim 1, wherein, The future trajectory planning data of the vehicle is obtained by the third time sequence network and the multi-task head using the image vector, the trajectory vector and the scene classification vector, including: The latent state vector sequence is iteratively generated by the third time sequence network combining the image vector and the trajectory vector; The scene adaptive relative displacement of the vehicle is parsed from the latent state vector sequence by the multi-task head using the scene classification vector to generate the future trajectory planning data of the vehicle.

9. The vehicle trajectory planning method of claim 1, wherein, The shared parameters of the vehicle trajectory planning model include the parameters of the first time sequence network, the parameters of the feature extraction network, the parameters of the compression module, the parameters of the second time sequence network and the parameters of the third time sequence network, and the task head parameters of the vehicle trajectory planning model include the parameters of each task head in the multi-task head. The vehicle trajectory planning model is obtained by the following steps: The first stage training is performed on the vehicle trajectory planning model to update the shared parameters and the task head parameters of the vehicle trajectory planning model at the same time; The task head parameters obtained in the first stage training are fixed, and the second stage training is performed on the vehicle trajectory planning model to update the shared parameters obtained in the first stage training; The shared parameters obtained in the second stage training are fixed, and the third stage training is performed on the vehicle trajectory planning model to update the task head parameters obtained in the first stage training.

10. The vehicle trajectory planning method of claim 9, wherein The convergence condition of the first stage training is that the average value of the first-order loss of the multi-task head is less than a pre-set simultaneous updating threshold; and / or The convergence condition of the second stage training is that the average value of the first-order loss of the multi-task head is less than a pre-set shared updating threshold; and / or The first-order loss of each task head in the multi-task head is less than a corresponding task updating threshold.

11. A vehicle trajectory planning device characterized by comprising: including: The trajectory vector encoding unit is configured to encode the historical trajectory data of the vehicle by the first time sequence network to generate the trajectory vector of the vehicle; The image vector encoding unit is configured to encode the current front-view image data of the vehicle by the feature extraction network to generate the image vector of the vehicle; The scene classification vector encoding unit is configured to encode the historical front-view image data and the current front-view image data of the vehicle by the compression module and the second time sequence network to generate the scene classification vector, which contains the feature information of the current scene of the vehicle; The trajectory determination unit is configured to obtain the future trajectory planning data of the vehicle by the third time sequence network and the multi-task head using the image vector, the trajectory vector and the scene classification vector, including iteratively generating the latent state vector sequence by the third time sequence network combining the image vector and the trajectory vector, wherein the latent state vector sequence contains m latent state vectors, and m is an integer greater than 1. The relative displacement, target detection information and semantic segmentation information are parsed from each potential state vector by a multi-task head using a scene classification vector; and the future trajectory planning data of the vehicle is obtained by combining the relative displacement, target detection information and semantic segmentation information of each potential state vector in time sequence.

12. An electronic device, comprising: Comprise: a memory storing execution instructions; and a processor executing the execution instructions stored in the memory, so that the processor executes the vehicle trajectory planning method in any one of claims 1 to 10.

13. A readable storage medium, characterized by, The readable storage medium stores execution instructions, and the execution instructions are executed by the processor to implement the vehicle trajectory planning method in any one of claims 1 to 10. The readable storage medium stores execution instructions, and the execution instructions are executed by the processor to implement the vehicle trajectory planning method in any one of claims 1 to 10.

Images