Vehicle driving track prediction method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses a vehicle driving track prediction method and device, electronic equipment and a storage medium, and the method comprises the steps: determining a bird's-eye view feature map sequence of an environment where a vehicle is located based on a look-around image sequence collected by the vehicle within a first preset time period before a current time point; predicting dynamic obstacles in a second preset time period after the current time point based on the aerial view feature map sequence to obtain a dynamic obstacle occupancy map sequence; based on the aerial view feature map sequence, determining a drivable area occupation map and static environment features of the vehicle at the current time point; and predicting a driving track of the vehicle based on the dynamic obstacle occupancy map sequence, the drivable area occupancy map and the static environment characteristics. According to the embodiment of the invention, a reasonable vehicle driving track with high safety can be predicted.

Description

Technical Field

[0001] The present disclosure relates to computer vision technology and autonomous driving technology, and in particular to a method, device, electronic device and storage medium for predicting a vehicle's driving trajectory. Background Art

[0002] In the field of autonomous driving or assisted driving, the vehicle environment information is usually sensed by the on-board environment sensing device, and predictions such as collision prediction and vehicle driving state prediction are made based on the sensed environment information and vehicle parameters, and the vehicle is controlled accordingly or the driver is prompted accordingly based on the prediction results. For example, when it is predicted that the vehicle is about to collide, the vehicle is controlled to slow down and turn to avoid the collision, or when it is predicted that the vehicle is about to violate traffic rules such as speeding and crossing the solid line, the vehicle is controlled to slow down or adjust the direction of travel to avoid violations of traffic rules.

[0003] In the related technologies of autonomous driving or assisted driving, object information in the environment of the vehicle that affects the vehicle trajectory planning can be identified based on the environmental image of the vehicle's environment, and then a spatiotemporal cost graph containing time and space dimensions for vehicle trajectory planning is generated based on the object information, and then the vehicle's planned trajectory is generated based on the spatiotemporal cost graph. However, the vehicle trajectory planning method based on the spatiotemporal cost graph has a large amount of calculation to generate the spatiotemporal cost graph, resulting in low efficiency of vehicle trajectory planning. Summary of the invention

[0004] In order to solve the above technical problems, the present disclosure is proposed. The embodiments of the present disclosure provide a method, device, electronic device and storage medium for predicting a vehicle driving trajectory.

[0005] According to one aspect of an embodiment of the present disclosure, a method for predicting a vehicle driving trajectory is provided, comprising:

[0006] Determine a sequence of bird's-eye view feature images of the environment where the vehicle is located based on a sequence of surround view images collected within a first preset time period before the current time point of the vehicle;

[0007] Based on the bird's-eye view feature map sequence, predict the dynamic obstacles within a second preset time period after the current time point to obtain a dynamic obstacle occupancy map sequence;

[0008] Based on the sequence of bird's-eye view feature images, determining a drivable area occupancy map and static environment features of the vehicle at the current time point;

[0009] The driving trajectory of the vehicle is predicted based on the dynamic obstacle occupancy map sequence, the drivable area occupancy map and the static environment characteristics.

[0010] According to another aspect of the embodiments of the present disclosure, a device for predicting a vehicle driving trajectory is provided, comprising:

[0011] A bird's-eye view feature image perception module, used to determine a bird's-eye view feature image sequence of the environment where the vehicle is located based on a surround view image sequence collected within a first preset time period before the vehicle is at a current time point;

[0012] A time series prediction module, used to predict the dynamic obstacles within a second preset time period after the current time point based on the bird's-eye view feature map sequence, and obtain a dynamic obstacle occupancy map sequence; the time series prediction model is also used to determine the drivable area occupancy map of the vehicle at the current time point based on the bird's-eye view feature map sequence;

[0013] The driving route prediction module is used to determine the static environment characteristics of the vehicle at the current time point based on the bird's-eye view feature map sequence; the trajectory planning module is also used to predict the driving trajectory of the vehicle based on the dynamic obstacle occupancy map sequence, the drivable area occupancy map and the static environment characteristics.

[0014] According to another aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and the computer program is used to execute the above-mentioned vehicle driving trajectory prediction method.

[0015] According to another aspect of the embodiments of the present disclosure, an electronic device is provided, the electronic device comprising:

[0016] processor;

[0017] a memory for storing instructions executable by the processor;

[0018] The processor is used to read the executable instructions from the memory and execute the instructions to implement the above-mentioned vehicle driving trajectory prediction method.

[0019] Based on the vehicle driving trajectory prediction method, device, electronic device and storage medium provided by the above-mentioned embodiments of the present disclosure, since the surround image of the vehicle's environment includes road condition information including obstacles near the vehicle and the drivable area, based on the surround image sequence of the vehicle's environment in the historical time period (i.e., the first preset time period before the current time point), not only the static road condition information including the vehicle's drivable area and nearby obstacles at the current time point can be determined, but also the position change information of the dynamic obstacles near the vehicle in the future time period (i.e., the second preset time period after the current time point) can be determined by dynamic position prediction. Then, according to the drivable area occupancy map and static environment characteristics representing the static road condition of the vehicle, combined with the dynamic obstacle occupancy map sequence representing the position change of dynamic obstacles near the vehicle in the future time period, the dynamic road condition in the future time period can be obtained, and accordingly, a reasonable driving trajectory of the vehicle to the driving target can be quickly predicted, which helps to improve the efficiency and flexibility of trajectory generation of the vehicle in the automatic driving mode or the assisted driving mode. The disclosed embodiment abandons the reliance on trajectory sampling on the spatiotemporal cost map, and can predict the dynamic road conditions in future time periods by extracting static road condition information and predicting the positions of dynamic obstacles, thereby quickly predicting a safe and reasonable driving trajectory for the vehicle in the future time period.

[0020] The technical solution of the present disclosure is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The above and other purposes, features and advantages of the present disclosure will become more apparent by describing the embodiments of the present disclosure in more detail in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation of the present disclosure. In the drawings, the same reference numerals generally represent the same components or steps.

[0022] Figure 1 is a structural block diagram of a vision-based vehicle trajectory generation system to which the present disclosure is applicable;

[0023] Figure 2 is a flow chart of a method for predicting a vehicle driving trajectory in one embodiment of the present disclosure;

[0024] Figure 3 is a flow chart of step S4 in one embodiment of the present disclosure;

[0025] Figure 4 is a flowchart of step S4-3 in one embodiment of the present disclosure;

[0026] Figure 5 is a flowchart of step S4-4 in one embodiment of the present disclosure;

[0027] Figure 6 is a working principle diagram of an autoregressive model in one embodiment of the present disclosure;

[0028] Figure 7 is a flow chart of step S4 in another embodiment of the present disclosure;

[0029] Figure 8 is a schematic diagram of generating multiple driving trajectories for the current time point in an example of the present disclosure;

[0030] Fig. 9 is a structural block diagram of a vehicle driving trajectory prediction device in one embodiment of the present disclosure;

[0031] Fig.10 is a structural block diagram of a driving trajectory prediction module 300 in one embodiment of the present disclosure;

[0032] Fig.11 is a structural block diagram of a driving trajectory prediction module 300 in another embodiment of the present disclosure;

[0033] Fig.12 is a structural block diagram of a driving trajectory prediction module 300 in another embodiment of the present disclosure;

[0034] Fig.13 It is a structural block diagram of an electronic device in one embodiment of the present disclosure. DETAILED DESCRIPTION

[0035] To explain the present disclosure, example embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. It should be understood that the present disclosure is not limited to the example embodiments.

[0036] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.

[0037] Application Overview

[0038] In the process of implementing the present disclosure, the inventors found that when using a vehicle trajectory planning model based on spatiotemporal cost map sampling, it is necessary to identify key environmental information in the vehicle environment that affects the vehicle trajectory planning based on the environmental image of the vehicle environment. The environmental information may include fixed obstacle information (which may include the position and volume of fixed obstacles), movable obstacle information, traffic sign information, and various map lines (which may include curbs and lane lines) in the vehicle environment, etc. During vehicle driving, the positions of fixed obstacles and traffic signs remain unchanged, the positions of movable obstacles will change in real time, and the vehicle's drivable area may change, resulting in a large memory overhead for maintaining a spatiotemporal cost map that is used to assist in vehicle trajectory sampling and includes time and space dimensions.

[0039] In addition, the flexibility of trajectory morphology is limited by the performance of the pre-sampler, and indexing the cost value on the spatio-temporal cost graph based on the sampled trajectory will also result in high computational time.

[0040] Exemplary Systems

[0041] Figure 1 is a structural block diagram of a vision-based vehicle trajectory generation system applicable to the present disclosure. Figure 1 As shown, the vehicle trajectory generation system may include: an image acquisition device 1 and an electronic device 2.

[0042] Among them, the image acquisition device 1 includes multiple vehicle-mounted cameras, which are arranged at preset positions in the vehicle body, and collect environmental images of preset perspectives at the same frequency within a specified historical time period before the current timestamp, so as to facilitate multi-perspective image feature fusion in the model in the subsequent stage to achieve more comprehensive environmental information representation.

[0043] The electronic device 2 may be a vehicle-mounted central control device, or a mobile electronic device including a mobile phone and a tablet computer that is communicatively connected to the vehicle-mounted central control device, or a server that is communicatively connected to the vehicle-mounted central control device.

[0044] The electronic device 2 receives the surround image sequence of the vehicle in the historical time period (i.e., the first preset time period before the current time point) collected by the image acquisition device 1, generates a bird's-eye view feature map sequence of the vehicle's environment in the corresponding time period based on the surround image sequence, and then uses a neural network to process the bird's-eye view feature map sequence to generate a dynamic obstacle (for example, it can include vehicles, cyclists and pedestrians) occupancy map sequence for the future specified time period (i.e., the second preset time period after the current time point), a static drivable area occupancy map representing the drivable position area of ​​the vehicle in the lane at the current time point, and a static environment feature representing the static road condition feature at the current time point, and then according to the static drivable area occupancy map and static environment feature at the current time point, combined with the dynamic obstacle occupancy prediction map sequence in the future specified time period, it can quickly predict the vehicle's safe and reasonable driving trajectory in the future specified time period. The disclosed embodiment can predict the dynamic road conditions in the future time period by extracting static road condition information and predicting the position of dynamic obstacles, and can quickly predict the vehicle's safe and reasonable driving trajectory in the future time period. It should be noted that, in this embodiment, the “reasonable driving trajectory” includes: the curvature of the driving trajectory is as low as possible and the length of the driving trajectory is as short as possible.

[0045] Exemplary Methods

[0046] Figure 2 FIG. 1 is a flow chart of a method for predicting a vehicle driving trajectory in one embodiment of the present disclosure. Figure 2 As shown, the following steps are included:

[0047] S1: Determine a sequence of bird's-eye view feature images of the environment where the vehicle is located based on a sequence of surround view images collected within a first preset time period before the current time point of the vehicle.

[0048] You can use Figure 1 The image acquisition device 1 of the system periodically acquires surround view images of the vehicle's environment within a first preset time period before a current time point, and sorts the multiple surround view images acquired according to the chronological order of the acquisition time of the surround view images to obtain a surround view image sequence within the first preset time period before the current time point.

[0049] Among them, the image acquisition device 1 may include multiple vehicle-mounted cameras set at preset positions on the vehicle and with different image acquisition ranges, and image data processing equipment (which can be integrated in the vehicle's central control system or independent of the vehicle's central control system). The multiple vehicle-mounted cameras may include a front-view camera, a rear-view camera, and a side camera. Among them, the front-view camera can capture the environmental image directly in front of the vehicle, the rear-view camera can capture the environmental image directly behind the vehicle, and the side camera can capture the environmental images of the left front, left rear, right front, and right rear of the vehicle. The front-view camera, the rear-view camera, and the side cameras periodically and synchronously capture environmental images, which facilitates the subsequent model to fuse multi-view images to generate a bird's-eye view feature map.

[0050] The electronic device 2 can extract and fuse features of the surround view images at each time point according to the temporal sequence of the surround view image sequence by, for example, the LSS (Lift, Splat, Shoot) method, thereby obtaining a bird's-eye view feature map sequence of the vehicle's environment, namely, a BEV (Bird's-Eye View) feature map sequence.

[0051] In one example of the present disclosure, the first preset time period (i.e., historical time period) may be N seconds (N is a real number greater than 0, such as 1), and the acquisition period of the image acquisition device is pre-set (e.g., the acquisition period is 0.5 seconds). Assuming the current time point is t, N is 1, and the acquisition period is 0.5 seconds, the surround image sequence includes: the surround image at the (t-2) time point (i.e., 1 second before the current time point), the surround image at the (t-1) time point (i.e., 0.5 seconds before the current time point), and the surround image at the t time point. The sequence of bird's-eye feature maps obtained by feature extraction and fusion of the surround image includes: the bird's-eye feature map at the (t-2) time point, the bird's-eye feature map at the (t-1) time point, and the bird's-eye feature map at the t time point.

[0052] S2: Based on the bird's-eye view feature map sequence, predict the dynamic obstacles within a second preset time period after the current time point to obtain a dynamic obstacle occupancy map sequence.

[0053] The network model carried by the electronic device can uniformly convert the bird's-eye view feature map sequence of the first preset time period (i.e., the historical time period) to the current time point in the ego-vehicle coordinate system for feature fusion, and obtain a single-frame bird's-eye view feature map at the current time point containing historical environmental information, and then extract the position and speed feature information of the movable obstacle relative to the ego-vehicle, generate the features characterizing the position information of the dynamic obstacles in the second preset time period after the current time point (i.e., the future time period), and generate a dynamic obstacle occupancy map sequence.

[0054] In an example of the present disclosure, the second preset time period may be M seconds (M is a real number greater than 0, for example, may be 3), and assuming the current time point is t, the dynamic obstacle occupancy diagram sequence includes: the obstacle occupancy diagram at the (t+1) time point (i.e., 0.5 seconds after the current time point), the obstacle occupancy diagram at the (t+2) time point (i.e., 1 second after the current time point), the obstacle occupancy diagram at the (t+3) time point (i.e., 1.5 seconds after the current time point), the obstacle occupancy diagram at the (t+4) time point (i.e., 2 seconds after the current time point), the obstacle occupancy diagram at the (t+5) time point (i.e., 2.5 seconds after the current time point), and the obstacle occupancy diagram at the (t+6) time point (i.e., 3 seconds after the current time point).

[0055] S3: Based on the bird's-eye view feature map sequence, determine the drivable area occupancy map and static environment features of the vehicle at the current time point.

[0056] The electronic device 2 processes the bird's-eye view feature image sequence according to step S2 to obtain a fused single-frame bird's-eye view feature image at the current time point, and generates a static drivable area occupancy map of the vehicle at the current time point. In addition, the electronic device 2 can also perform image feature extraction on the fused single-frame bird's-eye view feature image at the current time point to obtain the static environmental features of the vehicle at the current time point.

[0057] It should be noted that the drivable area in the embodiment of the present disclosure is a legal lane area in which the vehicle can travel in the direction of vehicle travel, and the drivable area does not take into account the influence of movable obstacles, that is, even if there are movable obstacles in the lane area in the direction of vehicle travel, the area where the movable obstacles are located will be regarded as a part of the drivable area.

[0058] S4: Predict the vehicle's driving trajectory based on the dynamic obstacle occupancy map sequence, the drivable area occupancy map and the static environment features.

[0059] The electronic device can use a pre-trained autoregressive network model to autoregressively generate the coordinates of the vehicle's future driving trajectory points in a time step order according to the predicted dynamic obstacle occupancy map sequence and the static drivable area occupancy map, and obtain the vehicle's driving trajectory in a future time period based on the future driving trajectory point coordinates.

[0060] In this embodiment, since the surround image of the vehicle's environment includes road condition information including obstacles near the vehicle and the drivable area, based on the surround image sequence of the vehicle's environment in the historical time period (i.e., the first preset time period before the current time point), not only can the static road condition information including the vehicle's drivable area and nearby obstacles at the current time point be determined, but also the position change information of the dynamic obstacles near the vehicle in the future time period (i.e., the second preset time period after the current time point) can be determined by means of position dynamic prediction, and then according to the drivable area occupancy map representing the static road condition of the vehicle and the static environment characteristics, combined with the dynamic obstacle occupancy map sequence representing the position change of dynamic obstacles near the vehicle in the future time period, the dynamic road condition in the future time period can be obtained, and accordingly, the reasonable driving trajectory of the vehicle to the driving target can be quickly predicted, which helps to improve the efficiency and flexibility of the trajectory generation of the vehicle in the automatic driving mode or the assisted driving mode. The disclosed embodiment abandons the reliance on trajectory sampling on the spatiotemporal cost map, and the dynamic road condition in the future time period can be predicted by the static road condition information extraction plus dynamic obstacle position prediction, and then the safe and reasonable driving trajectory of the vehicle in the future time period can be quickly predicted.

[0061] Figure 3 FIG. 4 is a flow chart of step S4 in one embodiment of the present disclosure. Figure 3 As shown, step S4 includes:

[0062] S4-1: Determine a dynamic obstacle image feature sequence based on the dynamic obstacle occupancy map sequence.

[0063] You can use Figure 1 The feature extraction network in the image processing model carried by the electronic device 2 of the system shown performs feature extraction and feature compression (which may include feature dimension reduction and removal of redundant information in the features) on each frame of the dynamic obstacle occupancy map in the dynamic obstacle occupancy map sequence of multiple time steps within the second preset time period, and obtains the dynamic obstacle distribution information features of multiple time steps within the second preset time period, which are defined as a dynamic obstacle image feature sequence.

[0064] S4-2: Determine the image features of the drivable area based on the drivable area occupancy map.

[0065] The electronic device 2 can use a feature extraction network to extract features from the drivable area occupancy map to obtain drivable area image features.

[0066] S4-3: Perform feature splicing based on the drivable area image feature and the dynamic obstacle image feature sequence to obtain a spliced ​​feature sequence.

[0067] The electronic device 2 can splice the fused drivable area image features at the current time point with each frame feature in the dynamic obstacle distribution feature sequence of multiple frames within the second preset time period to obtain multiple spliced ​​features, and sort the multiple spliced ​​features according to the time step to obtain a spliced ​​feature sequence.

[0068] S4-4: Determine the vehicle's driving trajectory based on the spliced ​​feature sequence and static environment features.

[0069] The electronic device 2 can use a pre-trained autoregressive trajectory generation model to directly generate a vehicle driving trajectory by concatenating the feature sequence and the static environment features of the current time step.

[0070] In this embodiment, since the dynamic obstacle occupancy map sequence can represent the position change information of obstacles near the vehicle in the future time period (i.e., the second preset time period), the drivable area occupancy map remains unchanged in the future time period, and the static environmental features at the current time point can represent the static road conditions near the vehicle at the current time point, the trajectory prediction model is used to predict the driving trajectory of the spliced ​​features of the dynamic obstacle occupancy map sequence corresponding to the dynamic obstacle occupancy map sequence and the drivable area image features corresponding to the drivable area occupancy map, as well as the static environmental features at the current time point, so that a safe and reasonable driving trajectory of the vehicle in the future time period can be quickly predicted.

[0071] Figure 4 FIG. 4 is a flow chart of step S4-3 in one embodiment of the present disclosure. Figure 4 As shown, step S4-3 includes:

[0072] S4-3-1: Identify lane information from the surround view image sequence.

[0073] When there are lane lines in the environment where the vehicle is located within the first preset time period, the surround view image sequence includes lane line information. At this time, the following method can be used: Figure 1 The electronic device 2 shown obtains lane line information by identifying the surround image sequence. The lane line information may include lane line position and lane line attributes. The lane line attributes may include: solid line and dotted line.

[0074] S4-3-2: Based on the bird's-eye view feature map sequence and lane line information, determine the lane line occupancy map of the road section where the vehicle is located at the current time point.

[0075] The electronic device 2 can generate a lane line occupancy map representing the position of the lane center line of the drivable lane in the vehicle's driving direction based on the bird's-eye view feature map sequence and the lane line information.

[0076] S4-3-3: Determine lane line image features based on the lane line occupancy map.

[0077] The electronic device 2 may utilize a feature extraction network to perform image feature extraction on the lane line occupancy map to obtain lane line image features.

[0078] S4-3-4: Feature splicing is performed on the drivable area feature and the lane line image feature respectively with each dynamic obstacle feature in the dynamic obstacle feature sequence to obtain a spliced ​​feature sequence.

[0079] In this embodiment, since the position of the dynamic obstacle will change over time in the future time period (i.e., the second preset time period), but the position of the drivable area and the lane centerline remains unchanged, the spliced ​​feature sequence obtained by splicing the drivable area image feature, the lane line image feature and the dynamic obstacle image feature sequence can characterize the characteristics of the dynamic obstacles, the lane centerline and the drivable area in the time dimension and the space dimension in the second preset time period, combined with the static environment characteristics at the current time point, it is helpful to quickly predict a safe and reasonable vehicle driving trajectory in the future time period.

[0080] Figure 5 FIG. 4 is a flow chart of step S4-4 in one embodiment of the present disclosure. Figure 5 As shown, step S4-4 includes:

[0081] S4-4-1: Extracting the splicing features of the first predicted time point within the second preset time period from the splicing feature sequence.

[0082] Figure 6 FIG. 1 is a diagram showing the working principle of an autoregressive model in one embodiment of the present disclosure. Figure 6 As shown, let the current time point be t, the first predicted time point be t+1, and the time points after the first predicted time point be t+2, ..., t+N, where N is an integer greater than 2.

[0083] You can use Figure 1 The electronic device 2 of the system shown extracts the splicing feature at time point t+1 from the splicing feature sequence.

[0084] S4-4-2: Obtain the linear velocity and angular velocity of the vehicle at the current time point, and obtain the first relative coordinate between the navigation position of the vehicle at the first predicted time point and the position of the vehicle at the current time point.

[0085] The electronic device 2 can collect the linear velocity vt and angular velocity wt of the vehicle at the current time point t through the vehicle's central control system, and can obtain the fixed global target position coordinates of the vehicle in the future time period at the current time point t through the navigation software (for example, it can include (x t+1 ,y t+1 ,z t+1 ),…,(x t+N ,yt+N ,z t+N The goal may include: a first relative coordinate between the navigation position at the first predicted time point and the position of the vehicle at the current time point, recorded as goal 1 .

[0086] S4-4-3: Based on the static environment characteristics, the splicing characteristics of the first predicted time point, the first relative coordinates, and the linear velocity and angular velocity of the vehicle at the current time point, generate the coordinate offset of the trajectory point of the vehicle at the first predicted time point relative to the actual position of the vehicle at the current time point.

[0087] according to Figure 6 From left to right in the figure, the electronic device 2 defines the gated recurrent unit (GRU) from the processing time point (t+1) to the processing time point (t+N) as GRU 1 To GRU N , and define the goals input from time point (t+1) to time point (t+N) as goals 1 to goal N .

[0088] Electronic devices using GRU 1 , for the static environment characteristics at the current time point t (i.e. Figure 6 The compressed feature vector of the concatenated features at time point (t+1) (i.e. Figure 6 The output of the corresponding inverted trapezoidal feature compression network at time point (t+1) and the goal in the local coordinate system of the vehicle at the current time point t 1 The relative coordinates are processed and the first trajectory point coordinate offset (Δx t+1 ,Δy t+1 ). Among them, the three-channel features of the input feature compression network at time point (t+1) respectively represent: the features of the dynamic obstacle prediction occupancy map at time point (t+1), the features of the drivable area occupancy map, and the features of the lane line occupancy map.

[0089] S4-4-4: Determine the predicted trajectory point coordinates of the vehicle at the first predicted time point based on the first trajectory coordinate offset and the current coordinates of the actual position of the vehicle at the current time point in the preset coordinate system.

[0090] The electronic device sums the first trajectory coordinate offset and the current coordinate to obtain the predicted trajectory point coordinate (x t+1 ,y t+1 ).

[0091] S4-4-5: Determine the second trajectory point coordinate offset based on the predicted trajectory point coordinates of the first predicted time point, the splicing features of the second predicted time point and the second relative coordinates, wherein the second predicted time point is the next predicted time point after the first predicted time point within the second preset time period, the second relative coordinates are the relative coordinates between the third navigation position and the second navigation position, the third navigation position is the navigation position of the vehicle at the second predicted time point, and the second trajectory point coordinate offset is the coordinate offset of the predicted position of the vehicle at the second sampling time point relative to the predicted position of the vehicle at the first predicted time point.

[0092] The processing method of the electronic device 2 for the time point (t+2) is similar to the processing method for the time point (t+1), except that the predicted trajectory point coordinates (x t+1 ,y t+1 ) replaces the linear velocity and angular velocity information of the vehicle, and the GRU 2 The predicted trajectory point coordinates (x t+1 ,y t+1 ), the concatenated features at time point (t+2) and the goal at time point (t+2) 2 Processing is performed to output the coordinate offset of the second trajectory point (Δx t+2 ,Δy t+2 ).

[0093] The electronic device 2 processes the time point (t+N) in the same way as the time point (t+N-1), and the coordinate offset (Δx t+N ,Δy t+N ).

[0094] In this embodiment, a network model including GRU can be used, for example, to process each spliced ​​feature of the spliced ​​feature sequence according to the temporal relationship of the spliced ​​feature sequence, and gradually output the coordinate offset of the latter time point relative to the previous time point within the second preset time period. Combined with the actual position of the vehicle at the current time point, a reasonable vehicle driving trajectory within the second preset time period can be quickly predicted.

[0095] In one embodiment of the present disclosure, a trajectory planning model (e.g., an autoregressive trajectory generation model) is used to determine the driving trajectory of a vehicle based on a spliced ​​feature sequence and static environmental features. The trajectory planning model is trained using input samples under the supervision of a preset trajectory objective function (e.g., which may include: an expert trajectory similarity loss function, a trajectory safety loss function, and a trajectory comfort loss function). The input samples of the trajectory planning model include a historical environmental image sequence of the vehicle, a speed parameter at the current time point, and a historical driving route.

[0096] After the trajectory planning model is pre-trained according to the process described in S4, it can be uploaded to the electronic device to run the inference process, which can specifically include the following steps:

[0097] A: Extract historical environment image sequences from input samples;

[0098] B: Generate a sequence of historical bird's-eye view feature maps based on the historical environment image sequence;

[0099] C: Extract the global environment feature vector containing historical observation information from the historical bird's-eye view feature map sequence, and then generate the dynamic obstacle occupancy map, static drivable area occupancy map and lane line occupancy map in the future time period;

[0100] D: The dynamic obstacle occupancy map sequence, the drivable area occupancy map and the lane line occupancy map in the future time period, the global environmental characteristics (i.e. the static environmental characteristics at the current time point) and the vehicle speed parameters at the current time point are input into the trajectory planning model, and the trajectory planning model gradually generates the coordinates of the vehicle's future driving trajectory points;

[0101] E: Check whether the planned trajectory formed by the gradually generated future driving trajectory point coordinates meets the preset safety and collision-free constraints;

[0102] F: If the generated trajectory satisfies the safety and collision-free constraints, the planned trajectory is directly output as the final planned trajectory; if the generated trajectory does not meet the safety and collision-free constraints, the trajectory sampler based on kinematic model deduction built into the electronic device 2 is called to evaluate each sampled trajectory, and the trajectory whose end point is closest to the given navigation target point is selected as the output on the basis of satisfying the safety constraints.

[0103] In this embodiment, when training the trajectory planning model, the coordinates of each planned trajectory point in the future time period can be gradually predicted based on the input sample, and then the planned trajectory of the input sample can be predicted. It is detected whether the predicted planned trajectory meets the preset safety and collision-free constraints. If the preset safety and collision-free constraints are met, the planned trajectory predicted by the trajectory planning model is characterized to have high safety (no vehicle collision will occur), and the planned trajectory can be directly output; if the preset safety and collision-free constraints are not met, the planned trajectory predicted by the trajectory planning model is characterized to have insufficient safety (there is a risk of vehicle collision). At this time, at least one safe driving trajectory can be predicted and evaluated based on the kinematic model trajectory sampler, and the trajectory with the terminal point closest to the given navigation target point is selected as the output on the basis of meeting the safety constraints.

[0104] In one embodiment of the present disclosure, the trajectory objective function used as supervisory information in the training phase includes three items: a trajectory similarity sub-function, a trajectory safety sub-function, and a trajectory comfort sub-function.

[0105] Among them, the trajectory similarity sub-function is a function for calculating the similarity between the predicted driving trajectory and the actual expert driving trajectory, and the predicted driving trajectory is obtained by the output of the trajectory planning model.

[0106] In one example of the present disclosure, the trajectory similarity subfunction l 1 as follows:

[0107]

[0108] Among them, ||*|| 1 with ||*|| 2 Respectively represent the l1 and l2 norms of the vector, P t represents the predicted trajectory coordinates at time t, Represents the actual trajectory coordinates at time t.

[0109] The trajectory safety sub-function is expressed as a function for calculating the collision risk value between the vehicle and the obstacle.

[0110] In one example of the present disclosure, safety is characterized by the collision probability of the remaining obstacles in the collision risk field of the vehicle (a two-dimensional Gaussian probability distribution centered at the midpoint of the rear axle of the vehicle), and the collision probability P at time t is t (x, y)~N can be written as:

[0111]

[0112] Trajectory safety subfunction l 2 as follows:

[0113]

[0114] Among them, μ represents the mean of the two-dimensional Gaussian distribution, P t (i, j) represents the two-dimensional Gaussian probability distribution centered on the vehicle at time t, S t (i, j) represents the obstacle occupancy map at time t (S represents the obstacle occupancy map, represented by binary value); the mean of the Gaussian distribution (x t ,y t ) is the predicted vehicle position coordinate at time t, and the variance parameters σxx, σyy, σxy and σyx can be set manually (for example, they can be 6, 12, 0, 0 respectively).

[0115] The trajectory comfort sub-function is a function for calculating the comfort parameter value of the vehicle, and the comfort parameter value is calculated based on the predicted driving route, historical driving parameters and driving parameter thresholds.

[0116] In one example of the present disclosure, the trajectory comfort sub-function l 3 as follows:

[0117] l 3 =k 1 *∑ t [v t -v max ] + +k 2 *∑ t ([a t -a max ] + +[a min -a t ] + )+k 3 *∑ t [j t -j max ] +

[0118] Among them, v t represents the vehicle speed at time t, v max They represent the upper speed threshold, a t represents the acceleration of the vehicle at time t, a max and a min They represent the upper and lower acceleration thresholds, respectively, t represents the jerk of the vehicle at time t, j max Indicates the upper jerk threshold, k 1 , k 2 and k 3 Respectively represent the vehicle speed coefficient, acceleration coefficient and jerk coefficient, and all are constants; [*] + If the number is negative, it will be 0, otherwise no processing will be done.

[0119] The preset trajectory objective function L is as follows:

[0120] L = αl 1 +βl 2 +γl 3

[0121] Here, α, β and γ represent the first weighting coefficient, the second weighting coefficient and the third weighting coefficient respectively, and α+β+γ=1.

[0122] In this embodiment, since a trajectory similarity sub-function that can quantify the similarity between the predicted driving trajectory and the actual driving trajectory, a trajectory safety sub-function that can quantify the safety of the predicted driving trajectory, and a trajectory comfort sub-function that can quantify the comfort of the predicted driving trajectory are set, the similarity, safety and comfort of the predicted driving trajectory and the actual driving trajectory can be evaluated based on the trajectory similarity sub-function, the trajectory safety sub-function and the trajectory comfort sub-function, which helps the trajectory planning model to predict a vehicle driving trajectory with high similarity to the actual driving trajectory, good safety and high comfort.

[0123] Figure 7 FIG. 4 is a flow chart of step S4 in another embodiment of the present disclosure. Figure 7 As shown, step S4 may include:

[0124] S4-A: Determine the planned trajectory of the vehicle based on the dynamic obstacle occupancy map sequence, the drivable area occupancy map and the static environment characteristics.

[0125] You can use Figure 1 The way in which the electronic device 2 of the system shown determines the planned route is similar to the way in which steps S4-1 to S4-4 in the above embodiment determine the driving route. The difference is that the model in this embodiment is in reasoning mode, and after the planned trajectory is generated, it is not directly used as the final driving trajectory of the vehicle. The planned trajectory needs to be detected through the following steps.

[0126] S4-B: Based on the dynamic obstacle occupancy map sequence, the vehicle's driving parameters and structural parameters, collision detection is performed on the planned trajectory to obtain the collision detection result of the planned trajectory.

[0127] The driving parameters of a vehicle may include: vehicle speed, acceleration and steering wheel angle. The structural parameters of a vehicle may include: the length and width of the vehicle, and may also include the wheelbase and minimum turning radius of the vehicle, etc.

[0128] The electronic device 2 can determine the position of each time point within the second preset time period according to the driving parameters of the vehicle, and can accurately detect whether the vehicle will collide with the dynamic obstacle at the time point in combination with the structural parameters of the vehicle and the dynamic obstacle occupancy diagram at the corresponding time point, thereby obtaining the collision detection result at each time point. The collision detection result of the planned trajectory can be generated according to the collision detection result at each time point within the second preset time period. Exemplarily, the collision detection result of the planned trajectory may include: detecting that the vehicle is about to collide within the second preset time period, or detecting that the vehicle will not collide within the second preset time period.

[0129] S4-C: Plan multiple driving trajectories for the vehicle based on the collision detection results of the planned trajectory.

[0130] When the electronic device 2 detects that the vehicle is about to collide within the second preset time period, it can plan multiple driving trajectories according to the preset driving trajectory planning rules.

[0131] S4-D: Determine at least one safe driving trajectory from multiple driving trajectories.

[0132] At least one safe driving trajectory is determined from the plurality of driving trajectories according to a preset safe driving trajectory determination rule.

[0133] S4-E: Determine a driving trajectory of the vehicle based on at least one safe driving trajectory.

[0134] A safe driving trajectory can be arbitrarily selected from at least one safe driving trajectory as the final driving route of the vehicle, or a safe driving trajectory can be selected from at least one safe driving trajectory as the final driving route of the vehicle according to a preset driving route selection rule. The final driving trajectory of the vehicle can be fitted to obtain the final driving route of the vehicle.

[0135] In this embodiment, after the planned trajectory of the vehicle is predicted by using the trajectory planning model, it is usually predicted that the planned trajectory of the vehicle has higher safety (the probability of vehicle collision is lower), but there is still a probability of vehicle collision. In order to further reduce the probability of collision when the vehicle is traveling along the final planned trajectory, after the planned route of the vehicle is generated, the planned route is subjected to collision detection, and multiple driving trajectories are planned according to the collision detection results. At least one safe driving trajectory is selected from the multiple driving trajectories, and then a safe driving trajectory is selected from the at least one safe driving trajectory to generate the final driving route of the vehicle, thereby greatly improving the safety of vehicle driving.

[0136] In one embodiment of the present disclosure, step S4-C may include: in response to the collision detection result indicating that the planned trajectory fails to be detected (i.e., fails to pass the safety collision detection), based on the planned starting position of the vehicle at the current time point and multiple sets of planned driving parameters of the vehicle, substituting into the kinematic model to forward deduce multiple driving trajectories. Among them, the multiple sets of planned driving parameters correspond to the multiple driving trajectories one by one, and each set of planned driving parameters includes: planned vehicle speed information and steering wheel angle information. The kinematic model used for forward trajectory deduction is as follows:

[0137]

[0138] Among them, (*)′ represents the derivative of *, (X,Y) represents the coordinates of the planned starting position of the vehicle, represents the rotation angle of the vehicle body relative to the world coordinate system, a represents the acceleration of the vehicle, V represents the speed of the vehicle, L r Indicates the distance from the center point of the vehicle to the rear of the vehicle, L f Indicates the distance from the center of the vehicle to the front of the vehicle.

[0139] Figure 8 FIG. 1 is a schematic diagram of generating multiple driving trajectories by sampling at the current time point in an example of the present disclosure. Figure 8 As shown, there are three drivable lanes in the driving direction of vehicle V. Figure 8From top to bottom, three lanes can be defined as the left lane, the current driving lane, and the right lane. When the collision detection of the planned trajectory fails, five corresponding driving trajectories can be generated according to the starting planned position P1 of the vehicle V and five sets of action parameters. The action parameters include: the linear velocity of the vehicle at the current time point, the fixed value of the vehicle body linear acceleration (the value of the acceleration can be 0) and the fixed value of the vehicle body angular velocity. The five driving trajectories specifically include: driving trajectories L1 and L2 for driving in the left lane, driving trajectory L3 for driving in a straight line along the current driving lane, and driving trajectories L4 and L5 for driving in the right lane. Among them, the lateral amplitude of driving trajectory L1 is greater than that of driving trajectory L2, and the lateral amplitude of driving trajectory L4 is smaller than that of driving trajectory L5.

[0140] In this embodiment, when the collision detection of the planned trajectory fails, it indicates that the vehicle may have a vehicle collision accident when driving along the planned trajectory, and the vehicle driving trajectory needs to be replanned. When replanning the vehicle driving trajectory, multiple driving trajectories can be quickly generated according to the planned starting position of the vehicle at the current time point and multiple sets of planned driving parameters of the vehicle, which helps to select a safe driving trajectory from multiple driving trajectories as the final driving trajectory of the vehicle, thereby improving the safety of the vehicle when following the final driving trajectory.

[0141] In one embodiment of the present disclosure, step S4-D may include: performing collision detection on multiple driving trajectories based on a dynamic obstacle occupancy map sequence, multiple sets of planned driving parameters and structural parameters of the vehicle to obtain collision detection results of the multiple driving trajectories; based on the collision detection results of the multiple driving trajectories, determining at least one safe driving trajectory from the multiple driving trajectories.

[0142] You can use Figure 1 After sampling and generating multiple driving trajectories, the electronic device 2 of the system performs collision detection respectively to obtain collision detection results of the multiple driving trajectories. The collision detection results of the multiple driving trajectories may include: according to each driving trajectory, the vehicle is controlled to collide with surrounding vehicles within the second preset time period or will not collide. The electronic device can classify all driving trajectories that will not collide as safe driving trajectories.

[0143] Please refer to Figure 8When judging whether any of the driving trajectories L1 to L5 is a safe driving trajectory, the electronic device 2 can determine the position of the movable obstacle A and the movable obstacle B at each time point in the second preset time period according to the dynamic obstacle occupancy diagram sequence. The electronic device can determine the position of the vehicle V at each time point in the second preset time period when driving according to the currently selected driving trajectory. The electronic device 2 can accurately judge whether the currently selected driving trajectory is a safe driving trajectory according to the position of the movable obstacle A, the position of the movable obstacle B, the position of the vehicle V, and the structural parameters of the vehicle V at each time point in the second preset time period.

[0144] In this embodiment, since the dynamic obstacle occupancy map sequence of the future time period (i.e., the second preset time period) can provide the position changes of the dynamic obstacles in the future time period, the vehicle's driving parameters (such as vehicle speed and driving direction) can characterize the vehicle's current driving state, and the vehicle's structural parameters (such as vehicle length and width) can characterize the vehicle's lane occupancy position in a bird's-eye view, multiple driving trajectories can be accurately detected for collision based on the dynamic obstacle occupancy map sequence, multiple groups of planned driving parameters, and the vehicle's structural parameters, and then at least one safe driving trajectory can be quickly and accurately determined from the multiple driving trajectories based on the collision detection results of the multiple driving trajectories.

[0145] In one embodiment of the present disclosure, step S4-E may include: determining the driving route of the vehicle from at least one safe driving trajectory based on the distance between the end trajectory point of the at least one safe driving trajectory and the target driving position of the vehicle.

[0146] Please refer to Figure 8 In the driving trajectories L1 to L5, the lateral distances between the current position P1 of the vehicle and the target driving position P2 are D1 to D5, D3<D4<D2<D1=D5. The electronic device can select a safe driving trajectory with the smallest lateral distance from the target driving position P2 from all safe driving trajectories to generate the driving route of the vehicle based on screening out the trajectory set that meets the safety constraints. For example, when the driving trajectories L2, L4 and L5 are all safe driving trajectories, since D4<D2<D5, the L4 trajectory with the closest end point to the navigation target point is further selected as the optimal trajectory output.

[0147] In this embodiment, since the distance between the end trajectory point of the safe driving trajectory and the target driving position of the vehicle can represent the deviation between the vehicle and the target driving position when the vehicle is traveling along the safe driving trajectory, a reasonable vehicle driving route can be quickly selected based on the distance between the end trajectory point of the safe driving trajectory and the target driving position of the vehicle.

[0148] In another embodiment of the present disclosure, after step S4-B, the following may also be included:

[0149] S4-F: In response to the collision detection result of the planned trajectory indicating that the planned trajectory passes the collision detection, the planned trajectory is determined as the driving trajectory of the vehicle.

[0150] You can use Figure 1 When the electronic device 2 of the system shown uses the pre-trained trajectory planning model to output the planned trajectory of the vehicle, since the output of the trajectory planning model usually has good performance in terms of trajectory similarity, trajectory safety and trajectory comfort, on the premise that the trajectory generated by the model meets the collision detection, using it as the final driving trajectory of the vehicle can basically meet the user's requirements for vehicle driving safety, comfort and human-likeness.

[0151] Any vehicle driving trajectory prediction method provided in the embodiments of the present disclosure may be executed by any appropriate device with data processing capabilities, including but not limited to: a terminal device and a server. Alternatively, any vehicle driving trajectory prediction method provided in the embodiments of the present disclosure may be executed by a processor, such as the processor executing any vehicle driving trajectory prediction method mentioned in the embodiments of the present disclosure by calling corresponding instructions stored in a memory. This will not be described in detail below.

[0152] Exemplary Devices

[0153] Fig. 9 FIG. 1 is a structural block diagram of a vehicle driving trajectory prediction device in one embodiment of the present disclosure. Fig. 9 As shown, including:

[0154] The bird's-eye view feature image perception module 100 is used to determine a bird's-eye view feature image sequence of the environment where the vehicle is located based on a surround view image sequence collected within a first preset time period before the vehicle is at a current time point;

[0155] The time series prediction module 200 is used to predict the dynamic obstacles within a second preset time period after the current time point based on the bird's-eye view feature map sequence to obtain a dynamic obstacle occupancy map sequence; the time series prediction model is also used to determine the drivable area occupancy map of the vehicle at the current time point based on the bird's-eye view feature map sequence;

[0156] The driving trajectory prediction module 300 is used to determine the static environment characteristics of the vehicle at the current time point based on the bird's-eye view feature map sequence; the driving route prediction module is also used to predict the driving trajectory of the vehicle based on the dynamic obstacle occupancy map sequence, the drivable area occupancy map and the static environment characteristics.

[0157] Fig.10 FIG. 3 is a block diagram of a driving trajectory prediction module 300 in one embodiment of the present disclosure. Fig.10 As shown, the driving trajectory prediction module 300 includes:

[0158] A first image feature determination unit 310, configured to determine a dynamic obstacle image feature sequence based on the dynamic obstacle occupancy map sequence;

[0159] A second image feature determination unit 320, configured to determine a dynamic obstacle image feature sequence based on the dynamic obstacle occupancy map sequence;

[0160] A feature stitching unit 330, configured to perform feature stitching based on the drivable area image feature and the dynamic obstacle image feature sequence to obtain a stitching feature sequence;

[0161] The driving trajectory prediction unit 340 is used to determine the driving route of the vehicle based on the spliced ​​feature sequence and the static environment feature.

[0162] In one embodiment of the present disclosure, the vehicle driving trajectory prediction device further includes:

[0163] A lane line information recognition module, used to recognize lane line information from the surround view image sequence;

[0164] The time series prediction module 200 is further used to determine the lane line occupancy map of the road section where the vehicle is located at the current time point based on the bird's-eye view feature map sequence and the lane line information.

[0165] Fig.11 FIG. 3 is a block diagram of a driving route prediction module 300 in another embodiment of the present disclosure. Fig.11 As shown, the driving route prediction module 300 further includes:

[0166] A third image feature determination unit 350, configured to determine a lane line image feature based on the lane line occupancy map;

[0167] The feature splicing unit 330 is used to perform feature splicing on the drivable area feature and the lane line image feature respectively with each dynamic obstacle feature in the dynamic obstacle feature sequence to obtain the spliced ​​feature sequence.

[0168] In one embodiment of the present disclosure, the driving route prediction unit 340 is used to extract the splicing features of the first predicted time point within the second preset time period from the splicing feature sequence;

[0169] The driving route prediction unit 340 is also used to obtain the linear velocity and angular velocity of the vehicle at the current time point, and obtain the first relative coordinates between the navigation position of the vehicle at the first predicted time point and the position of the vehicle at the current time point;

[0170] The driving route prediction unit 340 is further used to determine a first track point coordinate offset of the predicted driving position of the vehicle at the first predicted time point relative to the actual position of the vehicle at the current time point based on the static environment feature, the splicing feature of the first predicted time point, the first relative coordinate, and the linear velocity and angular velocity of the vehicle at the current time point;

[0171] The driving route prediction unit 340 is further used to determine the predicted trajectory point coordinates of the vehicle at the first predicted time point based on the first trajectory coordinate offset and the current coordinates of the actual position of the vehicle at the current time point in a preset coordinate system;

[0172] The driving route prediction unit 340 is also used to determine the second trajectory point coordinate offset based on the predicted trajectory point coordinates of the first prediction time point, the splicing features of the second prediction time point and the second relative coordinates, wherein the second prediction time point is the next prediction time point after the first prediction time point within the second preset time period, the second relative coordinates are the relative coordinates between the third navigation position and the second navigation position, the third navigation position is the navigation position of the vehicle at the second prediction time point, and the second trajectory point coordinate offset is the coordinate offset of the predicted position of the vehicle at the second sampling time point relative to the predicted position of the vehicle at the first prediction time point.

[0173] Fig.12 FIG. 3 is a structural block diagram of a driving route prediction module 300 in another embodiment of the present disclosure. Fig.12 As shown, the driving route prediction module 300 includes:

[0174] A planned route determination unit 301, configured to determine a planned trajectory of the vehicle based on the dynamic obstacle occupancy map sequence, the drivable area occupancy map and the static environment features;

[0175] A collision detection unit 302, configured to perform collision detection on the planned trajectory based on the dynamic obstacle occupancy map sequence, the driving parameters and the structural parameters of the vehicle, and obtain a collision detection result of the planned trajectory;

[0176] A driving trajectory planning unit 303, configured to plan multiple driving trajectories of the vehicle based on the collision detection results of the planned routes;

[0177] A safe driving trajectory determining unit 304, configured to determine at least one safe driving trajectory from the plurality of driving trajectories;

[0178] The driving route determination unit 305 is used to determine the driving trajectory of the vehicle based on the at least one safe driving trajectory.

[0179] In one embodiment of the present disclosure, the driving trajectory planning unit 303 is used to characterize the collision detection failure of the planned trajectory in response to the collision detection result of the planned trajectory, and determine the multiple driving trajectories based on the planned starting position of the vehicle at the current time point and multiple sets of planned driving parameters of the vehicle, wherein the multiple sets of driving parameters correspond one-to-one to the multiple driving trajectories, and each set of planned driving parameters includes planned vehicle speed information and steering wheel angle information.

[0180] In one embodiment of the present disclosure, the safe driving trajectory determination unit 304 is used to perform collision detection on the multiple driving trajectories based on the dynamic obstacle occupancy map sequence, the multiple groups of planned driving parameters and the structural parameters of the vehicle to obtain collision detection results of the multiple driving trajectories; the safe driving trajectory determination unit 304 is also used to determine the at least one safe driving trajectory from the multiple driving trajectories based on the collision detection results of the multiple driving trajectories.

[0181] In one embodiment of the present disclosure, the driving route determination unit 305 is used to determine the driving trajectory of the vehicle from the at least one safe driving trajectory based on the distance between the end trajectory point of the at least one safe driving trajectory and the target driving position of the vehicle.

[0182] In another embodiment of the present disclosure, the driving route determining unit 305 is further configured to determine the planned route as the driving route of the vehicle in response to a collision detection result of the planned route indicating that the planned route passes the collision detection.

[0183] It should be noted that the specific implementation of the vehicle driving trajectory prediction device of the embodiment of the present disclosure is similar to the specific implementation of the vehicle driving trajectory prediction method of the embodiment of the present disclosure. Please refer to the vehicle driving route prediction method part for details. In order to reduce redundancy, it will not be described in detail.

[0184] Exemplary Electronic Devices

[0185] Below, reference Fig.13 The electronic device according to the embodiment of the present disclosure is described. Fig.13 As shown, the electronic device includes one or more processors 10 and a memory 20 .

[0186] The processor 10 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0187] The memory 20 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 11 may run the program instructions to implement the vehicle driving route prediction method of each embodiment of the present disclosure described above and / or other desired functions. Various contents such as input signals, signal components, noise components, etc. may also be stored in the computer-readable storage medium.

[0188] In one example, the electronic device may further include: an input device 30 and an output device 40, which are interconnected via a bus system and / or other forms of connection mechanisms (not shown). The input device 30 may be, for example, a keyboard, a mouse, etc. The output device 40 may include, for example, a display, a speaker, a printer, and a communication network and a remote output device connected thereto, etc.

[0189] Of course, to simplify, Fig.13 Only some of the components related to the present disclosure in the electronic device are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device may further include any other appropriate components.

[0190] Exemplary computer-readable storage media

[0191] Computer readable storage media can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0192] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, and are not limitations. The above details do not limit the present disclosure to the necessity of adopting the above specific details to be implemented.

[0193] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0194] The block diagrams of the devices, apparatuses, equipment, and systems involved in this disclosure are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including," "comprising," "having," and the like are open words, referring to "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or," and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0195] The method and apparatus of the present disclosure may be implemented in many ways. For example, the method and apparatus of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above, unless otherwise specifically stated. In addition, in some embodiments, the present disclosure may also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present disclosure. Therefore, the present disclosure also covers a recording medium storing a program for executing the method according to the present disclosure.

[0196] It should also be noted that in the apparatus, device and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0197] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0198] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A method for predicting a vehicle driving trajectory, comprising: Determine a sequence of bird's-eye view feature images of the environment where the vehicle is located based on a sequence of surround view images collected within a first preset time period before the current time point of the vehicle; Based on the bird's-eye view feature map sequence, predict the dynamic obstacles within a second preset time period after the current time point to obtain a dynamic obstacle occupancy map sequence; Based on the sequence of bird's-eye view feature images, determining a drivable area occupancy map and static environment features of the vehicle at the current time point; The driving trajectory of the vehicle is predicted based on the dynamic obstacle occupancy map sequence, the drivable area occupancy map and the static environment characteristics.

2. The method according to claim 1, wherein: The predicting the driving trajectory of the vehicle based on the dynamic obstacle occupancy map sequence, the drivable area occupancy map and the static environment features includes: Determining a dynamic obstacle image feature sequence based on the dynamic obstacle occupancy map sequence; Determining image features of a drivable area based on the drivable area occupancy map; Performing feature splicing based on the drivable area image feature and the dynamic obstacle image feature sequence to obtain a spliced ​​feature sequence; Based on the splicing feature sequence and the static environment feature, the driving trajectory of the vehicle is determined.

3. The method according to claim 2, wherein: The step of performing feature splicing based on the drivable area feature and the dynamic obstacle feature sequence to obtain a spliced ​​feature sequence includes: Identifying lane line information from the surround view image sequence; Determine a lane line occupancy map of the road section where the vehicle is located at the current time point based on the bird's-eye view feature map sequence and the lane line information; Determining lane line image features based on the lane line occupancy map; The drivable area feature and the lane line image feature are respectively spliced ​​with each dynamic obstacle feature in the dynamic obstacle feature sequence to obtain the spliced ​​feature sequence.

4. The method according to claim 2 or 3, wherein: The determining the driving trajectory of the vehicle based on the splicing feature sequence and the static environment feature includes: Extracting the splicing feature of the first predicted time point within the second preset time period from the splicing feature sequence; Acquire the linear velocity and angular velocity of the vehicle at the current time point, and acquire the first relative coordinates between the navigation position of the vehicle at the first predicted time point and the position of the vehicle at the current time point; Determine a first trajectory point coordinate offset of a predicted driving position of the vehicle at the first predicted time point relative to an actual position of the vehicle at the current time point based on the static environment feature, the splicing feature of the first predicted time point, the first relative coordinate, and the linear velocity and angular velocity of the vehicle at the current time point; Determining predicted trajectory point coordinates of the vehicle at the first predicted time point based on the first trajectory coordinate offset and the current coordinates of the actual position of the vehicle at the current time point in a preset coordinate system; Based on the predicted trajectory point coordinates of the first predicted time point, the splicing features of the second predicted time point and the second relative coordinates, the second trajectory point coordinate offset is determined, wherein the second predicted time point is the next predicted time point after the first predicted time point within the second preset time period, the second relative coordinates are the relative coordinates between the third navigation position and the second navigation position, the third navigation position is the navigation position of the vehicle at the second predicted time point, and the second trajectory point coordinate offset is the coordinate offset of the predicted position of the vehicle at the second sampling time point relative to the predicted position of the vehicle at the first predicted time point.

5. The method according to any one of claims 1 to 3, wherein: The predicting the driving trajectory of the vehicle based on the dynamic obstacle occupancy map sequence, the drivable area occupancy map and the static environment features includes: Determining a planned trajectory of the vehicle based on the dynamic obstacle occupancy map sequence, the drivable area occupancy map and the static environment characteristics; Based on the dynamic obstacle occupancy map sequence, the driving parameters and the structural parameters of the vehicle, performing collision detection on the planned trajectory to obtain a collision detection result of the planned trajectory; Planning a plurality of driving trajectories of the vehicle based on the collision detection results of the planned trajectories; Determining at least one safe driving trajectory from the plurality of driving trajectories; Based on the at least one safe driving trajectory, a driving trajectory of the vehicle is determined.

6. The method according to claim 5, wherein: The step of determining at least one safe driving trajectory from the plurality of driving trajectories comprises: Based on the dynamic obstacle occupancy map sequence, the multiple sets of planned driving parameters and the structural parameters of the vehicle, performing collision detection on the multiple driving trajectories to obtain collision detection results of the multiple driving trajectories; Based on the collision detection results of the plurality of driving trajectories, the at least one safe driving trajectory is determined from the plurality of driving trajectories.

7. The method according to claim 5, wherein: The determining the driving trajectory of the vehicle based on the at least one safe driving trajectory includes: The driving trajectory of the vehicle is determined from the at least one safe driving trajectory based on a distance between an end trajectory point of the at least one safe driving trajectory and a target driving position of the vehicle.

8. A vehicle driving trajectory prediction device, comprising: A bird's-eye view feature image perception module, used to determine a bird's-eye view feature image sequence of the environment where the vehicle is located based on a surround view image sequence collected within a first preset time period before the vehicle is at a current time point; A time series prediction module, used to predict the dynamic obstacles within a second preset time period after the current time point based on the bird's-eye view feature map sequence, and obtain a dynamic obstacle occupancy map sequence; the time series prediction model is also used to determine the drivable area occupancy map of the vehicle at the current time point based on the bird's-eye view feature map sequence; A driving route prediction module is used to determine the static environment characteristics of the vehicle at the current time point based on the bird's-eye view feature map sequence; the driving route prediction module is also used to predict the driving trajectory of the vehicle based on the dynamic obstacle occupancy map sequence, the drivable area occupancy map and the static environment characteristics.

9. The device according to claim 8, wherein: The driving route prediction module comprises: a first image feature determination unit, configured to determine a dynamic obstacle image feature sequence based on the dynamic obstacle occupancy map sequence; a second image feature determination unit, configured to determine a dynamic obstacle image feature sequence based on the dynamic obstacle occupancy map sequence; A feature splicing unit, configured to perform feature splicing based on the drivable area image feature and the dynamic obstacle image feature sequence to obtain a spliced ​​feature sequence; A driving route prediction unit is used to determine the driving trajectory of the vehicle based on the spliced ​​feature sequence and the static environment features.

10. A computer-readable storage medium, wherein the storage medium stores a computer program, wherein the computer program is used to execute the method according to any one of claims 1 to 7.

11. An electronic device, comprising: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the instructions to implement the method described in any one of claims 1 to 7.

Images