Method, device and readable storage medium for predicting vehicle trajectories in urban open road scenarios

Abstract

The present invention relates to a method, device, and readable storage medium for predicting vehicle trajectories in urban open road scenarios. The method for predicting vehicle trajectories in urban open road scenarios comprises: S1, acquiring perception information; S2, storing and managing the perception information; S3, extracting high-precision map data based on the perception information; extracting perception information of all moving vehicles in the past k frames; extracting a motion state vector based on the perception information; S4, generating a lane line vector and a sidewalk vector of a target vehicle based on the high-precision map data and the motion state vector; generating a historical trajectory vector; S5, generating a minimum vector; S6, inputting a deep neural network model, and inferring and generating a predicted trajectory of the target vehicle within the next s seconds. The present invention proposes a method, device, and readable storage medium for predicting vehicle trajectories in urban open road scenarios, which have high prediction accuracy, short prediction time, and strong applicability.

Description

Technical Field

[0001] The present invention relates to the field of vehicle autonomous driving technology, and in particular to a method, device, and readable storage medium for predicting vehicle trajectories in urban open road scenarios. Background Art

[0002] Predicting vehicle trajectories is an important and fundamental problem for autonomous vehicles on open urban roads.

[0003] When a human drives a vehicle, he or she typically observes surrounding traffic participants and predicts their future positions before initiating new driving maneuvers, such as accelerating or changing lanes.

[0004] Similarly, when an autonomous vehicle is driving autonomously on open roads in a city, the traffic participants around the autonomous vehicle include moving vehicles and pedestrians. Accurately predicting the possible future trajectories of the surrounding moving vehicles and pedestrians can not only reduce the probability of danger when the autonomous vehicle performs special driving actions such as merging, changing lanes, and overtaking, but also improve the driving efficiency and comfort of the autonomous vehicle.

[0005] Whether it's a human-driven vehicle or an autonomous vehicle, encountering and avoiding moving vehicles on open urban roads is inevitable. Therefore, autonomous vehicles must be able to predict their trajectories to more effectively avoid moving vehicles and achieve autonomous driving.

[0006] In the prior art, trajectory prediction methods currently under study can be roughly divided into three categories based on their input type, output type, and intermediate processing steps.

[0007] The first type of prediction method is based on physical constraints. Since the vehicle cannot change its trajectory instantaneously due to inertia, the vehicle is represented as a dynamic entity governed by Newton's laws of physics. By linking the control state (such as wheel angle, acceleration, etc.), vehicle characteristics (such as vehicle weight) and road environment factors (such as static friction coefficient) with the evolution of vehicle state (such as position, heading, speed), dynamic models and kinematic models are used to predict the vehicle's future motion trajectory.

[0008] The second category of prediction methods is based on behavioral intention. This approach treats each vehicle as a moving individual performing a specific behavior (going straight, changing lanes, turning). The prediction process is divided into two parts: identifying the behavior and predicting the trajectory. First, the vehicle's behavior is identified (following the preceding vehicle in a straight line, turning left or right at an intersection, etc.). Then, the vehicle's future continuous physical state is predicted to ensure that the predicted trajectory corresponds to the likely behavioral intention.

[0009] The third type of prediction method based on deep learning uses the vehicle driving trajectory data and high-precision map information obtained by target detection and tracking as the input features of the model to train the deep neural network model. The trained model inference outputs the predicted vehicle's future driving trajectory.

[0010] The three aforementioned trajectory prediction methods have the following problems:

[0011] The first type of prediction method based on physical constraints only makes predictions through the vehicle's motion model, without considering the road structure, traffic rules, and interactions with other vehicles. In particular, when passing through intersections, the predicted trajectory deviation value is large.

[0012] The second category of prediction methods based on behavioral intention utilizes prior knowledge, such as vehicle status (speed, acceleration, and historical location), road geometry, speed limits, and other structured road information, but ignores a posteriori knowledge, including information about vehicle interactions and the driver's driving habits. In real-world traffic scenarios, this a posteriori knowledge (such as vehicle interactions and driver habits) influences the trajectory of vehicles with the same driving intention. Ignoring this a posteriori knowledge makes it difficult to guarantee the accuracy of the resulting trajectory.

[0013] The third type of prediction method based on deep learning is an end-to-end design based on deep neural networks. The training data used not only includes trajectory information, but also makes full use of richer environmental information, including static information of static map road network environment and perception information of dynamic targets (such as cars and people). Learning is based on the deep neural network model. As the amount of training data increases and the types of data scenarios increase, it can adapt to different driving scenarios and achieve better versatility and accuracy.

[0014] However, many deep learning-based prediction methods encode HD map information (such as lane lines and traffic lights) into color images and then use convolutional neural networks to extract lane line information from the color images. This approach has some limitations. For example:

[0015] a) The computation is large and time-consuming, resulting in poor real-time performance.

[0016] b) Convolutional neural networks tend to ignore long-distance road information, resulting in poor performance indicators for long-term trajectory prediction.

[0017] c) It is necessary to accumulate historical trajectory points for several cycles to perform predictions. Generally, 9 cycles of historical trajectories are required, which leads to poor ability to deal with emergencies. Summary of the Invention

[0018] In response to the above-mentioned problems in the prior art, the present invention proposes a method, device and readable storage medium for predicting vehicle trajectories in urban open road scenarios. It uses high-precision map data combined with the perception information obtained by the vehicle's automatic driving system to infer and generate the predicted trajectory of the target vehicle, with high accuracy, short time consumption and strong applicability.

[0019] Specifically, the present invention proposes a method for predicting vehicle trajectories in urban open road scenarios, comprising the following steps:

[0020] S1, obtains the perception information provided by the perception module in the autonomous driving system;

[0021] S2, storing and managing the perception information;

[0022] S3, extracting the center point coordinates of the target vehicle at the current moment based on the perception information, obtaining high-precision map data within a radius of r meters from the center point coordinates; extracting the perception information of all moving vehicles in the past k frames; and extracting the motion state vector of the target vehicle at the current moment based on the perception information;

[0023] S4, generating a lane line vector and a sidewalk vector of the target vehicle based on the high-precision map data and the motion state vector; generating a historical trajectory vector of all moving vehicles based on the perception information and motion state vectors of all moving vehicles in the past k frames;

[0024] S5, generating a corresponding minimum value vector based on the lane line vector, the sidewalk vector, and the historical trajectory vector;

[0025] S6: Input the lane line vector, sidewalk vector, historical trajectory vector, and minimum value vector into a deep neural network model to infer and generate a predicted trajectory of the target vehicle within the next s seconds.

[0026] According to one embodiment of the present invention, in step S1, the perception information includes the identification number of the identified object, the coordinates of the center point of the rectangular box corresponding to the identified object, the coordinates of the four vertices of the rectangular box corresponding to the identified object, the length, width, color, speed, acceleration, heading angle, and heading angle change rate of the identified object, and the identified object includes a moving vehicle.

[0027] According to one embodiment of the present invention, in step S2, the operation of managing the perception information includes:

[0028] Add: Add the perception information of new moving vehicles within the perception range of the perception module;

[0029] Deletion: deleting the perception information of existing moving vehicles that are beyond the perception range of the perception module;

[0030] Modification: updating the perception information of the existing moving vehicles within the perception range of the perception module;

[0031] Search: Search for the perception information of a target vehicle within the perception range of the perception module according to a certain search condition.

[0032] According to one embodiment of the present invention, in step S3, the motion state vector includes at least velocity, acceleration, heading angle, and heading angle change rate.

[0033] According to one embodiment of the present invention, after executing step S5 and before executing step S6, the minimum value vector is converted into a tensor format.

[0034] According to one embodiment of the present invention, 150≤r≤250, 15≤k≤25, and 1≤s≤5.

[0035] According to one embodiment of the present invention, the method for constructing a feature vector of the deep neural network model includes the steps of:

[0036] T1, obtains perception information and high-precision map data;

[0037] T2, generates the historical trajectory vectors of the target vehicle and all moving vehicles;

[0038] T3, generate lane line vector;

[0039] T4, generate sidewalk vector;

[0040] T5, generate the minimum vector corresponding to the historical trajectory vector, lane line vector and sidewalk vector.

[0041] According to one embodiment of the present invention, the deep neural network model includes a subGraph module, a globalGraph module and a backbone module;

[0042] The input of the subGraph module is the eigenvector of the (N, 10) matrix, and the output is the eigenvector of the (M, 64) matrix; the input of the globalGraph module is the output of the subGraph module and the minimum vector of (M, 2), and the output is the eigenvector of the (1, M, 64) matrix; the input of the backbone module is the output of the globalGraph module, and the output is the s-second predicted trajectory points of the (1, 60) matrix;

[0043] Among them, N = (target vehicle + number of moving vehicles) * 20 + (number of lane lines + number of sidewalks) * 10; M = (target vehicle + number of moving vehicles + number of lane lines + number of sidewalks).

[0044] According to one embodiment of the present invention, the training method of the deep neural network model includes the steps of:

[0045] U1, constructs the feature vector of the deep neural network model;

[0046] U2, input the feature vector into the deep neural network model to obtain a vehicle trajectory coordinate value (x t ,y t )(t=1,2,…T);

[0047] U3, calculate the loss function:

[0048]

[0049] Where I is the total number of predicted vehicles, and T is the number of coordinate points of a predicted trajectory;

[0050] U4, based on minimizing the value of the loss function, obtains the optimal model weight value

[0051] U5, repeat steps U1 to U4 to obtain the converged model weights

[0052] According to one embodiment of the present invention, in step U4, the Adam optimization method is used to optimize the model weight value. To update, the Adam optimization formula is:

[0053]

[0054] Where I is the total number of predicted vehicles and T is the number of coordinate points of a predicted trajectory.

[0055] The present invention also provides a device for predicting vehicle trajectories for urban open road scenarios, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of any of the aforementioned methods for predicting vehicle trajectories for urban open road scenarios are implemented.

[0056] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the aforementioned methods for predicting vehicle trajectories for urban open road scenarios.

[0057] The present invention provides a method, device and readable storage medium for predicting vehicle trajectories in urban open road scenarios. Through the perception information and high-definition map data obtained by the autonomous driving system, lane line vectors, sidewalk vectors and historical trajectory vectors are obtained, and the minimum value vector corresponding to the lane line vectors, sidewalk vectors and historical trajectory vectors is generated. All vectors are input into a deep neural network model, and then the predicted trajectory of the target vehicle is inferred. This method has high accuracy, a short inference process and strong applicability.

[0058] It is to be understood that both the foregoing general description and the following detailed description of the present invention are exemplary and explanatory and are intended to provide further explanation of the invention as claimed. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The accompanying drawings are included to provide further explanation of the present invention and are incorporated into and constitute a part of this application. The accompanying drawings illustrate embodiments of the present invention and together with the description serve to explain the principle of the present invention.

[0060] In the attached figure:

[0061] Figure 1 A flowchart of a method for predicting vehicle trajectories in an urban open road scenario according to an embodiment of the present invention is shown.

[0062] Figure 2 A flowchart of a method for constructing a feature vector of a deep neural network model according to an embodiment of the present invention is shown.

[0063] Figure 3 The figure shows the overall architecture of a deep neural network model according to an embodiment of the present invention.

[0064] Figure 4 A schematic structural diagram of a subGraph module of a deep neural network model according to an embodiment of the present invention is shown.

[0065] Figure 5 yes Figure 4 Schematic diagram of the structure of the MLP module in .

[0066] Figure 6 A schematic structural diagram of the globalGraph module of the deep neural network model according to an embodiment of the present invention is shown.

[0067] Figure 7 A schematic structural diagram of a backbone module of a deep neural network model according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0068] It should be noted that, unless there is any conflict, the embodiments and features in the embodiments of this application can be combined with each other.

[0069] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, rather than all embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is in no way intended to limit the present application and its application or use. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0070] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0071] Unless otherwise specified, the relative arrangement of the parts and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present application. Meanwhile, it should be understood that, for ease of description, the sizes of the various parts shown in the accompanying drawings are not drawn according to actual proportional relationships. Technology, methods and equipment known to those of ordinary skill in the relevant art may not be discussed in detail, but in appropriate cases, the technology, methods and equipment should be considered as a part of the specification. In all examples shown and discussed here, any specific value should be interpreted as being merely exemplary, rather than as a limitation. Therefore, other examples of exemplary embodiments can have different values. It should be noted that similar numbers and letters represent similar items in the following drawings, and therefore, once an item is defined in an accompanying drawing, it does not need to be further discussed in subsequent drawings.

[0072] In the description of this application, it should be understood that the directions or positional relationships indicated by directional words such as "front, back, up, down, left, right", "horizontal, vertical, vertical, horizontal" and "top, bottom" are usually based on the directions or positional relationships shown in the accompanying drawings. They are only for the convenience of describing this application and simplifying the description. Unless otherwise specified, these directional words do not indicate or imply that the device or element referred to must have a specific direction or be constructed and operated in a specific direction. Therefore, they cannot be understood as limiting the scope of protection of this application; the directional words "inside and outside" refer to the inside and outside relative to the outline of each component itself.

[0073] Furthermore, it should be noted that the use of terms such as "first" and "second" to define components is solely for the purpose of distinguishing the corresponding components. Unless otherwise stated, these terms have no special meaning and therefore should not be construed as limiting the scope of protection of this application. Furthermore, while the terms used in this application are selected from commonly known and commonly used terms, some terms mentioned in this specification may have been selected by the applicant at his or her discretion, and their detailed meanings are explained in the relevant sections of this description. Furthermore, this application should be understood not only by the actual terms used, but also by the meaning implied by each term.

[0074] Figure 1 A flowchart of a method for predicting vehicle trajectories in an urban open road scenario according to an embodiment of the present invention is shown. As shown in the figure, the present invention provides a method for predicting vehicle trajectories in an urban open road scenario, comprising the following steps:

[0075] S1, obtaining perception information provided by a perception module in the autonomous driving system. The perception module may be an on-board surround view detection module or a radar detection module.

[0076] S2, stores and manages perception information;

[0077] S3: Extract the current center point coordinates of the target vehicle based on the perception information, and obtain high-precision map data within a radius of r meters from the center point coordinates; extract the perception information of all moving vehicles in the past k frames; and extract the current motion state vector of the target vehicle based on the perception information.

[0078] S4: Generate the lane line vector and sidewalk vector of the target vehicle based on the high-precision map data and the motion state vector; generate the historical trajectory vector of all moving vehicles based on the perception information and motion state vectors of all moving vehicles in the past k frames;

[0079] S5, generating a corresponding minimum vector based on the lane line vector, the sidewalk vector and the historical trajectory vector;

[0080] S6: Input the lane line vector, sidewalk vector, historical trajectory vector, and minimum value vector into a deep neural network model to infer and generate a predicted trajectory of the target vehicle within the next s seconds.

[0081] As is readily understood, the present invention provides a method for predicting vehicle trajectories in urban open road scenarios, which is used to predict the trajectories of all moving vehicles within the perception range of an autonomous vehicle's perception module. This method predicts the trajectory of one moving vehicle at a time, with the vehicle whose trajectory is predicted being the target vehicle.

[0082] Preferably, in step S1, the perception information includes the identification number of the identified object, the coordinates of the center point of the rectangular box corresponding to the identified object, the coordinates of the four vertices of the rectangular box corresponding to the identified object, the length, width, color, speed, acceleration, heading angle, and heading angle change rate of the identified object, and the identified object includes at least a moving vehicle.

[0083] Preferably, in step S2, the operation of managing the perception information includes:

[0084] Added: Added the perception information of new moving vehicles within the perception range of the perception module;

[0085] Delete: Delete the perception information of existing moving vehicles that are beyond the perception range of the perception module;

[0086] Modification: Update the perception information of existing moving vehicles within the perception range of the perception module;

[0087] Search: Search for the perception information of a target vehicle within the perception range of the perception module based on a certain search condition.

[0088] Preferably, in step S3, the motion state vector includes at least velocity, acceleration, heading angle, and heading angle change rate.

[0089] Preferably, after executing step S5 and before executing step S6, the minimum value vector is converted into a tensor format, which is a data format suitable for a deep neural network model.

[0090] Preferably, 150≤r≤250, 15≤k≤25, 1≤s≤5, preferably r=200, k=20, s=3. Specifically, in step S3, the high-precision map data within a radius of 200 meters with the center point coordinates is obtained, and the perception information of all moving vehicles in the past 20 frames is extracted; in step S6, the predicted trajectory of the target vehicle in the next 3 seconds is inferred and generated.

[0091] Figure 2 A flowchart of a method for constructing a feature vector of a deep neural network model according to an embodiment of the present invention is shown. As shown in the figure, preferably, the method for constructing a feature vector of a deep neural network model includes the following steps:

[0092] T1, obtains perception information and high-precision map data;

[0093] T2, generates the historical trajectory vectors of the target vehicle and all moving vehicles;

[0094] T3, generate lane line vector;

[0095] T4, generate sidewalk vector;

[0096] T5, generate the minimum vector corresponding to the historical trajectory vector, lane line vector and sidewalk vector.

[0097] Furthermore, in step T2, the step of generating a historical trajectory vector includes:

[0098] T11, based on perception information and high-precision map data, extracts the target vehicle's current motion state vector, which includes the target vehicle's position (XY coordinates), speed, acceleration, heading angle (or yaw angle), and heading angle change rate;

[0099] T12, extract the motion state vector of the target vehicle at 20 historical moments;

[0100] T13, uses the motion state vectors of 20 historical moments to generate a historical trajectory vector of the target vehicle,

[0101] T14: Extract the motion state vectors of other moving vehicles at 20 historical moments within a radius of 100 meters with the current position of the target vehicle as the center.

[0102] T15, the motion state vectors of each moving vehicle at 20 historical moments generate a historical trajectory vector of the moving vehicle.

[0103] The historical trajectory vector includes a historical trajectory vector of the target vehicle and several historical trajectory vectors of moving vehicles.

[0104] In step T13, generating a historical trajectory vector of the target vehicle specifically includes the following steps:

[0105] T131, extracting the target vehicle's position points at 20 historical moments from the target vehicle's motion state vector at 20 historical moments;

[0106] T132, converts the position points of 20 historical moments into relative position points with the current position point of the target vehicle as the origin;

[0107] T133, construct 20 arrays (vectors) using the relative position points of 20 historical moments;

[0108] Each vector contains 10 variables, which are:

[0109] 1.x: the x coordinate value of the relative position point;

[0110] 2.y: the y coordinate value of the relative position point;

[0111] 3.delta_x: x-direction, the coordinate difference between time i+1 and time i, delta_x = at(i+1).x - at(i).x;

[0112] 4.delta_y: y direction, the coordinate difference between time i+1 and time i, delta_y = at(i+1).y - at(i).y;

[0113] 5.delta_index: indicates the number of vectors among the 20 vectors;

[0114] 6.0, just fill in 0;

[0115] 7.0, just fill in 0;

[0116] 8.0, just fill in 0;

[0117] 9.0, just fill in 0;

[0118] 10.Trajectory_index: trajectory number. Usually, the trajectory number of the target vehicle is 0, and the trajectory numbers of other moving vehicles start from 1 and increase.

[0119] Variables 6 to 9 are expandable items used to express the motion state more accurately.

[0120] At this point, a historical trajectory vector of the target vehicle is generated. The historical trajectory vector is a 20*10 matrix with 20 rows and 10 columns per row. The historical trajectory vector of the target vehicle can be expressed as {x, y, delta _ x,delta_y,delta_index,0,0,0,0,trajectory_index}.

[0121] Similarly, in step T15, the motion state vectors of each moving vehicle at 20 historical moments generate a historical trajectory vector of the moving vehicle, and this generation step is not repeated here.

[0122] In step T3, the steps of generating lane line vectors include:

[0123] T31, extract the current motion state vector of the target vehicle, which includes the target vehicle's position (XY coordinates), speed, acceleration, heading angle (or yaw angle), and heading angle change rate;

[0124] T32: Extract several lane lines within a radius of 100 meters with the current position of the target vehicle as the circle point. Each lane line is composed of several position points (XY coordinates);

[0125] T33, each lane line generates a lane line vector, and multiple lane lines generate multiple lane line vectors. The method of using a lane line to generate a lane line vector includes the following steps:

[0126] T331, starting from the beginning of a lane line, evenly pick points at intervals of several meters each time, and take a total of 20 points;

[0127] T332, convert the 20 position points into 20 relative position points with the current position of the target vehicle as the origin;

[0128] T333, construct 20 vectors using 20 relative position points.

[0129] Each vector contains 10 variables, which are:

[0130] x: x coordinate value of the relative position point;

[0131] y: y coordinate value of the relative position point;

[0132] delta_x: x direction, the coordinate difference between time i+1 and time i, delta_x = at(i+1).x - at(i).x

[0133] delta_y: y direction, the coordinate difference between time i+1 and time i, delta_y = at(i+1).y - at(i).y

[0134] delta_index: indicates the number of vectors among the 20 vectors;

[0135] 0, just fill in 0; (expandable item)

[0136] is_turn_left, whether it is a right turn lane;

[0137] is_turn_right, whether it is a left-turn lane;

[0138] is_intersection, whether it is an intersection;

[0139] lane_index: Lane number, starting from 0 and increasing in sequence.

[0140] At this point, a lane line vector is generated. The lane line vector is a 20*10 matrix with 20 rows and 10 columns per row. The lane line vector is represented as:

[0141] {x,y,delta_x,delta_y,delta_index,0,is_turn_left,is_turn_righ,is_intersection,lane_index}

[0142] In step T4, the steps of generating a sidewalk vector include:

[0143] T41: Extract the current motion state vector of the target vehicle. The motion state vector includes the target vehicle's position (XY coordinates), velocity, acceleration, heading angle (or yaw angle), and heading angle change rate.

[0144] T42, extracting a number of sidewalks within a radius of 100 meters with the current position of the target vehicle as the circle point. Each sidewalk is composed of a number of position points (XY coordinates).

[0145] T43, each sidewalk generates a sidewalk vector, and multiple sidewalks generate multiple sidewalk vectors. The method of using a sidewalk to generate a sidewalk vector includes the following steps:

[0146] T431, starting from the beginning of a sidewalk, evenly select points at intervals of several meters each time, and select a total of 20 points;

[0147] T432, convert the 20 position points into 20 relative position points with the current position of the target vehicle as the origin;

[0148] T433, uses 20 relative position points to construct 20 vectors, each vector contains 10 variables, the 10 variables are:

[0149] 1.x: the x coordinate value of the relative position point;

[0150] 2.y: the y coordinate value of the relative position point;

[0151] 3.delta_x: x-direction, the coordinate difference between time i+1 and time i, delta_x = at(i+1).x - at(i).x

[0152] 4.delta_y: y direction, the coordinate difference between time i+1 and time i, delta_y = at(i+1).y - at(i).y

[0153] 5.delta_index: indicates the number of vectors among the 20 vectors;

[0154] 6.0, just fill in 0; (This is an expandable item)

[0155] 7.0, just fill in 0; (This is an expandable item)

[0156] 8.0, just fill in 0; (This is an expandable item)

[0157] 9.0, just fill in 0; (This is an expandable item)

[0158] 10.crosswalk_index: crosswalk number, numbered starting from 0 and increasing.

[0159] At this point, a sidewalk vector is generated. The sidewalk vector is a 20*10 matrix with 20 rows and 10 columns per row. The sidewalk vector is represented as:

[0160] {x,y,delta_x,delta_y,delta_index,0,0,0,0,crosswalk_index}.

[0161] The method for generating a minimum value vector comprises the steps of:

[0162] T51, generate the minimum vector corresponding to the historical trajectory vector. Specifically, each historical trajectory vector is a 20*10 matrix with a total of 20 rows and 10 columns per row, where the first 2 columns of each row are the position point (x, y) values. In this way, each historical trajectory vector has 20 position point (x, y) values. Extract the minimum x value from the 20 position point x values, and extract the minimum y value from the 20 position point y values ​​to obtain the minimum (x, y) value of a historical trajectory vector. Similarly, multiple historical trajectory vectors can generate multiple pairs of minimum (x, y) values. A minimum vector can be generated from multiple pairs of minimum (x, y) values. The minimum vector is an n*2 matrix with a total of n rows and 2 columns per row, where n represents the sum of the number of target vehicles and moving vehicles.

[0163] T52, generate the minimum vector corresponding to the lane line vector. Since each lane line generates a lane line vector, each lane line vector is a 20*10 matrix with a total of 20 rows and 10 columns per row, where the first two columns of each row are the (x, y) values ​​of the position points. In this way, each lane line vector has 20 position point (x, y) values. Extract the minimum x value from the 20 position point x values ​​and the minimum y value from the 20 position point y values ​​to obtain a pair of minimum (x, y) values ​​for a lane line vector. Similarly, multiple lane line vectors can generate multiple pairs of minimum (x, y) values. Multiple pairs of minimum (x, y) values ​​can generate a minimum vector. The minimum vector is an m*2 matrix with a total of m rows and 2 columns per row, where m represents the number of lane lines.

[0164] T53, generate the minimum vector corresponding to the sidewalk vector. Since one sidewalk generates one sidewalk vector, each sidewalk vector is a 20*10 matrix with a total of 20 rows and 10 columns per row, where the first two columns of each row are the (x,y) values ​​of the location points. In this way, each sidewalk vector has 20 location point (x,y) values. Extract the minimum x value from the 20 location point x values, and extract the minimum y value from the 20 location point y values ​​to obtain a pair of minimum (x,y) values ​​for a sidewalk vector. Similarly, multiple sidewalk vectors can generate multiple pairs of minimum (x,y) values. A minimum vector can be generated from multiple pairs of minimum (x,y) values. The minimum vector is a k*2 matrix with a total of k rows and 2 columns per row, where k represents the number of sidewalks.

[0165] T54, merge minimum vectors. Combine the minimum vectors corresponding to the historical trajectory vector, the lane vector, and the sidewalk vector to generate a single minimum vector. This minimum vector is an M*2 matrix with M rows and 2 columns per row, where M = n + m + k.

[0166] Figure 3 The overall architecture diagram of a deep neural network model according to an embodiment of the present invention is shown. As shown in the figure, preferably, the deep neural network model includes a subGraph module, a globalGraph module, and a backbone module. The input of the subGraph module is the eigenvector of the (N, 10) matrix, and the output is the eigenvector of the (M, 64) matrix; the input of the globalGraph module is the output of the subGraph module and the minimum vector of (M, 2), and the output is the eigenvector of the (1, M, 64) matrix; the input of the backbone module is the output of the globalGraph module, and the output is the s-second predicted trajectory points of the (1, 60) matrix.

[0167] Among them, the parameter settings in the corresponding feature vector construction method are: N = (target vehicle + number of moving vehicles) * 20 + (number of lane lines + number of sidewalks) * 10; M = (target vehicle + number of moving vehicles + number of lane lines + number of sidewalks).

[0168] Figure 4 A schematic structural diagram of a subGraph module of a deep neural network model according to an embodiment of the present invention is shown. Figure 5 yes Figure 4 Schematic diagram of the structure of the MLP module in . Figure 6 A schematic structural diagram of the globalGraph module of the deep neural network model according to an embodiment of the present invention is shown. Figure 7This diagram shows the structure of the backbone module of a deep neural network model according to one embodiment of the present invention. The subGraph module, globalGraph module, and backbone module are primarily composed of an MLP (Multi-Layer Perceptron) module, a Max_Pool (Maximum Pooling) module, a Concatenante (Array Concatenation) module, and an FC (Fully Connected Layer) module.

[0169] refer to Figure 4 The input of the subGraph module is the eigenvector of the (N, 10) matrix, which is processed by three structured modules including the MLP module, the Max_Pool module and the Concatenante module in series, and the output is the eigenvector of the (M, 64) matrix. Figure 5 The input to the MLP module is a feature vector of (N, 10). After processing through the six-layer network, the output is a feature vector of (N, 64). The MLP module's function is to classify feature vectors at a higher dimension, thereby obtaining more significant and highly relevant feature vectors for each target vehicle. In general, the subGraph module's function is to obtain more significant and highly relevant feature vectors for each target vehicle by increasing the dimension to a higher-level vector space through the MLP module, extracting the maximum value through the Max_Pool module, and concatenating through the Concatenante module.

[0170] refer to Figure 6 The input of the globalGraph module is the output of the subGraph module and the minimum vector of the (M, 2) matrix, which is expanded to the (1, M, 66) dimension by the Concatenante module. The (1, M, 66)-dimensional feature vector is input to the self-attention layer represented by the dotted box. After processing by the self-attention layer, the (1, M, 64) matrix is ​​obtained as the output of the globalGraph.

[0171] The self-attention layer's function is to predict the future trajectory of a target vehicle, similar to how human drivers predict the future trajectories of surrounding vehicles. This requires considering the impact of other moving vehicles on the target vehicle, and therefore calculating the impact of other moving vehicles on the target vehicle. The self-attention layer calculates the impact of other moving vehicles on the target vehicle. The first matrix multiplication module multiplies the query matrix output by one FC module by the key matrix output by another FC module, and then divides the result by the integer 8 to obtain a (1, M, 64) matrix. This (1, M, 64) matrix is ​​then input into the softmax module.

[0172] refer to Figure 7The backbone input is the (1, M, 64) matrix output by the globalGraph. The target vehicle feature extraction module extracts the second dimension with an index of 0, resulting in a (1, 0, 64) feature vector, also known as the target vehicle feature vector. This (1, 0, 64) feature vector is reduced in dimension to obtain a (1, 64) feature vector. This (1, 64) feature vector is then processed through the MLP and FC layers, resulting in the output of the predicted trajectory points (1, 60).

[0173] Preferably, the training method of the deep neural network model includes the steps of:

[0174] U1, the feature vector for building a deep neural network model;

[0175] U2, inputs the feature vector into the deep neural network model to obtain the vehicle trajectory coordinate value (x t ,y t )(t=1,2,…T);

[0176] U3, calculate the loss function:

[0177]

[0178] Where I is the total number of predicted vehicles, and T is the number of coordinate points of a predicted trajectory;

[0179] U4, based on minimizing the value of the loss function, obtains the optimal model weight value

[0180] U5, repeat steps U1 to U4 to obtain the converged model weights

[0181] Preferably, in step U4, the Adam optimization method is used to optimize the model weights. To update, the Adam optimization calculation formula is:

[0182]

[0183] Where I is the total number of predicted vehicles and T is the number of coordinate points of a predicted trajectory.

[0184] The present invention also provides a device for predicting vehicle trajectories for urban open road scenarios, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of any of the aforementioned methods for predicting vehicle trajectories for urban open road scenarios are implemented.

[0185] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the aforementioned methods for predicting vehicle trajectories for urban open road scenarios.

[0186] Among them, the specific implementation methods and technical effects of the device for predicting vehicle trajectories for urban open road scenarios and the computer-readable storage medium can be referred to the embodiments of the method for predicting vehicle trajectories for urban open road scenarios provided by the above-mentioned present invention, and will not be repeated here.

[0187] The present invention provides a method, device, and readable storage medium for predicting vehicle trajectories in urban open road scenarios. The main features include using an array (vector) method to encode and represent high-precision map data and vehicle historical trajectories, as well as methods for establishing a deep neural network model, constructing the model's feature vectors, and training the model. The present invention has the following advantages:

[0188] 1) It has higher trajectory prediction accuracy, stronger applicability, fewer parameters in the deep neural network model, and a shorter model inference cycle.

[0189] 2) It not only considers road structure information and traffic rules, but also the interactions between different moving vehicles, making its prediction logic consistent with the logic used by human drivers to predict vehicle trajectories, thereby achieving a higher trajectory prediction accuracy.

[0190] 3) Because the method of generating vehicle predicted trajectories by deep neural network model inference is used, this method is not limited to certain scenarios (such as straight driving, right turning, etc.). This method is applicable to all scenarios of vehicles on open roads in cities, so it has stronger applicability.

[0191] 4) Because the array (vector) method is used to encode high-precision map data and historical trajectories of traffic vehicles, compared with existing image encoding methods, the method of the present invention has the advantage of using a simpler method to encode and represent the same information. The corresponding deep neural network model structure is simpler, the number of parameters of the deep neural network model is smaller, and a model inference cycle takes less time.

[0192] Those skilled in the art will further appreciate that the various illustrative logic blocks, modules, circuits, and algorithmic steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or a combination of the two. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps are generally described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. A skilled person may implement the described functionality in different ways for each specific application, but such implementation decisions should not be interpreted as resulting in a departure from the scope of the present invention.

[0193] The various illustrative logic modules and circuits described in conjunction with the embodiments disclosed herein may be implemented or executed using a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general-purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0194] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. The software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read and write information from / to the storage medium. In an alternative, the storage medium may be integrated into the processor. The processor and storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In an alternative, the processor and storage medium may reside in a user terminal as discrete components.

[0195] In one or more exemplary embodiments, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software as a computer program product, the functions may be stored on or transmitted via a computer-readable medium as one or more instructions or codes. Computer-readable media include both computer storage media and communication media, including any media that facilitates the transfer of a computer program from one location to another. A storage medium may be any available medium that can be accessed by a computer. By way of example and not limitation, such computer-readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. Any connection is also properly referred to as a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwaves, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwaves are included in the definition of medium. As used herein, disk and disc include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, where disks typically reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0196] It will be apparent to those skilled in the art that various modifications and variations may be made to the above exemplary embodiments of the present invention without departing from the spirit and scope of the present invention. Therefore, it is intended that the present invention cover modifications and variations of the present invention that fall within the scope of the appended claims and their equivalent technical solutions.

Claims

1. A method for predicting vehicle trajectories in an urban open road scenario, comprising the following steps: S1, obtains the perception information provided by the perception module in the autonomous driving system; S2, storing and managing the perception information; S3, extracting the center point coordinates of the target vehicle at the current moment based on the perception information, and obtaining high-precision map data within a radius of r meters from the center point coordinates; Extract the perception information of all moving vehicles in the past k frames; Extracting a current motion state vector of the target vehicle based on the perception information; S4, generating a lane line vector and a sidewalk vector of the target vehicle based on the high-precision map data and the motion state vector; generating a historical trajectory vector of all moving vehicles based on the perception information and motion state vectors of all moving vehicles in the past k frames; S5, generating a corresponding minimum value vector based on the lane line vector, the sidewalk vector, and the historical trajectory vector; S6, inputting the lane line vector, sidewalk vector, historical trajectory vector and minimum value vector into a deep neural network model, and inferring and generating a predicted trajectory of the target vehicle within the next s seconds; The method for constructing the feature vector of the deep neural network model includes the following steps: T1, obtains perception information and high-precision map data; T2, generates the historical trajectory vectors of the target vehicle and all moving vehicles; T3, generate lane line vector; T4, generate sidewalk vector; T5, generate the minimum vector corresponding to the historical trajectory vector, lane line vector and sidewalk vector.

2. The method for predicting vehicle trajectories in urban open road scenarios according to claim 1, characterized in that: In step S1, the perception information includes the identification number of the identified object, the coordinates of the center point of the rectangular box corresponding to the identified object, the coordinates of the four vertices of the rectangular box corresponding to the identified object, the length, width, color, speed, acceleration, heading angle, and heading angle change rate of the identified object, and the identified object includes a moving vehicle.

3. The method for predicting vehicle trajectories in urban open road scenarios according to claim 1, characterized in that: In step S2, the operation of managing the perception information includes: Add: Add the perception information of new moving vehicles within the perception range of the perception module; Deletion: deleting the perception information of existing moving vehicles that are beyond the perception range of the perception module; Modification: updating the perception information of the existing moving vehicles within the perception range of the perception module; Search: Search for the perception information of a target vehicle within the perception range of the perception module according to a certain search condition.

4. The method for predicting vehicle trajectories in urban open road scenarios according to claim 1, characterized in that: In step S3, the motion state vector includes at least velocity, acceleration, heading angle, and heading angle change rate.

5. The method for predicting vehicle trajectories in urban open road scenarios according to claim 1, characterized in that: After executing step S5 and before executing step S6, the minimum value vector is converted into a tensor format.

6. The method for predicting vehicle trajectories in urban open road scenarios according to claim 1, characterized in that: 150≤r≤250, 15≤k≤25, 1≤s≤5.

7. The method for predicting vehicle trajectories in urban open road scenarios according to claim 1, characterized in that: The deep neural network model includes a subGraph module, a globalGraph module and a backbone module; The input of the subGraph module is the eigenvector of the (N, 10) matrix, and the output is the eigenvector of the (M, 64) matrix; the input of the globalGraph module is the output of the subGraph module and the minimum vector of (M, 2), and the output is the eigenvector of the (1, M, 64) matrix; the input of the backbone module is the output of the globalGraph module, and the output is the s-second predicted trajectory points of the (1, 60) matrix; Among them, N = (target vehicle + number of moving vehicles) * 20 + (number of lane lines + number of sidewalks) * 10; M = (target vehicle + number of moving vehicles + number of lane lines + number of sidewalks).

8. The method for predicting vehicle trajectories in urban open road scenarios according to claim 7, characterized in that: The training method of the deep neural network model comprises the steps of: U1, constructs the feature vector of the deep neural network model; U2, input the feature vector into the deep neural network model to obtain a vehicle trajectory coordinate value (x t ,y t )(t=1,2,…T); U3, calculate the loss function: Where I is the total number of predicted vehicles, and T is the number of coordinate points of a predicted trajectory; U4, based on minimizing the value of the loss function, obtains the optimal model weight value U5, repeat steps U1 to U4 to obtain the converged model weights 9. The method for predicting vehicle trajectories in urban open road scenarios according to claim 8, characterized in that: In step U4, the Adam optimization method is used to optimize the model weights To update, the Adam optimization formula is: Where I is the total number of predicted vehicles and T is the number of coordinate points of a predicted trajectory.

10. A device for predicting vehicle trajectories in urban open road scenarios, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method for predicting vehicle trajectories for urban open road scenarios as described in any one of claims 1 to 6 are implemented.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for predicting vehicle trajectories for urban open road scenarios as described in any one of claims 1 to 6 are implemented.

Images