Vehicle motion prediction method and driving risk early warning method

Abstract

The invention belongs to the technical field of intelligent driving, and particularly relates to a vehicle motion prediction method and a driving risk early warning method. The vehicle motion prediction method comprises the following steps: S1, collecting vehicle-mounted visual data, serializing the vehicle-mounted visual data to obtain sequence data, and training a motion prediction model by using the sequence data to obtain the motion prediction model; s2, constructing an encoder of the motion prediction model, and obtaining an aggregation hidden state between surrounding vehicles; s3, constructing a vehicle space interaction module based on a graph structure, and iteratively calculating a space interaction weight between surrounding vehicles in a graph structure modeling mode; s4, constructing a decoder of the motion prediction model to obtain parameters describing binary Gaussian distribution, and randomly sampling control points of each surrounding vehicle in the binary Gaussian distribution; and S5, calculating a Bezier curve according to the control points, and obtaining a motion state high order according to the Bezier curve. According to the method, the surrounding vehicle motion prediction requirement of the current intelligent vehicle in an actual complex dynamic traffic scene is met.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent driving technology, and specifically relates to a vehicle motion prediction method and a driving risk warning method. Background Art

[0002] It is crucial for smart cars to predict the motion of surrounding traffic participants and evaluate potential driving risks. By predicting the motion state parameters such as the position, speed, acceleration, etc. of surrounding traffic participants in the future and evaluating and warning of potential driving risks in the predicted time domain, it helps smart cars make safer and more reasonable decisions and plans.

[0003] Vehicles are the main traffic participants in traffic scenes, so the motion prediction of vehicles is a typical motion prediction task. Vehicle motion prediction can usually be performed from two perspectives: ① based on the top-down perspective (drone shooting) or the near-top-down perspective (roadside visual sensor shooting) to predict the motion of vehicles in the shooting field of view; ② based on the visual sensor arranged in front of the vehicle to obtain the information of the vehicle in front. The perspective of the installation is similar to that of a human driver when driving a vehicle, which is defined as the vehicle-mounted perspective.

[0004] With the acceleration of mass production of smart cars equipped with autonomous driving technology, the vehicle motion prediction method based on the traffic perspective is difficult to meet the actual needs of autonomous driving. Compared with the vehicle motion prediction method based on the traffic perspective, the vehicle motion prediction method based on the vehicle perspective faces more challenges, such as more obvious target occlusion, more frequent changes in target depth information, etc. Therefore, the vehicle motion prediction method based on the vehicle perspective has become the focus of current research. In addition, in terms of driving risk assessment, deterministic safety indicators are often used to assess driving risks, such as Time To Collision (TTC). Such deterministic indicators usually ignore the uncertainty of the motion of the vehicle and surrounding vehicles, and lack flexibility in dealing with two-dimensional motion with lateral and longitudinal motion, and cannot better assess the potential risks of current driving. Therefore, the demand for driving risk assessment methods or indicators that can take into account the two-dimensional motion of the vehicle and its uncertainty is also increasing.

[0005] In the prior art, it is usually based on a data-driven method, which directly predicts the future motion state of the vehicle by a data-driven motion prediction method, and directly maps the input vehicle state information to the future high-order state quantity (such as speed, acceleration, etc.). This method directly predicts the high-order state of the vehicle's motion state in the future without complying with physical constraints such as vehicle kinematics / dynamics, which may cause the prediction result to be inconsistent with the true value. For example, the prediction result shows that the vehicle's speed is zero, but the vehicle still has a position displacement, which will cause confusion or even errors in the downstream decision and planning modules.

[0006] In the prior art, Chinese patent CN114820216A discloses a driving risk quantification method, classification method and prediction method, which uses deterministic risk assessment indicators such as TTC and THW (Time Head Way) to assess and quantify driving risks in the current traffic environment. This method lacks consideration of the uncertainty of surrounding vehicle movement and the subjective uncertainty of the driver in the driving cognitive risk method; at the same time, since the calculation formulas of TTC and THW are usually calculated based on the driving data of the self-vehicle and the target vehicle at the current moment, it is impossible to accurately quantify the driving risk in the future. Summary of the invention

[0007] The present invention aims to address the technical problem in the prior art that, when predicting vehicle motion and assessing risk, physical constraints are not taken into account or uncertainty is lacking, resulting in difficulty in accurately predicting the movement of surrounding vehicles and inaccurate risk assessment. The present invention aims to provide a vehicle motion prediction method and a driving risk warning method.

[0008] In order to solve the above technical problems, a first aspect of the present invention provides a vehicle motion prediction method, the vehicle motion prediction method comprising:

[0009] S1, collecting vehicle-mounted visual data, serializing the vehicle-mounted visual data for each surrounding vehicle along the order of collection time, obtaining sequence data with a plurality of segments, and using the sequence data to train a preset motion prediction model to obtain a motion prediction model corresponding to each surrounding vehicle;

[0010] S2, constructing an encoder of the motion prediction model, assigning an encoder to each pericar respectively, and obtaining the aggregated hidden state between the pericars;

[0011] S3, with each vehicle as a node and the spatial interaction relationship between vehicles as an edge, a vehicle spatial interaction module based on a graph structure is constructed, and the spatial interaction weights between surrounding vehicles are iteratively calculated in a graph structure modeling manner;

[0012] S4, constructing a decoder of the motion prediction model, splicing the aggregated hidden states between the pericars and the spatial interaction weights between the pericars, regressing the binary Gaussian distribution of the control points under the preset Bezier curve control point output paradigm, obtaining parameters describing the binary Gaussian distribution, and randomly sampling a preset number of control points in the binary Gaussian distribution for each pericar;

[0013] S5, calculating a Bezier curve according to the control points of each pericar, wherein the Bezier curve is the predicted trajectory of the pericar, and obtaining a high-order quantity of the motion state of the pericar in the predicted time domain by derivation of the Bezier curve.

[0014] Optionally, in the vehicle motion prediction method as described above, in step S1, collecting vehicle-mounted visual data includes:

[0015] One or more visual sensors are arranged at preset positions on the vehicle, and the internal and external parameters of the visual sensors are obtained using a preset calibration algorithm. After the calibration is successful, the vehicle-mounted visual data of the visual sensors are collected.

[0016] Optionally, in the vehicle motion prediction method as described above, in step S1, after collecting the vehicle-mounted visual data, the method further includes:

[0017] Acquire surrounding vehicle information, wherein the surrounding vehicle information includes the distance between each surrounding vehicle and the own vehicle and the vehicle number corresponding to each surrounding vehicle;

[0018] According to the different vehicle numbers, the vehicle-borne visual data is serialized for each vehicle along the collection time sequence to obtain sequence data.

[0019] Optionally, in the vehicle motion prediction method as described above, the obtaining surrounding vehicle information includes:

[0020] Using a preset target detection module to perform target detection on the vehicle-mounted visual data, and obtaining a target detection frame containing the position and category of each vehicle on each frame image;

[0021] Estimate the surrounding vehicles according to the target detection frame and the visual sensor parameters to obtain the distance between the surrounding vehicles and the self-vehicle;

[0022] A preset target tracking module is used to associate and track the surrounding vehicles detected at different times in the vehicle-mounted visual data, and a corresponding vehicle number is assigned to each surrounding vehicle.

[0023] Optionally, in the vehicle motion prediction method as described above, when the on-board visual data is serialized for each surrounding vehicle along the acquisition time sequence, a sliding window method is used for serialization, and the sliding window is the sequence length.

[0024] Optionally, in the vehicle motion prediction method as described above, after the on-board visual data is serialized for each surrounding vehicle along the acquisition time sequence, outliers are removed to obtain sequence data.

[0025] Optionally, in the vehicle motion prediction method as described above, in step S1, the preset motion prediction model is trained using the sequence data to obtain the motion prediction model corresponding to each surrounding vehicle, including:

[0026] The sequence data is evenly divided into K parts using a preset K-fold cross-validation method, one of which is selected as a validation set, and K-1 parts are selected as training sets. The first h seconds of each sequence are used as an observation sequence, and the last Hh seconds are used as a prediction sequence, where H is the sequence length and h is less than H;

[0027] The motion prediction model is trained using the training set, the trained motion prediction model is verified using the verification set, and an evaluation index is calculated. If the evaluation index reaches a preset expected index, the motion prediction model training is considered completed.

[0028] Optionally, in the vehicle motion prediction method as described above, the evaluation index is one of a final displacement error (Final Displacement Error, FDE) and an average displacement error (Average Displacement Error, ADE) or a combination thereof, and the calculation formulas are respectively formula (1) and formula (2):

[0029]

[0030] Among them, the medium T f represents the length of the prediction sequence, the subscript t represents each time step in the prediction time series, and x t and t are the ordinate and abscissa of the true value of the trajectory, and They are the ordinate and abscissa of the trajectory prediction value respectively.

[0031] Optionally, in the vehicle motion prediction method as described above, in step S2, an encoder is assigned to each surrounding vehicle to obtain the aggregated hidden state between the surrounding vehicles, including:

[0032] A LSTM (Long Short Term Memory network) encoder is assigned to each of the cycle vehicles in the observation sequence. The LSTM encoder is used to extract features from the historical motion state information in the observation sequence. The extracted features are output in the hidden state. The calculation formula for the aggregated hidden state between cycle vehicles is formula (3):

[0033]

[0034] in, represents the aggregated hidden state of vehicle i at time step t, j represents another vehicle different from vehicle i, N represents the total number of weekly vehicles, M j represents the hidden state aggregation process from vehicle j to vehicle i, is the hidden state of vehicle j at time step t, is the hidden state of vehicle i at time step t.

[0035] Optionally, in the vehicle motion prediction method as described above, the hidden state aggregation process adopts one or a combination of concatenation, addition and multiplication methods.

[0036] Optionally, in the vehicle motion prediction method as described above, in step S3, the vehicle space interaction module is an undirected graph, and the condition for the existence of an edge between two nodes is that the Euclidean distance between the two nodes is less than a preset distance threshold.

[0037] Optionally, in the vehicle motion prediction method as described above, in step S3, the spatial interaction weights between surrounding vehicles are iteratively calculated in a graph structure modeling manner, including:

[0038] Each vehicle is regarded as a node v in the graph structure, where v = {v1, v2, v3, ..., v N}, N is the total number of weekly vehicles;

[0039] At time step t, establish spatial interaction based on graph structure and define any node v in the undirected graph G i and node v j The symmetric adjacency matrix of is A(i, j), as shown in formula (4):

[0040]

[0041] in, Represents node v i With node v j The spatial interaction relationship between thresh is the distance threshold;

[0042] The unregularized Laplacian matrix L = DA of the undirected graph G is defined as a symmetric matrix as shown in formula (5):

[0043]

[0044] Where D(i, j) is the main diagonal of the degree matrix D of the undirected graph G, The remaining elements are all 0;

[0045] The Laplace matrix defined at time step t is denoted as L t , then the Laplace matrix L of the next time step t+1 The update method of is as follows:

[0046]

[0047] Among them, ff transrepresents the outer product of rank 2, f is a sparse matrix and satisfies ||f||0<<N. Each time a new non-zero item appears in the j-th row of f, it means that a new week car has been added to the current undirected graph G. The maximum number of nodes in the undirected graph G is the total number of week cars N.

[0048] Optionally, in the vehicle motion prediction method as described above, in step S4, the formula of the n-order Bezier curve is as follows:

[0049]

[0050] in, represents the Bessel basis function of the i-th order, η represents the Bessel proportional coefficient, P i represents the Bezier curve control point, t f is any time step in the prediction domain, T f represents the predicted sequence length,

[0051] Optionally, in the vehicle motion prediction method as described above, in step S4, when regressing the binary Gaussian distribution of the control point, a preset multi-layer perceptron (Multi Layer Perception, MLP) is used to regress the binary Gaussian distribution of the Bezier curve control point to obtain the parameter μ describing the binary Gaussian distribution. x , μ y , σ x , σ y and ρ represent the x-mean, y-mean, x-standard deviation, y-standard deviation, and correlation coefficient of the bivariate Gaussian distribution, respectively.

[0052] Optionally, in the vehicle motion prediction method as described above, in step S5, the Bezier curve coordinates are formula (12):

[0053]

[0054] Among them, Y pos represents the position coordinates, B represents the constant Bessel basis matrix, and P represents the predicted control point coordinates.

[0055] Optionally, in the vehicle motion prediction method as described above, in step S5, the high-order quantity of the motion state includes at least one of velocity, acceleration and jerk, and the high-order quantity of the motion state is obtained by taking high-order derivatives of the Bezier curve.

[0056] In order to solve the above technical problems, a second aspect of the present invention provides a driving risk warning method, the driving risk warning method comprising:

[0057] The parameters of the surrounding vehicles in the binary Gaussian distribution obtained in step S4 are used to calculate the collision probability between the ego vehicle and the surrounding vehicles in the prediction time domain. The high-order quantity of the motion state obtained in step S5 is used to calculate the expected collision severity of the ego vehicle. The product of the collision probability and the expected collision severity of the ego vehicle is used as the driving risk of the ego vehicle.

[0058] The driving risk is compared with a preset risk threshold, and if the driving risk is greater than the risk threshold, a risk warning is issued.

[0059] Optionally, in the aforementioned driving risk warning method, the parameters of the surrounding vehicles in the binary Gaussian distribution obtained in step S4 are used to calculate the collision probability between the vehicle and the surrounding vehicles in the prediction time domain, including:

[0060] The parameters of the bivariate Gaussian distribution of the pericycle obtained in step S4 are given by formula (13):

[0061]

[0062] Among them, the parameter μ of the bivariate Gaussian distribution is x , μ y , σ x , σ y and ρ, respectively represent the x-mean, y-mean, x-standard deviation, y-standard deviation and correlation coefficient of the bivariate Gaussian distribution, with the superscript t f is any time step in the prediction domain, It is the Gaussian distribution representation of the two-dimensional motion of the pericycle;

[0063] When the vehicle is driven by a human driver, there is still subjective driving uncertainty. Formula (13) is rewritten and the collision probability is calculated by formula (14):

[0064]

[0065] in, They are the parameter representations after introducing the binary Gaussian distribution of the subjective uncertainty of human drivers, and represents the standard deviation of the Gaussian distribution of subjective uncertainty of human drivers;

[0066] When the vehicle is in the automatic driving state, the collision probability is calculated by formula (15):

[0067]

[0068] in, x , and is the vertical and horizontal coordinates of the vehicle at any time step in the prediction time domain, and Provided by the preset autonomous driving planning module, L sub and L sur is the length of the ego vehicle and the length of the surrounding vehicle, W sub and W sur is the width of the vehicle and the surrounding vehicles, L sub and W sub is the default value, L sur and W sur The vehicle-mounted visual data collected in step S1 is obtained through target detection.

[0069] Optionally, in the aforementioned driving risk warning method, the high-order quantity of the motion state obtained in step S5 is used to calculate the expected collision severity of the vehicle, including:

[0070] The expected collision severity of the ego vehicle is calculated by formula (16):

[0071] s(t+Δt)=0.5Mβ 2 (Δv(t+Δt)) 2 #(16)

[0072] Among them, M is the weight of the vehicle, M is the preset value, β is the weight ratio of the surrounding vehicle, M sur is the weight of the vehicle, M sur The empirical value is obtained according to the preset empirical formula after the type and length and width information of the surrounding vehicles are obtained through target detection using the on-board visual data collected in step S1. Δv(t+Δt) is the speed difference between the ego vehicle and the surrounding vehicles between time step t and time increment Δt. The ego vehicle speed is provided by the preset autonomous driving planning module, and the surrounding vehicle speed is provided by the high-order quantity of the motion state obtained in step S5.

[0073] Optionally, in the driving risk warning method as described above, the risk threshold is a dynamic risk threshold, and the dynamic risk threshold has corresponding different risk levels, and the risk levels are obtained through subjective driving risk cognition experiments in different driving scenarios.

[0074] The positive and progressive effects of the present invention are:

[0075] 1. The present invention better associates the upstream perception module with the motion prediction module from the vehicle-mounted perspective to realize the surrounding vehicle motion prediction and driving risk warning method, which helps smart cars to achieve safer and more reasonable decision-making and planning, and meets the current smart car's surrounding vehicle motion prediction needs in actual complex dynamic traffic scenarios.

[0076] 2. The spatial interaction module of surrounding vehicles based on the graph structure of the present invention reflects the spatial interaction relationship between surrounding vehicles in the actual traffic environment. The model output paradigm based on the Bezier curve control points overcomes the problem that the previous data-driven motion prediction method may make mistakes in directly predicting high-order quantities of motion states without physical constraints.

[0077] 3. Based on the uncertainty of vehicle motion prediction, the present invention introduces the driver's subjective uncertainty for human-machine co-driving situations, and more objectively describes the subjective cognitive risks of drivers and passengers in complex dynamic scenarios. Under the premise of meeting objective safety, the present invention enables drivers and passengers to better recognize the driving risks of smart cars. The present invention can provide a reliable reference for the decision-making and planning modules downstream of the autonomous driving system.

[0078] 4. The present invention determines the dynamic risk threshold through the driver's subjective risk perception results, and warns the driver of the vehicle when the driving risk is greater than the risk threshold of the current scenario. This solves the problem that deterministic safety assessment indicators cannot objectively describe driving risks in complex dynamic traffic scenarios, and thus it is difficult to provide accurate driving risk warnings for the vehicle. This can provide safety guarantees for the realization of reasonable decision-making planning for smart cars.

[0079] 5. The present invention solves the problem that the vehicle motion prediction method based on a bird's-eye view or near bird's-eye view is difficult to accurately predict the motion of surrounding vehicles due to occlusion by surrounding vehicles and frequent changes in depth information in actual traffic scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] The disclosure of the present invention will become more apparent with reference to the accompanying drawings. It should be understood that these drawings are only for illustrative purposes and are not intended to limit the scope of protection of the present invention. In the drawings:

[0081] Figure 1 A method flow chart of the present invention;

[0082] FIG2( a ) is a view of the acquisition viewing angle after the vehicle-mounted visual sensor is installed in Example 1 of the present invention;

[0083] FIG2( b ) is a schematic diagram of a collection section on an electronic map in Embodiment 1 of the present invention;

[0084] Figure 3 is the target detection result of a frame of image after target detection in Embodiment 1 of the present invention;

[0085] Figure 4 This is a schematic diagram of the space interaction of the vehicle in Example 1 of the present invention. DETAILED DESCRIPTION

[0086] The following describes the embodiments of the present invention through specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention.

[0087] It should be noted that the following embodiments and features therein may be combined with each other if there is no conflict.

[0088] In the description of the present invention, it should be noted that directional words such as the terms "outside", "middle", "inside", "outside", etc., which indicate directions and positional relationships, are based on the directions or positional relationships shown in the drawings. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and cannot be understood as limiting the specific protection scope of the present invention.

[0089] In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features. Therefore, the definition of "first" and "second" features can explicitly or implicitly include one or more of the features. In the description of the present invention, "several" and "a number" mean two or more, unless otherwise clearly and specifically defined.

[0090] Reference Figure 1 An embodiment of the present invention provides a vehicle motion prediction method, which accurately predicts the motion of surrounding vehicles and evaluates the current driving risk from a vehicle-mounted perspective. It specifically utilizes the surrounding environment information from a vehicle-mounted perspective, and proposes a surrounding vehicle motion prediction method that considers the spatial interaction of surrounding vehicles. At the same time, an output paradigm based on Bezier curve control points is proposed in the prediction process. This method allows the motion state of surrounding vehicles and their high-order derivatives at any time step in the prediction time domain.

[0091] The vehicle motion prediction method comprises the following specific steps:

[0092] S1, data collection and preprocessing: collect vehicle-mounted visual data, serialize the vehicle-mounted visual data for each surrounding vehicle along the collection time sequence, obtain sequence data with several segments, use the sequence data to train the preset motion prediction model, and obtain the motion prediction model corresponding to each surrounding vehicle.

[0093] The data collected in this step is the vehicle-mounted visual data from the vehicle-mounted perspective. By processing the vehicle-mounted visual data using existing methods, the surrounding vehicle information under the vehicle-mounted vision can be obtained, including the position of each vehicle in a frame of image, vehicle type, vehicle length and width, vehicle code and other information.

[0094] In this step, when using sequence data for training, each perimeter vehicle is trained separately to obtain a motion prediction model corresponding to each perimeter vehicle.

[0095] In some embodiments, in step S1, collecting vehicle-mounted visual data includes calibrating the vehicle-mounted visual sensor and collecting vehicle-mounted view angle data. Specifically, one or more visual sensors are arranged at preset positions on the vehicle, and the internal and external parameters of the visual sensor are obtained using a preset calibration algorithm. After the calibration is successful, the vehicle-mounted visual data of the visual sensor is collected.

[0096] In this embodiment, the calibration algorithm is an existing technology, such as Zhang's calibration method, calibration based on optimization algorithm, calibration based on genetic algorithm and calibration based on neural network. After the calibration is successful, the vehicle is driven to collect image data from the vehicle's perspective, that is, vehicle-mounted visual data.

[0097] Preferably, the perception range of the visual sensor can be selected as 200 meters, the time window can be selected as 10 minutes, and the acquisition frequency can be 10 Hz.

[0098] In some embodiments, in step S1, after collecting the vehicle-mounted visual data, the following steps are further included:

[0099] Obtain surrounding vehicle information, which includes the distance between each surrounding vehicle and the own vehicle and the vehicle number corresponding to each surrounding vehicle.

[0100] Serialize vehicle vision data: According to the different vehicle numbers, the vehicle vision data is serialized for each vehicle in the order of collection time to obtain sequence data.

[0101] In this embodiment, when obtaining surrounding vehicle information, the target detection module and target tracking module based on the intelligent automobile environment perception system in the prior art can be used to detect and track vehicles within the perception range of the visual sensor to obtain information about the vehicle during the collection process.

[0102] In this embodiment, the vehicle-mounted visual data and the surrounding vehicle information obtained from the vehicle-mounted visual data can be saved in a csv file for integration, and serialized according to the different vehicle numbers and along the collection time sequence.

[0103] In some embodiments, the process of obtaining surrounding vehicle information based on the target detection module and the target tracking module is as follows:

[0104] The target detection module is used to detect the target in the vehicle visual data, and the target detection frame containing the position and category of each surrounding vehicle in each frame image is obtained. The surrounding vehicles are estimated according to the target detection frame and the visual sensor parameters to obtain the distance between the surrounding vehicles and the vehicle. The target tracking module is used to associate and track the surrounding vehicles detected at different times in the vehicle visual data, and the corresponding vehicle number is assigned to each surrounding vehicle.

[0105] For target detection, the target detection frame of the surrounding vehicles is obtained from a frame of image data in the vehicle-mounted visual data. Depending on the visual sensor used, the dimension of the target detection frame obtained may be two-dimensional or three-dimensional, but it should at least include the length, width and center point information of the target detection frame. The target detection module can use existing technology. For example, the method for the target detection module to perform vehicle target detection can be a single-stage method (such as the YOLO series algorithm, the DETR series algorithm, etc.), a two-stage method (RCNN series algorithm) and a Transformer-based method.

[0106] For target tracking, the surrounding vehicles detected at different times when the visual sensor collects data are associated and tracked to obtain independent numbers of the surrounding vehicles, that is, each vehicle in the scene has its own unique vehicle number. The target tracking module can use existing technology, for example, the method used by the target tracking module can be one of a rule-based method and a deep learning-based method or a combination thereof.

[0107] In some embodiments, when the vehicle-mounted visual data is serialized for each vehicle along the order of collection time, a sliding window method is used for serialization, and the sliding window is the sequence length.

[0108] For example, the sliding window is set to 5 seconds, that is, the sequence length is also 5 seconds.

[0109] In some embodiments, after the on-board visual data is serialized for each vehicle along the acquisition time sequence, outliers are removed to obtain sequence data.

[0110] In the serialized sequence data, there may be outliers due to possible reasons such as target detection error and target tracking error. Outliers include but are not limited to vehicle position mutations caused by inconsistent vehicle numbers due to target tracking errors. The method for finding and eliminating outliers can be one of the following methods: manual inspection, statistical feature inspection, and methods based on domain knowledge and specific rules, or a combination thereof.

[0111] In some embodiments, in step S1, the preset motion prediction model is trained using the sequence data to obtain the motion prediction model corresponding to each surrounding vehicle, including dividing the sequence data set using a K-fold cross-validation method.

[0112] Specifically: The sequence data is evenly divided into K parts using the K-fold cross-validation method, one of which is selected as the validation set, and K-1 parts are selected as the training set. The first h seconds of each sequence are used as the observation sequence, and the last Hh seconds are used as the prediction sequence, where H is the sequence length and h is less than H. The motion prediction model is trained using the training set, and the motion prediction model obtained by the training is verified using the validation set, and the evaluation index is calculated. If the evaluation index reaches the preset expected index, the motion prediction model training is considered to be completed.

[0113] The serialized sequence data has several segments. The K-fold cross-validation method is used to divide the data set into K equal subsets, and multiple iterations of training and validation are performed to overcome the limitations of the simple training set-test set division. Especially when it is necessary to maximize the value of limited data utilization, the K-fold cross-validation method can effectively utilize data, reduce overfitting, and make the evaluation of model generalization ability more reliable. The value of K is preferably between [2,10].

[0114] This embodiment also divides each sequence into an observation sequence and a prediction sequence. For example, when the sequence length is 5 seconds, the first 3 seconds are used as the observation sequence and the last 2 seconds are used as the prediction sequence.

[0115] The evaluation index and its calculation method in this embodiment are prior art, and the evaluation index is preferably one of the final displacement error (Final Displacement Error, FDE) and the average displacement error (Average Displacement Error, ADE) or a combination thereof. The calculation formula of FDE is formula (1), and the calculation formula of ADE is formula (2):

[0116]

[0117] Among them, the medium T f represents the length of the prediction sequence, the subscript t represents each time step in the prediction time series, and x t and t are the ordinate and abscissa of the true value of the trajectory, and They are the ordinate and abscissa of the trajectory prediction value respectively.

[0118] FDE and ADE can be used to evaluate the accuracy and precision of the motion prediction model. Smaller FDE and ADE values, such as those less than the corresponding expected indicators, indicate that the predicted results are closer to the actual trajectory position, indicating that the motion prediction model has good performance and has completed training. Of course, if the FDE and ADE values ​​are not less than the expected indicators, the model can be adjusted by modifying the model parameters, and training verification and evaluation can be performed again to complete the training.

[0119] S2, construct an encoder to extract features: construct an encoder for the motion prediction model, assign an encoder to each week car, and obtain the aggregated hidden state between the week cars.

[0120] The encoder in this step is used to encode the feature information in the motion prediction model into an intermediate state for use in generating trajectories in the subsequent decoding stage.

[0121] In some embodiments, in step S2, an encoder is assigned to each surrounding vehicle to obtain the aggregated hidden state between the surrounding vehicles, including:

[0122] A LSTM (Long Short Term Memory network) encoder is assigned to each of the pericycles in the observation sequence. The LSTM encoder is used to extract features of the historical motion state information in the observation sequence. The extracted features are output in the hidden state and then used as the input of the motion prediction model decoder.

[0123] The calculation formula of the aggregated hidden state between the weekly vehicles is formula (3):

[0124]

[0125] in, represents the aggregated hidden state of vehicle i at time step t, j represents another vehicle different from vehicle i, N represents the total number of weekly vehicles, M j represents the hidden state aggregation process from vehicle j to vehicle i, is the hidden state of vehicle j at time step t, is the hidden state of vehicle i at time step t.

[0126] The hidden state aggregation process uses one or a combination of concatenation, addition and multiplication methods.

[0127] S3, construct a vehicle space interaction module based on a graph structure: take each vehicle as a node and the spatial interaction relationship between vehicles as an edge, construct a vehicle space interaction module based on a graph structure, and iteratively calculate the spatial interaction weights between surrounding vehicles using a graph structure modeling method.

[0128] This step uses a graph structure to express the interaction between surrounding vehicles in the traffic scene. A graph is a nonlinear data structure that consists of nodes (also called vertices) and edges. A graph can be used to represent various relationship networks. A node is the basic unit in a graph and represents an entity. An edge is a line connecting two nodes, indicating a certain relationship between the two nodes. Among them, the edges of an undirected graph have no directionality. If there is an edge from A to B, there is also an edge from B to A. Degree represents the number of edges connected to a node.

[0129] The spatial interaction weights between surrounding vehicles are iteratively calculated by graph structure modeling, so that at time step t, the smaller the relative distance between two vehicles, the stronger the interaction.

[0130] In some embodiments, in step S3, the vehicle space interaction module is an undirected graph, and the condition for the existence of an edge between two nodes is that the Euclidean distance between the two nodes is less than a preset distance threshold.

[0131] In the traffic scene, each vehicle is regarded as a node v in the graph structure, where v = {v1, v2, v3, ..., v N}, the subscript N is the total number of vehicles in the scene, and the condition for defining the existence of an edge between two nodes is that the Euclidean distance between the two nodes is less than a pre-set distance threshold d thresh , then the spatial relative position relationship between vehicles is regarded as an edge.

[0132] In some embodiments, in step S3, the spatial interaction weights between surrounding vehicles are iteratively calculated in a graph structure modeling manner, including:

[0133] Each vehicle is regarded as a node v in the graph structure, where v = {v1, v2, v3, ..., v N}, N is the total number of weekly vehicles;

[0134] a. Establish spatial interaction based on graph structure in the scenario of time step t and total number of vehicles N. Define any node v in the undirected graph G. i and node v j The symmetric adjacency matrix of is A(i, j), as shown in formula (4):

[0135]

[0136] in, Represents node v i With node v j The spatial interaction relationship between thresh is the distance threshold; of course, v iRepresents vehicle i as a node in the undirected graph G, v j Represent another vehicle different from vehicle i as another node in the undirected graph G;

[0137] b. Define the non-regularized Laplacian matrix L=DA of the undirected graph G as a symmetric matrix as shown in formula (5):

[0138]

[0139] Where D(i, j) is the main diagonal of the degree matrix D of the undirected graph G, The remaining elements are all 0;

[0140] c. The Laplace matrix of each time step is related to the Laplace matrix of the past time step. The Laplace matrix defined at time step t is expressed as L t , then the Laplace matrix L of the next time step t+1 The update method of is as follows:

[0141]

[0142] Among them, ff trans represents the outer product of rank 2, f is a sparse matrix and satisfies ||f||0<<N. Each time a new non-zero item appears in the j-th row of f, it means that a new week car has been added to the current undirected graph G. The maximum number of nodes in the undirected graph G is the total number of week cars N.

[0143] S4, construct the decoder of the motion prediction model and output Bezier curve control points: concatenate the aggregated hidden states between the weekcars and the spatial interaction weights between the weekcars, regress the binary Gaussian distribution of the control points under the preset Bezier curve control point output paradigm, obtain the parameters describing the binary Gaussian distribution, and randomly sample a preset number of control points for each weekcar in the binary Gaussian distribution.

[0144] In this step, in order to reliably predict the motion state of the pericar and its high-order quantity for reference by the downstream decision-making and planning modules, a model output paradigm based on Bezier curve control points is proposed. The aggregated hidden state between the pericars obtained in step S2 and the spatial interaction weights between the pericars obtained in step S3 are spliced, and under the Bezier curve control point output paradigm of this step, the binary Gaussian distribution of the control points is finally regressed to obtain the parameter μ describing the binary Gaussian distribution. x , μ y , σ x , σ yand ρ, respectively represent the x-mean value, y-mean value, x-standard deviation, y-standard deviation and correlation coefficient of the distribution, and randomly sample a preset number of control points under the distribution. The number of control points is determined according to the Bezier curve, that is, for an n-order Bezier curve, n+1 control points are sampled. For example, if a 5-order Bezier curve is predetermined, 6 control points need to be randomly sampled under a bivariate Gaussian distribution.

[0145] In some embodiments, in step S4, the formula of the n-order Bezier curve is as follows:

[0146]

[0147] in, represents the Bessel basis function of the i-th order, represents the binomial coefficient, η represents the Bessel proportionality coefficient, P i Represents the control points of the Bezier curve. For a Bezier curve of order n, there are n+1 control points;

[0148] It should be noted that since η∈[0,1], the time step values ​​in the prediction time domain are also normalized, as shown in formula (8):

[0149]

[0150] Among them, t f is any time step in the prediction domain, T f Indicates the predicted sequence length;

[0151] Then formula (7) can be rewritten as formula (9):

[0152]

[0153] At this point, the formula for the n-th order Bezier curve is obtained as shown in formula (9).

[0154] In some embodiments, in step S4, when regressing the binary Gaussian distribution of the control points, a preset multi-layer perceptron (MLP) is used to regress the binary Gaussian distribution of the Bezier curve control points to obtain the parameter μ describing the binary Gaussian distribution. x , μ y , σ x , σ y and ρ represent the x-mean, y-mean, x-standard deviation, y-standard deviation, and correlation coefficient of the bivariate Gaussian distribution, respectively.

[0155] S5, calculate the Bezier curve and obtain the motion state and its high-order quantities: According to the control points of each pericyte, calculate the Bezier curve, which is the predicted trajectory of the pericyte. According to the derivation of the Bezier curve, the high-order quantities of the motion state of the pericyte in the predicted time domain are obtained.

[0156] By using the control points of each perimeter vehicle obtained in step S4, the Bezier curve of each perimeter vehicle can be obtained.

[0157] The high-order quantity of the motion state includes at least one of velocity, acceleration and jerk. According to the property that the derivative of the Bezier curve is still the Bezier curve, the Bezier curve representing the trajectory can be first-order, second-order or higher-order derivative to obtain the high-order quantity of the motion state. For example, the first-order derivative of the Bezier curve is obtained to obtain the Bezier curve representing the velocity, the second-order derivative of the Bezier curve is obtained to obtain the Bezier curve representing the acceleration, the third-order derivative of the Bezier curve is obtained to obtain the Bezier curve representing the jerk, and so on. According to the derivative of the Bezier curve, the high-order quantity of the motion state of the pericycle in the prediction time domain is obtained.

[0158] In some embodiments, when calculating the Bezier curve from the control points of each pericentric obtained in step S4, the mathematical property of the curve, that is, the higher-order derivative of the Bezier curve is still the Bezier curve, and the control point expression is as shown in formula (10), where the subscript i represents the i-th order and the superscript represents the first-order derivative of the control point of that order; the higher-order derivative expression of the Bezier curve is as shown in formula (11):

[0159] P i (1) =n(P i+1 -P i ) (10)

[0160]

[0161] In some embodiments, after the control points of each periscope are obtained in step S4, the position coordinates in the motion state can be obtained by formula (12):

[0162]

[0163] Among them, Y pos represents the position coordinates, B represents the constant Bessel basis matrix, and P represents the predicted control point coordinates.

[0164] In formula (12), the dimension of the Bessel basis matrix is ​​T f ×(n+1), the dimension of the control point coordinates is (n+1)×2. After multiplying according to formula (12), the dimension of the trajectory coordinates is T f ×2, which means that in the prediction time domain T f The 2D position coordinates of the pericycle motion state within.

[0165] Based on this, the high-order quantities of motion state required by the downstream decision-making and planning modules can be further obtained. The high-order quantities of motion state can be: speed, acceleration, jerk, etc. For example, after taking the first-order derivative of the Bezier curve, formula (12) can be rewritten to obtain the speed information of the surrounding vehicles at any time step in the prediction time domain. After taking the second-order derivative, the acceleration information of the surrounding vehicles at any time step in the prediction time domain can be obtained for use in the steps of the driving risk warning method.

[0166] Steps S2 to S5 constitute the pericar motion prediction process. Through steps S1 to S5, the predicted trajectory (i.e., motion state) and high-order quantities of the motion state of each pericar are finally obtained.

[0167] The embodiment of the present invention further provides a driving risk warning method, which includes the following steps:

[0168] S6, determine the driving risk of the ego vehicle: the parameters of the surrounding vehicles in the binary Gaussian distribution obtained in step S4 are used to calculate the collision probability between the ego vehicle and the surrounding vehicles in the predicted time domain. The high-order quantity of the motion state obtained in step S5 is used to calculate the expected collision severity of the ego vehicle. The product of the collision probability and the expected collision severity of the ego vehicle is the driving risk of the ego vehicle.

[0169] The driving risk of the ego vehicle is obtained by multiplying two factors, namely, the collision probability between the ego vehicle and surrounding vehicles in the prediction time domain and the expected collision severity of the ego vehicle. The collision probability between the ego vehicle and surrounding vehicles in the prediction time domain and the expected collision severity of the ego vehicle can be calculated simultaneously or one of them can be calculated first and the other later.

[0170] In some embodiments, the parameters of the surrounding vehicle in the binary Gaussian distribution obtained in step S4 are used to calculate the collision probability between the self-vehicle and the surrounding vehicle in the prediction time domain, including:

[0171] The parameters of the bivariate Gaussian distribution of the pericycle obtained in step S4 are given by formula (13):

[0172]

[0173] Among them, the parameter μ of the bivariate Gaussian distribution is x , μ y , σ x , σ y and ρ, respectively represent the x-mean, y-mean, x-standard deviation, y-standard deviation and correlation coefficient of the bivariate Gaussian distribution, with the superscript t f is any time step in the prediction domain, It is the Gaussian distribution representation of the two-dimensional motion of the pericycle.

[0174] The collision probability between the ego vehicle and surrounding vehicles in the predicted time domain is also divided into two scenarios: the ego vehicle is driven by a human driver and the ego vehicle is in an automatic driving state.

[0175] When the ego vehicle is driven by a human driver, there is also subjective driving uncertainty. For this reason, an additional binary Gaussian distribution is set to describe the subjective uncertainty brought by the human driver. The parameters of this binary Gaussian distribution are consistent with the two-dimensional Gaussian distribution that describes the motion of surrounding vehicles. Formula (13) is rewritten, and the collision probability when the ego vehicle is driven by a human driver is calculated by formula (14):

[0176]

[0177] in, They are the parameter representations after introducing the binary Gaussian distribution of the subjective uncertainty of human drivers, and Represents the standard deviation of the Gaussian distribution of subjective uncertainty of human drivers, which is preset;

[0178] When the ego vehicle is in the automatic driving state, the motion state (position, speed, etc.) of the ego vehicle in the prediction time domain can be provided by the automatic driving planning module. A rectangle is used to describe the occupancy of the ego vehicle and surrounding vehicles. The length and width information of the ego vehicle is known, and the length and width information of the surrounding vehicles is provided by the target detection module and the target tracking module in step S1. The collision probability when the ego vehicle is in the automatic driving state is calculated by formula (15):

[0179]

[0180] in, x , and is the vertical and horizontal coordinates of the vehicle at any time step in the prediction time domain, and Provided by the preset autonomous driving planning module, L sub and L sur is the length of the ego vehicle and the length of the surrounding vehicle, W sub and W sur is the width of the vehicle and the surrounding vehicles, L sub and W sub is the default value, L sur and W sur The vehicle-mounted visual data collected in step S1 is obtained through target detection.

[0181] In some embodiments, the high-order motion state quantity obtained in step S5 is used to calculate the expected collision severity of the vehicle, including:

[0182] The expected collision severity of the ego vehicle is calculated by formula (16):

[0183] s(t+Δt)=0.5Mβ 2 (Δv(t+Δt)) 2 #(16)

[0184] Among them, M is the weight of the vehicle, M is the preset value, β is the weight ratio of the surrounding vehicle, M sur is the weight of the vehicle, M sur The empirical value is obtained according to the preset empirical formula after the type and length and width information of the surrounding vehicles are obtained through target detection using the on-board visual data collected in step S1. Δv(t+Δt) is the speed difference between the ego vehicle and the surrounding vehicles between time step t and time increment Δt. The ego vehicle speed is provided by the preset autonomous driving planning module. The surrounding vehicle speed is obtained by taking the first-order derivative of the Bezier curve obtained in step S5.

[0185] S7, determining whether to issue a risk warning: comparing the driving risk with a preset risk threshold, and if the driving risk is greater than the risk threshold, issuing a risk warning.

[0186] In some embodiments, the risk threshold is a dynamic risk threshold, and the dynamic risk threshold has corresponding different risk levels, and the risk levels are obtained through subjective driving risk cognition experiments in different driving scenarios.

[0187] The influencing factors related to the driver's subjective driving risk perception include but are not limited to the driver's driving style, driving skills, driving physical and mental state, etc. For this purpose, a subjective driving risk perception experiment is conducted, and the risk level of the driving scenario is used as a quantitative indicator of the driver's subjective risk perception results, and a corresponding risk threshold is set for each risk level.

[0188] The risk level can be divided into multiple levels, such as level 2, level 3, level 4 or more, and each risk level corresponds to a risk threshold. For example, the risk level is divided into 4 levels, namely absolutely safe, relatively safe, relatively dangerous, and absolutely dangerous, and the corresponding risk thresholds are 50 joules, 40 joules, 30 joules, and 10 joules.

[0189] The risk level can be determined manually according to different driving scenarios. When the driving risk is greater than the risk threshold corresponding to the risk level, a risk warning will be issued.

[0190] In some embodiments, the subjective driving risk perception experiment is performed in the following manner.

[0191] Recruit drivers of different driving experience and gender to participate in the subjective risk perception data collection experiment: The subjective risk perception data collection experiment is based on a driving simulator. The driving scenarios include highway scenarios, urban road scenarios, etc., and the starting point and end point are set in the scenario. Participants are required to complete the designated road section in the corresponding scenario according to their daily driving habits. Finally, the driving data of each participant throughout the process is collected, including the corresponding driving video, the position, speed, acceleration, etc. of the surrounding vehicles.

[0192] Collect the subjective risk perception results of the participants: let each driver participating in the experiment watch a driving video other than his own (the driving video can be divided into several segments), and give corresponding quantitative indicators of risk assessment; in order to prevent the participants' subjective evaluation from becoming habitual, the order of the videos will be randomly disrupted to ensure that the participants will not watch consecutive segments from the same driving video continuously; at the same time, in order to reduce the error of subjective risk assessment results, the participants will watch and evaluate two rounds of driving videos.

[0193] The embodiment of the present invention further provides a vehicle motion prediction and driving risk warning device, the vehicle motion prediction and driving risk warning device comprising:

[0194] The data acquisition and preprocessing module is used to collect vehicle-mounted visual data, serialize the vehicle-mounted visual data for each surrounding vehicle along the acquisition time sequence, obtain sequence data with several segments, and use the sequence data to train the preset motion prediction model to obtain the motion prediction model corresponding to each surrounding vehicle;

[0195] The perimeter vehicle motion prediction module is used to construct an encoder of the motion prediction model. An encoder is assigned to each perimeter vehicle to obtain the aggregated hidden state between the perimeter vehicles. With each vehicle as a node and the spatial interaction relationship between vehicles as an edge, a vehicle spatial interaction module based on a graph structure is constructed, and the spatial interaction weights between the perimeter vehicles are iteratively calculated in a graph structure modeling manner. The decoder of the motion prediction model is constructed to splice the aggregated hidden state between the perimeter vehicles and the spatial interaction weights between the perimeter vehicles. Under the preset Bezier curve control point output paradigm, the binary Gaussian distribution of the control points is regressed to obtain the parameters describing the binary Gaussian distribution, and a preset number of control points are randomly sampled for each perimeter vehicle in the binary Gaussian distribution. According to the control points of each perimeter vehicle, the Bezier curve is calculated. The Bezier curve is the predicted trajectory of the perimeter vehicle. According to the derivation of the Bezier curve, the high-order quantity of the motion state of the perimeter vehicle in the prediction time domain is obtained.

[0196] The driving risk warning module is used to calculate the collision probability between the vehicle and surrounding vehicles in the predicted time domain, calculate the expected collision severity of the vehicle, and take the product of the collision probability and the expected collision severity of the vehicle as the driving risk of the vehicle; compare the driving risk with the preset risk threshold, and if the driving risk is greater than the risk threshold, a risk warning is issued.

[0197] Embodiment 1:

[0198] Taking the following intelligent car road test experiment as an example, the effectiveness of the vehicle motion prediction method and driving risk warning method proposed in the present invention from a vehicle-mounted perspective is illustrated.

[0199] The first step is to calibrate the on-board visual sensor and collect on-board visual data.

[0200] First, the visual sensor needs to be arranged in the smart car. The visual sensor selected in this embodiment is a binocular camera. The arrangement form shown in Figure 2(a) is a desirable one to obtain the vehicle-mounted visual data. At the same time, the Zhang calibration method and Matlab Camera Calibration toolbox are used to calibrate the internal and external parameters of the visual sensor. After the calibration is completed, the smart car is driven on the road section marked with blue lines as shown in Figure 2(b) to collect data.

[0201] The second step is to obtain the surrounding vehicle information based on the target detection module and the target tracking module.

[0202] Based on the data collected in the first step, this embodiment uses the single-stage target detection algorithm YOLOv8 to detect surrounding vehicles. This algorithm is a commonly used target detection algorithm in the industry. The location and category of surrounding vehicles on the image are obtained. The target detection effect of one frame is as follows: Figure 3 As shown, the orange rectangular frame is the target detection frame, and each target detection frame contains the length, width and center point information of the rectangle. The depth information of the surrounding vehicle is estimated by combining the target detection frame information of the surrounding vehicles and the principle of monocular ranging, and then the distance of the surrounding vehicle relative to the vehicle is obtained. The surrounding vehicle number information is obtained based on the target tracking module. In this embodiment, the multi-target tracking algorithm DeepSORT (Deep Learning based ObjectTracking and Data Association for Multi-Object Tracking) based on deep learning is used to track the surrounding vehicles and assign each vehicle a unique vehicle number.

[0203] In the third step, the output results obtained in the second step are saved in a csv file for integration, and serialized according to the vehicle number and along the collection time sequence.

[0204] In this embodiment, each data segment processed in the third and fourth steps is serialized using the sliding window method, and the sliding window is set to 5 seconds, that is, the sequence length is also 5 seconds, of which the first 2 seconds are the observation sequence and the last 3 seconds are the prediction sequence. At the same time, this embodiment uses a method based on manual inspection to handle outliers, that is, if an outlier is found in the sequence data, the sequence is discarded.

[0205] The fourth step is to evenly divide the sequence data obtained in the third step based on the K-fold cross-validation method.

[0206] In this embodiment, K=5 is set, that is, one of the sets is selected as the validation set, and the other four sets are selected as the training set. The models are trained for the data sets obtained by the five division methods, and finally the performance of each model is evaluated using the evaluation index. The FDE evaluation index of formula (1) and the ADE evaluation index of formula (2) are used for the evaluation. Of course, other evaluation indexes of the prior art can also be used to evaluate the model.

[0207] In the fifth step, the encoder of the motion prediction model extracts sequence data features.

[0208] Each vehicle in the scene is assigned an LSTM-based encoder, and the aggregated hidden state between the surrounding vehicles is obtained according to formula (3). In this embodiment, the hidden state aggregation method between the surrounding vehicles is selected as addition, and the dimension of the hidden state is set to [N, T h , d model ], where N is the total number of weekly vehicles in this scenario, T h is the length of the observation sequence. Since the sampling frequency in this embodiment is 10 Hz and the observation time is 3 seconds, T h =30,d model = 128 is the specified hidden state dimension.

[0209] The sixth step is to build a vehicle-space interaction module based on graph structure.

[0210] In order to reflect the spatial interaction relationship between surrounding vehicles in an actual traffic environment, an undirected graph is established with vehicles as nodes and the spatial interaction relationship between vehicles as edges. In this embodiment, the spatial interaction relationship between vehicles is the relative distance between vehicles, that is, Figure 4 The relative distance between car A1 and cars A2 to A5. The condition for the existence of a spatial interaction relationship between the two nodes is that the relative distance between the two nodes is less than the artificially set distance threshold d thresh , where the distance threshold is set to 100 meters. The spatial interaction weights between the surrounding vehicles are iteratively calculated in a graph structure modeling manner. In this embodiment, the corresponding calculations are completed according to formulas (4), (5), and (6), and finally the spatial interaction weights between the surrounding vehicles are obtained. The physical meaning of this weight is that at time step t, the smaller the relative distance between the two vehicles, the stronger the interaction.

[0211] Step 7: Build the decoder part of the motion prediction model and output Bezier curve control points.

[0212] Different from most previous models that directly output the future motion state of surrounding vehicles, this embodiment outputs Bezier curve control points. The control points have specific spatial meanings. For an n-order Bezier curve, there are n+1 control points. This embodiment selects a 5th-order Bezier curve to describe the future trajectory of surrounding vehicles. The predicted time domain length T f =20, that is, the number of control points required for the curve is 6. In the decoder, the aggregated hidden state between the surrounding vehicles obtained in the fifth step and the spatial interaction weights between the surrounding vehicles obtained in the sixth step are concatenated and their dimensions are 128. The uncertainty of the future motion state is described using a binary Gaussian distribution. A multi-layer perceptron is used to receive the concatenation result and output the five parameters μ of the two-dimensional Gaussian distribution used to describe the uncertainty of the motion of the surrounding vehicles at each time step. x , μ y , σ x , σ y and ρ, the output dimension is [N, 20, 5]. Finally, each pericar is randomly sampled 6 times in this two-dimensional Gaussian distribution to obtain the 6 control points of each pericar.

[0213] Step 8: Calculate the Bezier curve and obtain the motion state and its high-order quantities.

[0214] The control point information of the surrounding vehicle in the prediction time domain is obtained in the seventh step, and the Bezier curve can be obtained. This embodiment uses formula (12) to calculate the Bezier curve coordinates corresponding to the control point, that is, the predicted trajectory of the surrounding vehicles. At the same time, according to formula (10) and formula (11), the first-order derivative of the Bezier curve can be obtained and then formula (12) can be rewritten to obtain the speed information of the surrounding vehicles at any time step in the prediction time domain. After obtaining the second-order derivative, the acceleration information of the surrounding vehicles at any time step in the prediction time domain can be obtained for use in the steps of driving risk warning.

[0215] The ninth step is to calculate the components of driving risk, which are the collision probability between the vehicle and surrounding vehicles in the predicted time domain and the expected collision severity of the vehicle.

[0216] For the collision probability between the ego vehicle and the surrounding vehicles in the prediction time domain, in this embodiment, it is assumed that the ego vehicle is driven by a human driver. Therefore, formula (14) is used to take the driver's subjective uncertainty into account when calculating the collision probability. The length of the ego vehicle used in this embodiment is 4.88m and the width is 1.82m. The length and width information of the surrounding vehicles are given by the target detection module. All weights are based on the curb weight. The weight of the car is set to 1200kg and the weight of the truck is set to 6000kg based on the medium-sized truck. Finally, the collision probability is calculated according to formula (15). For the expected collision severity of the ego vehicle, according to formula (16), the speed information of the surrounding vehicle at any time step in the prediction time domain can be known from the prediction results of the surrounding vehicle motion, and the expected collision severity of the ego vehicle is calculated.

[0217] The tenth step is to assess the driving risk of the vehicle and determine whether to issue a risk warning.

[0218] First, the driving risk of the ego vehicle is calculated. The driving risk of the ego vehicle is the product of the collision probability between the ego vehicle and surrounding vehicles in the prediction time domain and the expected collision severity of the ego vehicle.

[0219] Secondly, conduct a driver's subjective risk perception experiment. In this example, 3 male and 3 female drivers with different driving experience (1 year, 3 years, 5 years) were recruited, and a total of 18 participants participated in the experiment. They drove for 30 minutes in the highway scene and urban road scene on the driving simulator, and finally collected a total of 36 driving videos. Slice according to the time window of 5 minutes, let each participant watch the driving video other than his own, and conduct subjective risk assessment.

[0220] In this embodiment, four scene risk levels are set, namely absolute safety, relative safety, relative danger, and absolute danger. A corresponding driving risk threshold is set for each risk level. The risk threshold corresponding to each risk level is determined according to the results of the subjective risk cognition experiment. The driving risk threshold in the absolutely safe scene is 50 joules, the driving risk threshold in the relatively safe scene is 40 joules, the driving risk threshold in the relatively dangerous scene is 30 joules, and the driving risk threshold in the absolutely dangerous scene is 10 joules.

[0221] Finally, the driver selects a certain risk level according to driving habits and driving scenarios, and the risk threshold corresponding to the risk level is the dynamic risk threshold of the current driving scenario. When the driving risk of the vehicle exceeds the dynamic risk threshold of the current driving scenario, a risk warning is issued.

[0222] The present invention is described in detail above in conjunction with the embodiments of the accompanying drawings. A person skilled in the art can make various variations of the present invention according to the above description. Therefore, some details in the embodiments should not be construed as limiting the present invention, and the present invention shall be protected by the scope defined by the attached claims.

Claims

1. A vehicle motion prediction method, characterized in that: The vehicle motion prediction method comprises: S1, collecting vehicle-mounted visual data, serializing the vehicle-mounted visual data for each surrounding vehicle along the order of collection time, obtaining sequence data with a plurality of segments, and using the sequence data to train a preset motion prediction model to obtain a motion prediction model corresponding to each surrounding vehicle; S2, constructing an encoder of the motion prediction model, assigning an encoder to each pericar respectively, and obtaining the aggregated hidden state between the pericars; S3, with each vehicle as a node and the spatial interaction relationship between vehicles as an edge, a vehicle spatial interaction module based on a graph structure is constructed, and the spatial interaction weights between surrounding vehicles are iteratively calculated in a graph structure modeling manner; S4, constructing a decoder of the motion prediction model, splicing the aggregated hidden states between the pericars and the spatial interaction weights between the pericars, regressing the binary Gaussian distribution of the control points under the preset Bezier curve control point output paradigm, obtaining parameters describing the binary Gaussian distribution, and randomly sampling a preset number of control points in the binary Gaussian distribution for each pericar; S5, calculating a Bezier curve according to the control points of each pericar, wherein the Bezier curve is the predicted trajectory of the pericar, and obtaining a high-order quantity of the motion state of the pericar in the predicted time domain by derivation of the Bezier curve.

2. The vehicle motion prediction method according to claim 1, characterized in that: In step S1, vehicle-mounted visual data is collected, including: Arrange one or more visual sensors at preset positions on the vehicle, use a preset calibration algorithm to obtain internal and external parameters of the visual sensor, and after successful calibration, collect vehicle-mounted visual data of the visual sensor; And / or, in step S1, after collecting the vehicle-mounted visual data, the method further includes: Acquire surrounding vehicle information, wherein the surrounding vehicle information includes the distance between each surrounding vehicle and the own vehicle and the vehicle number corresponding to each surrounding vehicle; According to the different vehicle numbers, the vehicle-mounted visual data is serialized for each vehicle along the order of collection time to obtain sequence data; And / or, in step S1, the preset motion prediction model is trained using the sequence data to obtain the motion prediction model corresponding to each surrounding vehicle, including: The sequence data is evenly divided into K parts using a preset K-fold cross-validation method, one of which is selected as a validation set, and K-1 parts are selected as training sets. The first h seconds of each sequence are used as an observation sequence, and the last Hh seconds are used as a prediction sequence, where H is the sequence length and h is less than H; The motion prediction model is trained using the training set, the trained motion prediction model is verified using the verification set, and an evaluation index is calculated. If the evaluation index reaches a preset expected index, the motion prediction model training is considered completed.

3. The vehicle motion prediction method according to claim 2, characterized in that: The obtaining of weekly vehicle information includes: Using a preset target detection module to perform target detection on the vehicle-mounted visual data, and obtaining a target detection frame containing the position and category of each vehicle on each frame image; Estimate the surrounding vehicles according to the target detection frame and the visual sensor parameters to obtain the distance between the surrounding vehicles and the self-vehicle; Using a preset target tracking module to associate and track the surrounding vehicles detected at different times in the vehicle-mounted visual data, and assigning a corresponding vehicle number to each surrounding vehicle; And / or, when the vehicle-mounted visual data is serialized for each vehicle along the order of acquisition time, a sliding window method is used for serialization, and the sliding window is the sequence length; And / or, after the on-board visual data is serialized for each vehicle along the order of collection time, outliers are removed to obtain sequence data; And / or, the evaluation index is one of the final displacement error and the average displacement error or a combination thereof, and the calculation formulas are respectively formula (1) and formula (2): Among them, the medium T f represents the length of the prediction sequence, the subscript t represents each time step in the prediction sequence, and x t and t are the ordinate and abscissa of the true value of the trajectory, and are the ordinate and abscissa of the trajectory prediction value respectively.

4. The vehicle motion prediction method according to claim 1, characterized in that: In step S2, an encoder is assigned to each surrounding vehicle to obtain the aggregated hidden state between the surrounding vehicles, including: An LSTM encoder is assigned to each of the cycle vehicles in the observation sequence. The LSTM encoder is used to extract features from the historical motion state information in the observation sequence. The extracted features are output in the hidden state. The calculation formula for the aggregated hidden state between cycle vehicles is formula (3): in, represents the aggregated hidden state of vehicle i at time step t, j represents another vehicle different from vehicle i, N represents the total number of weekly vehicles, M j represents the hidden state aggregation process from vehicle j to vehicle i, is the hidden state of vehicle j at time step t, is the hidden state of vehicle i at time step t; The hidden state aggregation process adopts one or a combination of concatenation, addition and multiplication methods.

5. The vehicle motion prediction method according to claim 1, characterized in that: In step S3, the vehicle space interaction module is an undirected graph, and the condition for the existence of an edge between two nodes is that the Euclidean distance between the two nodes is less than a preset distance threshold; And / or, in step S3, the spatial interaction weights between surrounding vehicles are iteratively calculated in a graph structure modeling manner, including: Each vehicle is regarded as a node v in the graph structure, where v = {v1, v2, v3, ..., v N }, N is the total number of weekly vehicles; At time step t, establish spatial interaction based on graph structure and define any node v in the undirected graph G i and node v j The symmetric adjacency matrix of is A(i, j), as shown in formula (4): in, Represents node v i With node v j The spatial interaction relationship between thresh is the distance threshold; The unregularized Laplacian matrix L = DA of the undirected graph G is defined as a symmetric matrix as shown in formula (5): Where D(i, j) is the main diagonal of the degree matrix D of the undirected graph G, The remaining elements are all 0; The Laplace matrix defined at time step t is denoted as L t , then the Laplace matrix L of the next time step t+1 The update method of is as follows: Among them, ff trans represents the outer product of rank 2, f is a sparse matrix and satisfies ||f||0<<N. Each time a new non-zero item appears in the j-th row of f, it means that a new week car has been added to the current undirected graph G. The maximum number of nodes in the undirected graph G is the total number of week cars N.

6. The vehicle motion prediction method according to claim 1, characterized in that: In step S4, the formula of the n-th order Bezier curve is as follows: in, represents the Bessel basis function of the i-th order, η represents the Bessel proportional coefficient, P i represents the Bezier curve control point, t f is any time step in the prediction domain, T f represents the predicted sequence length, And / or, in step S4, when regressing the binary Gaussian distribution of the control points, a preset multi-layer perceptron is used to regress the binary Gaussian distribution of the Bezier curve control points to obtain the parameter μ describing the binary Gaussian distribution. x , μ y , σ x , σ y and ρ represent the x-mean, y-mean, x-standard deviation, y-standard deviation, and correlation coefficient of the bivariate Gaussian distribution, respectively.

7. The vehicle motion prediction method according to claim 6, characterized in that: In step S5, the Bezier curve coordinates are given by formula (12): Among them, Y pos represents the position coordinates, B represents the constant Bessel basis matrix, and P represents the predicted control point coordinates; And / or, in step S5, the high-order quantity of the motion state includes at least one of velocity, acceleration and jerk, and the high-order quantity of the motion state is obtained by obtaining a high-order derivative of the Bezier curve.

8. A driving risk warning method, characterized in that: The driving risk warning method comprises: The parameters of the surrounding vehicles in the binary Gaussian distribution obtained by the vehicle motion prediction method described in any one of claims 1 to 7 are used to calculate the collision probability between the self-vehicle and the surrounding vehicles in the prediction time domain; the high-order quantity of the motion state obtained by the vehicle motion prediction method described in any one of claims 1 to 7 is used to calculate the expected collision severity of the self-vehicle, and the product of the collision probability and the expected collision severity of the self-vehicle is used as the driving risk of the self-vehicle; The driving risk is compared with a preset risk threshold, and if the driving risk is greater than the risk threshold, a risk warning is issued.

9. The driving risk warning method according to claim 8, characterized in that: Calculate the collision probability between the vehicle and surrounding vehicles in the prediction time domain, including: The parameters of the bivariate Gaussian distribution of Zhouche are given by formula (13): Among them, the parameter μ of the bivariate Gaussian distribution is x , μ y , σ x , σ y and ρ, respectively represent the x-mean, y-mean, x-standard deviation, y-standard deviation and correlation coefficient of the bivariate Gaussian distribution, with the superscript t f is any time step in the prediction domain, It is the Gaussian distribution representation of the two-dimensional motion of the pericycle; When the vehicle is driven by a human driver, there is still subjective driving uncertainty. Formula (13) is rewritten and the collision probability is calculated by formula (14): in, They are the parameter representations after introducing the binary Gaussian distribution of the subjective uncertainty of human drivers, and represents the standard deviation of the Gaussian distribution of subjective uncertainty of human drivers; When the vehicle is in the automatic driving state, the collision probability is calculated by formula (15): in, x , and is the vertical and horizontal coordinates of the vehicle at any time step in the prediction time domain, and Provided by the preset autonomous driving planning module, L sub and L sur is the length of the ego vehicle and the length of the surrounding vehicle, W sub and W sur is the width of the vehicle and the surrounding vehicles, L sub and W sub is the default value, L sur and W sur The vehicle-mounted visual data collected in step S1 is obtained through target detection; and / or, calculating the expected collision severity of the ego vehicle, including: The expected collision severity of the ego vehicle is calculated by formula (16): s(t+Δt)=0.5Mβ 2 (Δv(t+Δt)) 2 #(16) Among them, M is the weight of the vehicle, M is the preset value, β is the weight ratio of the surrounding vehicle, M sur is the weight of the vehicle, M sur The empirical value is obtained according to the preset empirical formula after the type and length and width information of the surrounding vehicles are obtained through target detection using the on-board visual data collected in step S1. Δv(t+Δt) is the speed difference between the ego vehicle and the surrounding vehicles between time step t and time increment Δt. The ego vehicle speed is provided by the preset autonomous driving planning module, and the surrounding vehicle speed is provided by the high-order quantity of the motion state obtained in step S5.

10. The driving risk warning method according to claim 8, characterized in that: The risk threshold is a dynamic risk threshold, and the dynamic risk threshold has corresponding different risk levels. The risk levels are obtained through subjective driving risk cognition experiments in different driving scenarios.

Images