Vehicle behavior prediction method based on fusion of long short-term memory and multi-graph convolutional network
Through the fusion method of long and short-term memory and multi-graph convolutional networks, combined with virtual reality and traffic simulation technology, the problem of low motion state detection accuracy of autonomous driving cars in hybrid flow environments is solved, and the safety of autonomous driving and traffic safety is improved .
Patent Information
- Application Number
- CN202210381107.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-04-12
- Publication Date
- 2024-12-31
- Estimated Expiration
- 2042-04-12
AI Technical Summary
The existing autonomous vehicle identification and behavior prediction algorithms have problems with inaccurate accuracy, resulting in low accuracy in detecting motion states of autonomous vehicles in hybrid flow environments, affecting traffic safety.
Using a method based on long and short-term memory and multi-graph convolutional network fusion, a hybrid flow vehicle traffic operation simulation model is constructed, combining virtual reality and digital twin technology, an autonomous vehicle driving traffic environment model is established, and a multi-input LSTM layer and full connection are used. Dense layer prediction trajectory, using multi-graph convolution network to predict speed, fusing graph elements of distance, direction and position relationship.
It improves the accuracy of the motion state detection of autonomous driving vehicles, enhances the reliability of the automatic driving active safety functions, and improves the level of traffic safety.
Smart Images
Figure CN115062529B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of traffic safety, and in particular relates to a method for predicting the behavior of an autonomous driving vehicle based on the fusion of a novel long short-term memory and multi-graph convolutional network. Background Art
[0002] Due to problems such as imperfect video detection functions, many autonomous driving vehicles have collided with pedestrians and non-motor vehicles. The safety issue of autonomous driving has important research significance in the field of traffic safety.
[0003] According to the latest global autonomous vehicle forecast report, the shipment volume of vehicles with autonomous driving functions is increasing, but the current technology has problems with the accuracy of autonomous vehicle recognition and behavior prediction algorithms. Vehicles with autonomous driving assistance functions cannot make timely predictions in the design of active safety protection. Their image recognition algorithms consider only a single factor, which may produce inconsistent results on the recognition results, resulting in low prediction accuracy. Summary of the invention
[0004] Technical problem solved: The purpose of the present invention is to overcome the shortcomings of the prior art and provide a method for predicting the behavior of an autonomous driving vehicle based on the fusion of a new long short-term memory and a multi-graph convolutional network. The method integrates a multi-input network of a new long short-term memory fully connected dense layer and a multi-graph convolutional network that considers the relationship between distance, direction and position to detect the motion state of an autonomous driving vehicle in a mixed flow environment of vehicles with autonomous driving functions, solve the shortcomings of the current active safety of autonomous driving functions, improve the detection accuracy of the motion state of autonomous driving vehicles in the field of video image recognition, and improve the level of traffic safety.
[0005] Technical solution:.
[0006] A vehicle behavior prediction method based on long short-term memory and multi-graph convolutional network fusion, the vehicle behavior prediction method comprising the following steps:
[0007] S1, according to the motion state of the vehicle with the automatic driving assistance function, establish a mixed flow vehicle traffic operation simulation, and map the mixed flow vehicle traffic operation simulation state to the constructed automatic driving vehicle driving traffic environment model; in the constructed automatic driving vehicle driving traffic environment model, connect the virtual reality head-mounted device and the driving simulation device to construct a virtual reality automatic driving vehicle driving experimental environment;
[0008] S2, the experimenter wears a virtual reality helmet and conducts an experiment on a fixed car device. The traffic scene of the self-driving car is directly projected into the helmet worn by the experimenter. The real driving conditions of the self-driving car are simulated through a cyclic experiment. The influence of the behavior of surrounding vehicles on the experimenter's driving is simulated by a computer. The experimenter crosses the street under different conditions in the experiment; the experimenter's coordinates (x0, y0), head direction o0 and distance d0 to the neighboring vehicle are recorded at intervals, and a number of sample data are obtained through multiple experimental records;
[0009] S3, based on the long short-term memory network FCD-LSTM composed of multi-input LSTM layers and fully connected dense layers, the trajectory prediction model of the autonomous driving vehicle is constructed. The coordinates (x0, y0), head direction o0 and distance d0 to the neighboring vehicle in step S2 are input into the LSTM layer. The autonomous driving vehicle trajectory prediction model is trained by two identical loss functions under the regularization mechanism. After the training is completed, the time series data S = {x0, y0, o0, d0} is used to predict the future trajectory T = {x f ,y f};
[0010] S4, collects existing autonomous vehicle big data, uses Z-score normalization processing on the trajectory data of each autonomous vehicle recorded by GPS, and retains the direction information of the link; determines the edge weights including distance information, traffic connectivity, and functional similarity in the graph convolutional network to reflect the network geometry, uses the multi-graph convolutional network DDP-GCN to merge these graph elements, inputs the processed trajectory data into DDP-GCN training, and merges the distance W D , direction W θ and position relationship W P , predicting the speed of the autonomous car after training and testing the model;
[0011] S5, verify and test the result of the long short-term memory network FCD-LSTM obtained in step S3, input the obtained autonomous driving car VR trajectory data set into the multi-graph convolutional network DDP-GCN in step S4, fuse the training results of different data sets, obtain the speed prediction data of the autonomous driving car, and complete the prediction of the behavior of the autonomous driving car.
[0012] Furthermore, in step S1, a mixed flow vehicle traffic operation simulation is established according to the motion state of the vehicle with the automatic driving assistance function, and the process of mapping the mixed flow vehicle traffic operation simulation state to the established automatic driving vehicle driving traffic environment model includes the following steps:
[0013] According to the motion state of vehicles with automatic driving assistance functions, a high-precision mixed-flow vehicle traffic simulation is established, and the traffic network path simulation is completed by using one of the three methods: traffic simulation software SUMO to convert OSM map, VISSIM traffic simulation software road network conversion, or OpenDRIVE standard high-precision map conversion;
[0014] The vehicle following model is input into SUMO to describe the simulation of traffic flow operation status;
[0015] Utilize a variety of modeling software and rendering platforms, build a traffic environment model for autonomous vehicles through GIS raster vector data modeling methods or BIM technology high-precision modeling methods, import it into the Unity3D engine for fusion rendering, and map the mixed flow vehicle traffic operation simulation conditions into the constructed three-dimensional model.
[0016] Furthermore, in the constructed autonomous driving vehicle driving traffic environment model, the process of connecting the virtual reality head-mounted device and the driving simulation device to construct a virtual reality autonomous driving vehicle driving experimental environment includes:
[0017] The mixed-flow vehicle traffic operation simulation operation file is converted into command line code, and the conversion results are integrated using the traffic simulation software SUMO. The road network is edited using the netedit function of SUMO to obtain a high-precision simulation of mixed-flow vehicle traffic operation. Combined with the traffic movement characteristics of different scenarios, the corresponding following model is selected, and the traffic simulation results that conform to vehicle kinematics are edited in SUMO;
[0018] The constructed 3D model of the driving environment of the autonomous vehicle is integrated, spliced and rendered in sequence, and a variety of information display tools are added using the control function of Unity3D to obtain a digital twin modeling system;
[0019] The Traci interface of SUMO traffic simulation software is used for secondary development. For the mixed-flow vehicle traffic operation simulation and the traffic flow simulation results that conform to vehicle kinematics, the Visual Studio development tool is used to write scripts, import the System, System.Collections, CodingConnected.TraCI.NET, and UnityEngine databases, and map the simulation data in SUMO to Unity3D, where the mixed-flow vehicle traffic operation simulation is mapped in the road model, and the traffic flow simulation results that conform to vehicle kinematics are mapped in the vehicle model.
[0020] After the autonomous driving car driving simulation modeling and mapping is completed, the virtual reality headset and driving simulator device are connected to restore the autonomous driving car driving environment under mixed flow in the device. After the experimenter wears the device, he drives in the driving simulation device. The vehicle model mapped by the traffic simulation results and the autonomous driving car model driven by the experimenter move simultaneously in the mixed flow driving scene of Unity3D to realize the virtual reality digital twin connection.
[0021] Furthermore, in step S2, the experimenter's coordinates (x0, y0), head direction (o0) and distance to neighboring vehicles (d0) are recorded at intervals of 0.1 seconds, and sufficient sample data are recorded through multiple experiments.
[0022] Furthermore, in step S3, the architecture of the autonomous driving vehicle trajectory prediction model is as follows:
[0023] LSTM hidden layer: H l =L(S,W l ,σ)
[0024] Dense hidden layer variables: H d =f(H l ,C,W d ,σ)
[0025] Output layer: T = f(H d ,W o ,φ)
[0026] Among them, W l ,W d ,W o are the weights of the hidden LSTM, hidden dense layer, and output dense layer, σ and φ are the nonlinear ReLU and Sigmoid activation functions, and L and f are the calculation functions of the LSTM and dense layer, respectively.
[0027] Furthermore, in step S4, edge weights including distance information, traffic connectivity, and functional similarity are determined in the graph convolutional network to reflect the network geometry, and the multi-graph convolutional network DDP-GCN is used to merge these graph elements. The processed trajectory data is input into the DDP-GCN training, and the merged distance W D , direction W θ and position relationship W P The process includes the following steps:
[0028] The distance d(i,j) is taken as the shortest link distance on the path. When the vectors on the link are directly connected, that is, Calculate the average link length After all pairwise distances are calculated, the Gaussian kernel threshold is used to define the distance W D :
[0029]
[0030] Where σ and κ are hyperparameters;
[0031] Direction W θ Defined as:
[0032]
[0033] definition is a weighted adjacency matrix containing the encounter position information of two extended link vectors, defined as follows:
[0034]
[0035] Among them, k∈{1,2,3,4}.
[0036] Furthermore, in step S2, while recording the time series data of the experimenter, the environmental variables in the virtual reality scene are recorded, and the environmental variables include road type, road speed limit, lane width, weather conditions, time of day, and vehicle flow rate. The recorded environmental variables are used as parameter input values of the image recognition algorithm of the autonomous driving vehicle.
[0037] Beneficial effects:
[0038] The method for predicting the behavior of an autonomous vehicle based on the fusion of long short-term memory and multi-graph convolutional network of the present invention, based on the long short-term memory network LSTM, considers the historical trajectory of the autonomous vehicle, the head direction of the driver, and the distance from the neighboring vehicles as input data to form a new long short-term memory network FCD-LSTM, and trains the model to predict the future trajectory of the autonomous vehicle; based on the graph convolutional network GCN, the edge weight is determined to reflect the network geometry, and the three graph elements of distance, direction, and position relationship are merged to form a multi-graph convolutional network DDP-GCN, and the training model is used to predict the future speed of the autonomous vehicle. The present invention integrates the multi-input network of the new long short-term memory fully connected dense layer and the multi-graph convolutional network considering distance, direction and position relationship, detects the motion state of the autonomous vehicle in the current road environment and the mixed flow environment of vehicles with autonomous driving functions, solves the current shortcomings of the active safety of the autonomous driving function, improves the detection accuracy of the motion state of the autonomous vehicle in the field of video image recognition, improves the reliability of the active safety function of the autonomous driving, and improves the level of traffic safety. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] Figure 1 This is a framework diagram of the autonomous driving vehicle behavior prediction method based on the fusion of long short-term memory and multi-graph convolutional network;
[0040] Figure 2Flowchart constructed for a VR scenario of autonomous vehicle driving under mixed flow. DETAILED DESCRIPTION
[0041] The following examples will enable those skilled in the art to more fully understand the present invention, but are not intended to limit the present invention in any way.
[0042] See also Figure 1 and Figure 2 The best embodiment of the present invention is an autonomous driving vehicle behavior prediction method based on the fusion of long short-term memory and multi-graph convolutional network, including a novel long short-term memory network autonomous driving vehicle trajectory prediction method and a multi-graph convolutional network autonomous driving vehicle speed prediction method.
[0043] The novel LSTM autonomous vehicle trajectory prediction method of this embodiment is based on the LSTM network. In addition to considering the historical trajectory of the autonomous vehicle, it adds the head direction of the driver of the autonomous vehicle and the distance to neighboring vehicles as input data, and trains the model to predict the future trajectory of the autonomous vehicle by fusing it with environmental variables.
[0044] The two types of data, head direction and distance to neighboring vehicles, are collected by the experimenter using a car simulation device in a virtual reality scene. The scene simulates the driving environment of an urban self-driving car, with a mixed flow of human-driven and self-driving cars. A variety of digital modeling technologies are combined, edited and rendered in the modeling engine, and the virtual reality headset and car simulation device are connected to build an experimental scene.
[0045] Environmental variables are scene contents other than the three time series data of the historical trajectory of the autonomous vehicle, the driver's head direction, and the distance to neighboring vehicles. They include road type, road speed limit, lane width, weather conditions, time of day, and vehicle flow rate. Recording the above variables can be used as parameter input values for the image recognition algorithm of the autonomous vehicle.
[0046] The multi-graph convolutional network autonomous driving car speed prediction method of this embodiment requires determining edge weights in the graph convolutional network to reflect the network geometry, and proposes using a multi-graph convolutional network to merge three graph elements, namely, distance, direction, and position relationship, to extract effective information from the collected autonomous driving car big data, and use the network to train a model and predict the future speed of the autonomous driving car.
[0047] Autonomous driving car big data refers to the vehicle data of existing autonomous driving car brands within a specific period of time, including the time, car information, longitude coordinates, latitude coordinates and other information of each autonomous driving car.
[0048] After the new long short-term memory network trains the model, the experimenter's future driving trajectory is predicted and input into the multi-graph convolutional network to predict the experimenter's future driving speed. This is an autonomous driving vehicle behavior prediction method based on the fusion of the new long short-term memory and multi-graph convolutional network.
[0049] The virtual reality environment is a highly restored three-dimensional model of the driving environment of an autonomous vehicle built using data from different software and platforms. It restores the mixed traffic conditions of manually driven and autonomous vehicles in traffic simulation software, simulates the actual driving conditions of autonomous vehicles, and combines simulation and modeling with virtual reality equipment.
[0050] 1. Building a virtual reality scene for autonomous vehicles
[0051] like Figure 2 As shown, the construction of the virtual reality scene of autonomous driving car driving includes three parts: mixed flow vehicle traffic operation simulation, three-dimensional modeling of the autonomous driving car driving environment, and virtual reality digital twin connection.
[0052] (1) Simulation of mixed-flow vehicle traffic operation
[0053] Mixed-flow vehicle traffic operation simulation, including traffic network path simulation and vehicle flow operation status simulation.
[0054] (1.1) Traffic network path simulation can be completed by using one of three methods: converting OSM maps using traffic simulation software SUMO, converting road networks using VISSIM traffic simulation software, or converting high-precision maps using OpenDRIVE standards.
[0055] First, for SUMO traffic simulation software: select the required autonomous driving vehicle driving area in the OSM map, export the map in map.osm format, open the command line tool in the path where the map in this format is located, and enter the code: netconvert--osm-files map.osm-o map.net.xml. After running, you will get the map.net.xml file, which is the road network file of SUMO traffic simulation software.
[0056] Second, for VISSIM traffic simulation software: In VISSIM traffic simulation software, use the road segment connector editing function to draw the road network structure of mixed flow traffic according to the required scene traffic connection method, and edit the elevation of the road network according to the situation in the area, so that the road network file corresponds to the real environment in data. After editing, the VISSIM traffic simulation software road network file of map.inpx is obtained. Open the command line tool in the path where the road network file is located, and enter the code: netconvert--vissim-file=map.inpx--output-file=map.net.xml. After running, the map.net.xml file is obtained, which is the road network file of SUMO traffic simulation software.
[0057] Third, for the conversion of OpenDRIVE standard high-precision maps: according to the selected autonomous driving car area, based on real road data, combined with OpenSCENARIO and OpenCRG standards, program the hybrid flow road network file map.xodr of the OpenDREIVE standard, open the command line tool in the path where the road network file is located, and enter the code: netconvert--opendrive map.xodr-o map.net.xml. After running, you will get the map.net.xml file, which is the road network file of the SUMO traffic simulation software.
[0058] (1.2) Traffic flow operation status simulation
[0059] Traffic flow operation state simulation is to input various vehicle following models into SUMO traffic simulation software, which can describe in detail the acceleration, deceleration, lane change, overtaking, queuing and other situations of vehicles in mixed flow operation. The model is based on the behavioral relationship between the vehicle and the preceding vehicle to reflect the vehicle behavior in mixed flow traffic scenarios. The specific description of the following model is as follows:
[0060] The Krauss model is based on the safety distance. In this model, the following vehicle chooses the speed to be maintained in the current time period according to the distance and speed between the leading and following vehicles in the previous time period.
[0061] In the IDM model, the vehicle's acceleration at the next moment is determined by the deviation between the current speed and the maximum speed, as well as the deviation between the current distance to the vehicle in front and the expected following distance. The relative value of the deviation between the actual value at this moment and the expected value is measured, and power calculation is used to simulate the driver's sensitivity to this deviation. Ultimately, the following acceleration is determined, and then the speed and position at the next moment are determined. All vehicles follow this rule to complete the traffic flow following modeling.
[0062] The Wiedemann model uses a series of thresholds to distinguish different vehicle driving behaviors, uses the distance and relative speed between the current vehicle and the vehicle in front to determine the driving state of the vehicle, and finally calculates the acceleration based on the acceleration formula under this state. This model fully considers the impact and constraints of the driver's physiological and psychological factors on driving behavior and the resulting different driving behaviors, and is closer to the actual situation in terms of modeling methods.
[0063] The Bkerner model is based on random three-item traffic flow modeling and simulates the choice of whether a vehicle continues to accelerate or decelerate or maintain its current speed during random acceleration or random deceleration, taking into account the driver's reaction delay.
[0064] The PWagner model is an action point model of human behavior. The driver can randomly decide whether to change the current acceleration at any time step.
[0065] (2) 3D modeling of the driving environment of autonomous vehicles
[0066] Three-dimensional modeling of the driving environment of autonomous vehicles uses different software and platform data to build a highly restored traffic environment model of autonomous vehicles. The modeling technologies used include: GIS raster vector data modeling, automatic modeling of map road network data, high-precision modeling with BIM technology, and modeling with drone oblique photography. After the modeling is completed, it is imported into the Unity3D engine for fusion rendering.
[0067] GIS raster vector data modeling, the spatial data model in the GIS geographic information system is an abstraction of the real world using a series of data objects that can support spatial query and analysis, map editing and display, that is, abstract modeling of the real world. The result is a spatial data model, including: vector data model, raster data model, raster integration model, mosaic data model, object-oriented data model, etc. Among them, the spatial data of the raster data model is easy to overlay and combine, which is conducive to the matching application and analysis of remote sensing data.
[0068] Use ArcGIS or QGIS geographic information system software to import the tif format image file of the selected autonomous driving car area into the software, select the appropriate projection coordinate system according to the geographical location, and then import the shp format vector graphic file of the buildings and roads within the range, edit the height and position in the software, and import the edited tif and shp files into Blender or other 3D modeling software. Use software plug-ins or built-in interfaces to generate 3D terrain and building models with geographic information, edit the elevation and terrain style according to the actual situation of the location, and restore the real scene.
[0069] Automatic modeling of map road network data is to crawl the required terrain, buildings, roads and other files from the existing map resources on the Internet, and use the plug-in function in the 3D modeling software to automatically model the area where the self-driving car is located.
[0070] Select the required range in the area on the Google Earth map, switch to the 3D perspective, use the RenderDoc software to capture the map data within the corresponding range, and obtain an image file in rdc format. Install the MapsModelImporter plug-in in the Blender software and import the rdc file to generate a 3D model within the selected range.
[0071] In the Shuijingzhu or BIGEMAP map software, download the tif or shp file of the selected range, and obtain the three-dimensional model according to the above-mentioned GIS raster vector data modeling method.
[0072] The high-precision modeling of BIM technology is to build a regional building information model by integrating the digital and informational models of the building and editing with a variety of modeling software. In three-dimensional modeling software such as Autodesk Revit, CityEngine, Autodesk3ds Max, SketchUp, Blender, Autodesk Infraworks, Rhinoceros 3D, etc., the modeling work is carried out and detailed according to the road design drawings to restore the three-dimensional structure of the scene and the appearance of the surrounding buildings and road environment.
[0073] Drone oblique photography modeling uses drones to simultaneously collect images from one vertical, four oblique, and five different perspectives above the ground to obtain the top surfaces and high-resolution textures of buildings and roads. It can truly reflect the scene conditions, obtain texture information with high precision, and generate real three-dimensional models through positioning, fusion, modeling and other technologies.
[0074] Use a DJI Mavic 2 drone to fly around the points of interest in the selected area, use a 3s continuous shooting mode, adjust the orbit radius according to the size of the scene, take photos from different angles according to the above specifications, and import the photos into the Smart3D ContextCapture software after they are taken. The software's aerial triangulation algorithm is efficient, and its unique distributed aerial triangulation algorithm makes the measurement results more accurate and robust. It can realize parallel processing of the entire process of oblique real-scene 3D big data. After importing, oblique photography 3D modeling can be performed according to needs and software operating specifications.
[0075] (3) Virtual reality digital twin connection
[0076] The virtual reality digital twin connection is used to connect the three-dimensional model of the driving environment of the autonomous vehicle, map the traffic operation simulation status of the mixed flow vehicles to the constructed three-dimensional model, and connect the virtual reality head-mounted device and the driving simulation device to restore the real scene of the mixed autonomous driving vehicle. The mapping is to map the running status of the mixed flow vehicles, the running results of the traffic simulation software, into the constructed high-reduction three-dimensional scene using a digitized script editing method, where the traffic path data is mapped in the road model of the driving environment of the autonomous vehicle, and the vehicle motion data is mapped in the vehicle model.
[0077] The mixed-flow vehicle traffic operation simulation operation files obtained by different methods are converted into command line codes and integrated into the traffic simulation software SUMO. The road network is edited using the netedit function of SUMO to obtain a high-precision simulation of mixed-flow vehicle traffic operation. Combined with the traffic movement characteristics of different scenarios, the appropriate following model is selected, and the traffic simulation results that conform to vehicle kinematics are edited in SUMO.
[0078] The three-dimensional model of the driving environment of the autonomous driving car is constructed by different methods. After integration, splicing, rendering and other processes, a variety of information display tools are added using the control function of Unity3D to obtain a digital twin modeling system.
[0079] The Traci interface of the SUMO traffic simulation software is used for secondary development. The above-mentioned mixed-flow vehicle traffic operation simulation and the traffic flow simulation results that conform to vehicle kinematics are scripted using the Visual Studio development tool and imported into the System, System.Collections, CodingConnected.TraCI.NET, UnityEngine and other libraries. The simulation data in SUMO is mapped in Unity3D, where the mixed-flow vehicle traffic operation simulation is mapped in the road model, and the traffic flow simulation results that conform to vehicle kinematics are mapped in the vehicle model.
[0080] After the above-mentioned autonomous driving car driving simulation modeling and mapping is completed, connect virtual reality headsets and driving simulators and other devices to restore the autonomous driving car driving environment under mixed flow in the devices. After the experimenter wears the equipment, he drives in the driving simulation device. The vehicle model mapped by the traffic simulation results and the autonomous driving car model driven by the experimenter move simultaneously in the mixed flow driving scene of Unity3D, achieving the effect of virtual reality digital twin connection.
[0081] 2. Predicting the behavior of autonomous vehicles
[0082] After the autonomous driving car driving virtual reality scene is built, the autonomous driving car behavior prediction method based on the fusion of the new long short-term memory and multi-graph convolutional network is carried out in the following steps:
[0083] First, data collection:
[0084] A highly realistic three-dimensional model of the driving environment of an autonomous vehicle is built using data from different software and platforms. The mixed traffic conditions of manually driven and autonomous vehicles are restored in the traffic simulation software, the real driving conditions of an autonomous vehicle are simulated, and simulation and modeling are combined with virtual reality equipment.
[0085] The driving state of an autonomous vehicle is simulated in a virtual reality environment. The experimenter wears a virtual reality helmet and conducts the experiment on a fixed car device. The traffic scene of the autonomous vehicle is directly projected into the helmet worn by the experimenter. The real driving condition of the autonomous vehicle is simulated through a cyclic experiment. The influence of the behavior of surrounding vehicles on the experimenter's driving is simulated by a computer. The experimenter crosses the street under different conditions in the experiment. The experimenter's coordinates, head direction, and distance to neighboring vehicles are recorded at intervals of 0.1 seconds. Sufficient sample data are recorded through multiple experiments.
[0086] In addition to recording the experimenter's time series data, the environmental variables in the virtual reality scene are recorded, including road type, road speed limit, lane width, weather conditions, time of day, and vehicle flow rate. The above variables can be used as parameter input values for the image recognition algorithm of the autonomous vehicle.
[0087] Second, predict future trajectories
[0088] The long short-term memory network LSTM is modified for the new long short-term memory network autonomous vehicle trajectory prediction method, and a new framework FCD-LSTM consisting of a multi-input LSTM layer and a fully connected dense layer is proposed to predict the output of the next step coordinates of the autonomous vehicle. The coordinates (x0, y0), head direction (o0) and distance to neighboring vehicles (d0) recorded during the experiment are used as input to the LSTM layer. The data output in the LSTM layer will be fused with the information of the environmental variables and enter a series of fully connected dense layers to predict the trajectory (x f ,y f ), using time series data (S = {x0, y0, o0, d0}) to predict the future trajectory of the autonomous vehicle (T = {x f ,y f The neural network architecture of}) is as follows:
[0089] LSTM hidden layer: H l =L(S,W l ,σ)
[0090] Dense hidden layer variables: H d =f(H l ,C,W d ,σ)
[0091] Output layer: T = f(H d ,W o ,φ)
[0092] Where W l ,W d ,W o are the weights of the hidden LSTM, hidden dense layer, and output dense layer, σ and φ are the nonlinear ReLU and Sigmoid activation functions, and L and f are the LSTM and dense layers, respectively.
[0093] The above model is supervised by two identical loss functions under regularization. Both loss functions are defined as the mean square error of the predicted coordinates. By using the loss function after the LSTM layer, the framework can be trained faster.
[0094] Third, prediction speed
[0095] For the multi-graph convolutional network autonomous driving car speed prediction method, when using the graph convolutional network GCN to predict traffic, it is necessary to determine the edge weights to reflect the network geometry. In the present invention, the speed of the autonomous driving car needs to be predicted. Considering the edge weights such as distance information, traffic connectivity, and functional similarity, a multi-graph convolutional network is used to merge these graph elements, that is, a multi-graph convolutional network DDP-GCN of distance, direction, and position relationship.
[0096] The distance d(i,j) is taken as the shortest link distance on the path. When the vectors on the link are directly connected, that is, Calculate the average link length After all pairwise distances are calculated, the Gaussian kernel threshold is used to define the distance W D , where σ and κ are hyperparameters:
[0097]
[0098] Long-distance links are related to direction. Links in the same direction are more correlated than links in opposite directions. Direction W θ Defined as:
[0099]
[0100] The relevance of links depends on how they are connected, and their positional relationship W P It can provide additional information for distance and understand how links affect each other. DThe shortest path case is considered in , but two links can be connected through different paths. Considering this situation, we define is a weighted adjacency matrix containing the encounter position information of two extended link vectors, defined as follows, where k∈{1,2,3,4}:
[0101]
[0102] The above multi-graph convolutional network DDP-GCN collects big data of autonomous vehicles in the selected area, processes and calculates the average speed of each link based on the trajectory data of each autonomous vehicle recorded by GPS. The selected area needs to contain a complex network of a large number of intersections and traffic signals so that the links have directional information. The collected data set is normalized using Z-score. 70% of the data is used for training, 10% for validation, and 20% for testing. After the settings are completed, the multi-graph convolutional network training begins.
[0103] The above process verifies and tests the training results of the new long short-term memory network FCD-LSTM, predicts the coordinates of the experimenter's driving experiment trajectory, and obtains the autonomous driving car virtual reality experiment trajectory dataset. The multi-graph convolutional network DDP-GCN model is then trained based on this dataset, and the training results of different datasets are integrated to obtain the speed prediction result of the autonomous driving car.
[0104] In summary, the method for predicting the behavior of an autonomous vehicle based on the fusion of a novel long short-term memory and a multi-graph convolutional network of the present invention uses a novel long short-term memory network to predict the motion trajectory of an autonomous vehicle in a mixed flow environment with vehicles having autonomous driving functions; uses a multi-graph convolutional network that considers the relationship between distance, direction, and position to predict the speed of an autonomous vehicle; combines traffic simulation, three-dimensional modeling, and virtual reality technology to construct a virtual reality environment for autonomous vehicles under mixed flow, and collects data through experiments. The method of integrating a novel long short-term memory network and a multi-graph convolutional network to study the motion laws of autonomous vehicles can predict the motion state of autonomous vehicles in the current road environment and in the future mixed flow environment of autonomous vehicles, improve the reliability of the active safety function of autonomous driving, and enhance the level of road safety.
[0105] Those of ordinary skill in the art will appreciate that the methods or steps of the embodiments described in conjunction with the embodiments disclosed herein can be implemented in a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.
Claims
1. A vehicle behavior prediction method based on the fusion of long short-term memory and multi-graph convolutional network, characterized in that: The vehicle behavior prediction method comprises the following steps: S1, according to the motion state of the vehicle with the automatic driving assistance function, establish a mixed flow vehicle traffic operation simulation, and map the mixed flow vehicle traffic operation simulation state to the constructed automatic driving vehicle driving traffic environment model; in the constructed automatic driving vehicle driving traffic environment model, connect the virtual reality head-mounted device and the driving simulation device to construct a virtual reality automatic driving vehicle driving experimental environment; S2, the experimenter wears a virtual reality helmet and conducts an experiment on a fixed car device. The traffic scene of the self-driving car is directly projected into the helmet worn by the experimenter. The real driving conditions of the self-driving car are simulated through a cyclic experiment. The influence of the behavior of surrounding vehicles on the experimenter's driving is simulated by a computer. The experimenter crosses the street under different conditions in the experiment; the experimenter's coordinates (x0, y0), head direction o0 and distance d0 to the neighboring vehicle are recorded at intervals, and a number of sample data are obtained through multiple experimental records; S3, based on the long short-term memory network FCD-LSTM composed of multi-input LSTM layers and fully connected dense layers, the trajectory prediction model of the autonomous driving vehicle is constructed. The coordinates (x0, y0), head direction o0 and distance d0 to the neighboring vehicle in step S2 are input into the LSTM layer. The autonomous driving vehicle trajectory prediction model is trained by two identical loss functions under the regularization mechanism. After the training is completed, the time series data S = {x0, y0, o0, d0} is used to predict the future trajectory T = {x f ,y f }; S4, collects existing autonomous vehicle big data, uses Z-score normalization processing on the trajectory data of each autonomous vehicle recorded by GPS, and retains the direction information of the link; determines the edge weights including distance information, traffic connectivity, and functional similarity in the graph convolutional network to reflect the network geometry, uses the multi-graph convolutional network DDP-GCN to merge these graph elements, inputs the processed trajectory data into DDP-GCN training, and merges the distance W D , direction W θ and position relationship W P , predicting the speed of the autonomous car after training and testing the model; S5, verifying and testing the result of the long short-term memory network FCD-LSTM obtained in step S3, inputting the obtained autonomous driving car VR trajectory data set into the multi-graph convolutional network DDP-GCN in step S4, fusing the training results of different data sets, obtaining the speed prediction data of the autonomous driving car, and completing the prediction of the behavior of the autonomous driving car; In step S3, the architecture of the autonomous driving vehicle trajectory prediction model is as follows: LSTM hidden layer: H l =L(S,W l ,σ) Dense hidden layer variables: H d =f(H l ,C,W d ,σ) Output layer: T = f(H d ,W o ,φ) Among them, W l ,W d ,W o are the weights of the hidden LSTM, hidden dense layer, and output dense layer, σ and φ are the nonlinear ReLU and Sigmoid activation functions, L and f are the calculation functions of the LSTM and dense layer, respectively; In step S4, edge weights including distance information, traffic connectivity, and functional similarity are determined in the graph convolutional network to reflect the network geometry. The multi-graph convolutional network DDP-GCN is used to merge these graph elements. The processed trajectory data is input into the DDP-GCN training to merge the distance W. D , direction W θ and position relationship W P The process includes the following steps: The distance d(i,j) is taken as the shortest link distance on the path. When the vectors on the link are directly connected, that is, Calculate the average link length After all pairwise distances are calculated, the Gaussian kernel threshold is used to define the distance W D : Where σ and κ are hyperparameters; Direction W θ Defined as: definition is a weighted adjacency matrix containing the encounter position information of two extended link vectors, defined as follows: Among them, k∈{1,2,3,4}.
2. The vehicle behavior prediction method based on long short-term memory and multi-graph convolutional network fusion according to claim 1 is characterized in that: In step S1, a mixed flow vehicle traffic operation simulation is established according to the motion state of the vehicle with the automatic driving assistance function, and the process of mapping the mixed flow vehicle traffic operation simulation state to the established automatic driving vehicle driving traffic environment model includes the following steps: According to the motion state of vehicles with automatic driving assistance functions, a high-precision mixed-flow vehicle traffic simulation is established, and the traffic network path simulation is completed by using one of the three methods: traffic simulation software SUMO to convert OSM map, VISSIM traffic simulation software road network conversion, or OpenDRIVE standard high-precision map conversion; The vehicle following model is input into SUMO to describe the simulation of traffic flow operation status; Utilize a variety of modeling software and rendering platforms, build a traffic environment model for autonomous vehicles through GIS raster vector data modeling methods or BIM technology high-precision modeling methods, import it into the Unity3D engine for fusion rendering, and map the mixed flow vehicle traffic operation simulation conditions into the constructed three-dimensional model.
3. The vehicle behavior prediction method based on long short-term memory and multi-graph convolutional network fusion according to claim 1 is characterized in that: In the constructed autonomous driving car driving traffic environment model, the process of connecting the virtual reality head-mounted device and the driving simulation device to construct a virtual reality autonomous driving car driving experimental environment includes: The mixed-flow vehicle traffic operation simulation operation file is converted into command line code, and the conversion results are integrated using the traffic simulation software SUMO. The road network is edited using the netedit function of SUMO to obtain a high-precision simulation of mixed-flow vehicle traffic operation. Combined with the traffic movement characteristics of different scenarios, the corresponding following model is selected, and the traffic simulation results that conform to vehicle kinematics are edited in SUMO; The constructed 3D model of the driving environment of the autonomous vehicle is integrated, spliced and rendered in sequence, and a variety of information display tools are added using the control function of Unity3D to obtain a digital twin modeling system; The Traci interface of SUMO traffic simulation software is used for secondary development. For the mixed-flow vehicle traffic operation simulation and the traffic flow simulation results that conform to vehicle kinematics, the Visual Studio development tool is used to write scripts, import the System, System.Collections, CodingConnected.TraCI.NET, and UnityEngine databases, and map the simulation data in SUMO to Unity3D, where the mixed-flow vehicle traffic operation simulation is mapped in the road model, and the traffic flow simulation results that conform to vehicle kinematics are mapped in the vehicle model. After the autonomous driving car driving simulation modeling and mapping is completed, the virtual reality headset and driving simulator device are connected to restore the autonomous driving car driving environment under mixed flow in the device. After the experimenter wears the device, he drives in the driving simulation device. The vehicle model mapped by the traffic simulation results and the autonomous driving car model driven by the experimenter move simultaneously in the mixed flow driving scene of Unity3D to realize the virtual reality digital twin connection.
4. The vehicle behavior prediction method based on long short-term memory and multi-graph convolutional network fusion according to claim 1 is characterized in that: In step S2, the experimenter's coordinates (x0, y0), head direction (o0), and distance to neighboring vehicles (d0) are recorded at intervals of 0.1 seconds, and sufficient sample data are recorded through multiple experiments.
5. The vehicle behavior prediction method based on long short-term memory and multi-graph convolutional network fusion according to claim 1 is characterized in that: In step S2, while recording the experimenter's time series data, the environmental variables in the virtual reality scene are recorded. The environmental variables include road type, road speed limit, lane width, weather conditions, time of day, and vehicle flow rate. The recorded environmental variables are used as parameter input values of the image recognition algorithm of the autonomous driving vehicle.