Driving behavior prediction method and device, equipment and storage medium

Abstract

The invention provides a driving behavior prediction method and device, equipment and a storage medium. Relates to the technical field of automatic driving. The method comprises the following steps: constructing a virtual human model and a driving simulation environment, configuring the virtual human model in the driving simulation environment, enabling the virtual human model to interact with a vehicle model, controlling the vehicle model to act, establishing a deep learning model, and according to input first person view angle data of a driver, carrying out deep learning on the vehicle model. And outputting a control action and outputting the control action to the virtual human model, training the deep learning model by using the training data set, maximizing accumulated rewards through two stages of imitation learning and reinforcement learning to obtain a strategy function, and continuously updating the strategy function through interaction with environment information by using near-end strategy optimization to obtain the virtual human model. Obtaining a trained deep learning model; and realizing driving behavior prediction by using the trained deep learning model. The decision-making ability of the automatic driving system is improved by simulating the decision-making process of the human driver.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, and in particular, to a driving behavior prediction method, device, equipment, and storage medium. Background Art

[0002] With the development of autonomous driving technology and intelligent transportation systems, how to accurately predict and simulate driver behavior has become an important research direction. Traditional driving behavior prediction methods mostly rely on rules and sensor data, but these methods show limitations in complex road and traffic environments.

[0003] To solve this problem, artificial intelligence, especially deep learning and neural network technologies, have been widely applied to the field of driving behavior prediction. Neural networks can be trained with a large amount of driving data to automatically learn and extract the characteristics and rules of driver behavior, thereby enhancing the accuracy of driving behavior prediction. However, existing autonomous driving systems often have difficulty reacting quickly and effectively in the face of dynamic traffic conditions and diverse driving behaviors. Traditional methods that rely on manual rules and sensor data have limitations in dealing with complex scenarios. Summary of the Invention

[0004] This application provides a driving behavior prediction method, device, equipment, and storage medium, which can improve the decision-making ability of the autonomous driving system by simulating the decision-making process of human drivers.

[0005] In a first aspect, this application provides a driving behavior prediction method, including: Constructing a virtual human model, where the virtual human model includes a joint model for simulating driving actions and postures; Constructing a driving simulation environment, where the driving simulation environment includes a vehicle model and environmental information, configuring the virtual human model in the driving simulation environment so that the virtual human model interacts with the vehicle model and controls the actions of the vehicle model; Build a deep learning model, which is used to output control actions according to the input first-person perspective data of the driver and output them to the virtual human model. The virtual human model controls the actions of the vehicle model in response to the input control actions. The deep learning model includes a brain-inspired perception and decision-making network and a brain-inspired control network. Among them, the processing flow of the brain-inspired perception and decision-making network is as follows: Using the first-person perspective data of the driver as input data, a brain-inspired feature extraction network anatomically aligned with the brain perception pathway is constructed through a convolutional neural network and a recurrent neural network structure to extract the activation response of the input data and encode the activation response into a first feature map. The brain-inspired control network includes several recurrent networks, and each recurrent network is used to simulate the prefrontal cortex, motor cortex, sensory cortex, spinal cord, red nucleus, cerebellum, VLc, superior colliculus, and reticular nucleus. The processing flow of the brain-inspired control network is as follows: Obtain the first feature map, gradually extract spatial and temporal features from the first feature map through multiple layers of convolution to obtain a second feature map, and obtain a target Q value based on the second feature map. The target Q value is sequentially normalized, activated, feedback-regulated, and normalized to output control actions. The activation process uses a linear activation parameter including a slope parameter, and the slope parameter is used to process the negative value region. The feedback regulation includes: when the target Q value is not within the set threshold range, re-perform multi-layer convolution processing on the first feature map; Obtain a training data set according to real driving data; among them, the training data set includes multiple state-action pairs; Use the training data set to train the deep learning model, so that the deep learning model maximizes the cumulative reward through two stages of imitation learning and reinforcement learning to obtain a policy function, and through interaction with environmental information, continuously update the policy function using proximal policy optimization, and finally obtain the trained deep learning model; Use the trained deep learning model to realize driving behavior prediction.

[0006] In a possible design, the calculation process of obtaining the target Q value based on the second feature map is as follows: ; In the formula, represents the target Q value, represents the immediate reward, represents the discount factor, which is used to weigh current and future rewards, represents the next state the maximum Q value under, calculated by the target network parameter calculate, represents the next action.

[0007] In a possible design, obtaining a training data set according to real driving data includes: Obtain the kinematic data of the driver during driving, where the kinematic data includes the angles, speeds, and accelerations of joints; Based on the kinematic data, extract key points and generate pose data in 3D space; wherein, the key points include shoulders, elbows, knees, and / or ankles, and the pose data in 3D space includes multiple data points, and each data point corresponds to a driving task, and the driving tasks include steering, braking, and accelerating; Generate state-action pairs according to the pose data in 3D space.

[0008] In a possible design, using the training data set to train the deep learning model, so that the deep learning model maximizes the cumulative reward through two stages of imitation learning and reinforcement learning, obtains a policy function, and updates the policy function continuously using proximal policy optimization by interacting with environmental information, including: In the imitation learning stage, with the goal of minimizing the error between the predicted action and the actual action of the expert, construct a loss function, and implement the training in the imitation learning stage based on the loss function. The loss function is expressed as: ; In the formula, represents the loss function, represents the action predicted by the model, represents the actual action of the expert, represents the model parameters, represents the total number of demonstration data, represents the serial number of the demonstration data, represents the th state; In the reinforcement learning stage, learn the policy function through the following formula: ; In the formula, represents the policy function, E represents the expectation of randomness in the environment, represents the immediate reward obtained at time t, represents the discount factor, represents choosing the next action that can maximize at the next state , a represents the current action, s represents the current state; By interacting with environmental information, update the policy function through the following objective function: ; In the formula, ​Denote the objective function for policy optimization, Denote the model parameters, Denote the advantage function, Denote the hyperparameter that controls the magnitude of policy update, and min denotes the minimum value function, , denote the probability ratio, π Denote pi, a t Denote t the action at time s t Denote t the state at time Denote clipping the ratio to the interval , Denote the probability that the new policy selects action at state , Denote the probability that the old policy selects the same action at the same state.

[0009] In a possible design, after completing imitation learning and reinforcement learning, the trained deep learning model generates a control instruction, which is expressed as: ; In the formula, Denote t the throttle amplitude at time Denote t the braking force at time Denote t the steering wheel swing angle at time s t Denote t the state at time is the weight of the network, is the bias term.

[0010] In a possible design, when the deep learning model configures a multi-task reinforcement learning model and involves multiple driving tasks, it learns multiple tasks by sharing network weights, and the learning process is expressed as: ; In the formula, Denote the sum of fitting errors for all tasks, Denote the number of tasks, Denote the Q-value function for the i th task, Denote the estimated Q-value obtained through the policy, Denote the expectation of the state and the action .

[0011] In a possible design, after finally obtaining the trained deep learning model, the method further includes: Measuring the similarity between the actions of real people and the actions output by the trained deep learning model, and adjusting the model architecture and hyperparameters through coding analysis and representation similarity analysis.

[0012] In a second aspect, the present application provides a driving behavior prediction device, which includes a controller configured to: Construct a virtual human model, where the virtual human model includes a joint model for simulating driving actions and postures; Construct a driving simulation environment, where the driving simulation environment includes a vehicle model and environmental information, configure the virtual human model in the driving simulation environment, so that the virtual human model interacts with the vehicle model and controls the actions of the vehicle model; Construct a deep learning model, which is used to output a control action based on the input first-person perspective data of the driver and output it to the virtual human model. The virtual human model controls the actions of the vehicle model in response to the input control action. The deep learning model includes a brain-inspired perception decision-making network and a brain-inspired control network; among them, the processing process of the brain-inspired perception decision-making network is: using the first-person perspective data of the driver as input data, constructing a brain-inspired feature extraction network anatomically aligned with the brain perception pathway through a convolutional neural network and a recurrent neural network structure, extracting the activation response of the input data, and encoding the activation response as a first feature map; the brain-inspired control network includes several recurrent networks, each recurrent network is used to simulate the prefrontal cortex, motor cortex, sensory cortex, spinal cord, red nucleus, cerebellum, VLc, superior colliculus and reticular nucleus. The processing process of the brain-inspired control network is: obtaining the first feature map, gradually extracting spatial and temporal features from the first feature map through multiple layers of convolution to obtain a second feature map, and obtaining a target Q value based on the second feature map. The target Q value is sequentially normalized, activated, feedback-regulated and normalized, and a control action is output. The activation process uses a linear activation parameter including a slope parameter, and the slope parameter is used to process the negative value region. The feedback regulation includes: when the target Q value is not within the set threshold range, re-performing multiple layers of convolution processing on the first feature map; Obtain a training data set according to real driving data; where the training data set includes multiple state-action pairs; Use the training data set to train the deep learning model, so that the deep learning model maximizes the cumulative reward through two stages of imitation learning and reinforcement learning to obtain a policy function, and continuously updates the policy function using proximal policy optimization through interaction with environmental information, and finally obtains a trained deep learning model; use the trained deep learning model to implement driving behavior prediction.

[0013] In a third aspect, an embodiment of the present application provides an electronic device, including: at least one processor and a memory; the memory stores computer-executable instructions; the at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the driving behavior prediction method described in the first aspect above and various possible designs of the first aspect.

[0014] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the driving behavior prediction method described in the first aspect above and various possible designs of the first aspect are implemented.

[0015] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program. When the computer program is executed by a processor, the driving behavior prediction method described in the first aspect above and various possible designs of the first aspect are implemented.

[0016] The driving behavior prediction method, device, equipment, and storage medium provided by the present application have at least the following beneficial effects: By constructing a virtual human model, the present application uses the virtual human model as an intelligent agent, and through the internal artificial neural network, it responds to behaviors in different driving scenarios and can continuously learn and optimize, gradually approaching the behaviors of actual human drivers. This not only provides more reliable behavior prediction for the autonomous driving system but also significantly improves the safety and adaptability of the system in complex road scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings here are incorporated into the specification and form a part of this specification, showing embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application.

[0018] Figure 1 It is a flowchart of a driving behavior prediction method provided by an embodiment of the present application; Figure 2 It is a schematic diagram of a brain-like perception and decision-making network implementing brain-like perception and decision-making provided by an embodiment of the present application; Figure 3 It is a schematic diagram of brain-like control provided by an embodiment of the present application; Figure 4 It is a structural diagram of a brain-like control network provided by an embodiment of the present application.

[0019] Through the above-mentioned accompanying drawings, specific embodiments of the present application have been shown, and there will be more detailed descriptions hereinafter. These drawings and written descriptions are not intended to limit the scope of the concept of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. Detailed Description of the Embodiments

[0020] Here, the exemplary embodiments will be described in detail, and the examples are shown in the accompanying drawings. When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0021] Next, the technical solution of the present application and how the technical solution of the present application solves the above technical problems will be described in detail with specific embodiments. These several specific embodiments below can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. Next, the embodiments of the present application will be described with reference to the accompanying drawings.

[0022] The embodiment of the present application provides a driving behavior prediction method. The purpose of this method is to construct a model that can realize driving behavior prediction. The constructed model can be configured in the automatic driving system of a vehicle to execute the driving behavior prediction function, such as outputting a control instruction for the vehicle based on the first perspective data (image data collected by an in-vehicle camera) observed by the driver, and can realize the automatic control of the vehicle. As Figure 1 shown, it is a flowchart of a driving behavior prediction method provided by the embodiment of the present application. This driving behavior prediction method includes the following steps S100 - S500.

[0023] S100: Construct a virtual human model, and the virtual human model includes a joint model for simulating driving actions and postures.

[0024] In this embodiment, the constructed virtual human model is used to simulate a driver. In this embodiment, the virtual human model is designed in a sitting position on the driver's seat, with natural movements and in line with the physiological characteristics of human driving. The modeling process starts from components such as the head, torso, and limbs, constructing them one by one, and creating a detailed skeletal system for the virtual human, including key joints such as the spine, shoulders, elbows, and knee joints, enabling it to perform natural movements such as turning the head, raising the arm, stepping on the accelerator and brake. Using the automatic weight assignment technology, the bones are bound to the mesh of the model to ensure that the movement of each bone can drive the corresponding part of the mesh, guaranteeing the smoothness and naturalness of the movements. In terms of physical property settings, each body part (such as the arm, leg, torso, etc.) in the virtual human model is given a rigid body and associated with a collision body to enable physical interaction with objects in the environment. To improve the simulation accuracy, appropriate collision bodies (such as box-shaped collision bodies and spherical collision bodies) are added to each rigid body part, and the friction coefficient and elastic parameters are adjusted to ensure that the virtual human behaves realistically when interacting with objects such as the seat, steering wheel, and pedals. Joint constraints are set for parts such as the hands, feet, and knee joints to ensure that the movements are within the natural range of human activities and avoid unreasonable movements. After the virtual human model is constructed, it is imported into the NVIDIA PhysX SDK for further physical simulation. The rigid bodies and collision bodies of each body part are associated in PhysX (the physics engine), and physical constraints such as spring constraints and rotational constraints are used to ensure the biological rationality of joint movements (such as the bending angle of the knee joint not exceeding the physiological limit).

[0025] It should be noted that the NVIDIA PhysX SDK is an open platform designed for virtual collaboration and accurate real-time simulation of physical properties.

[0026] In an exemplary embodiment, the specific method for constructing the virtual human model is as follows: The construction of the virtual human model first uses Blender for modeling to create the basic mesh of the virtual human, including components such as the head, torso, and limbs. Each component is modeled separately for subsequent bone binding and motion control. Design a biomechanical skeletal system for the virtual human, including key joints such as the spine, shoulders, elbows, knee joints, and ankle joints, and set reasonable rotation ranges to ensure natural and smooth movements. Use Blender's automatic weight assignment technology to bind the skeleton to the mesh to ensure that the movement of the bones can drive the mesh of the body parts. To ensure the biomechanical rationality of the movements, the model uses the physics engine PhysX to set rigid bodies for each body part (such as the arm, leg, etc.) and add collision bodies (such as box-shaped or spherical) to simulate the interaction with objects in the environment (such as the seat, steering wheel, accelerator, and brake pedals).

[0027] It should be noted that Blender is a comprehensive three-dimensional graphics software.

[0028] S200: Construct a driving simulation environment, where the driving simulation environment includes a vehicle model and environmental information, configure the virtual human model in the driving simulation environment so that the virtual human model interacts with the vehicle model, and control the actions of the vehicle model.

[0029] In this embodiment, the purpose of step S200 is to create a virtual driving simulation environment and configure the virtual human model constructed in step S100 in this driving simulation environment.

[0030] In an exemplary embodiment, after the virtual human model is established, it is imported into CARLA, and an urban scene is created in CARLA as the simulation environment, and then the virtual human model is integrated with the vehicle. The specific steps include: starting the CARLA server and connecting to the client, selecting and loading the simulation scene, creating and configuring the vehicle, setting the position and action control of the virtual human model in the vehicle (hands controlling the steering wheel, feet controlling the accelerator and brake), and configuring sensors to provide environmental information.

[0031] It should be noted that CARLA is an open-source simulator for autonomous driving research.

[0032] Specifically, in the physical simulation of the driving environment, the CARLA simulation platform is used to create a driving environment, including elements such as urban roads, traffic signs, pedestrians, and vehicles. Town01 is selected as the simulation scene. The vehicle model vehicle.audi.a2 is selected, and its physical properties, such as vehicle speed, brake response, steering system, etc., are set to ensure that it can interact with the virtual human model. The interaction between the virtual human model and the objects in the vehicle is set through the PhysX physics engine to ensure that the hands of the virtual human can collide with the steering wheel and the feet can interact with the accelerator and brake pedals. At the same time, the friction coefficient and elastic parameters are set to simulate real physical reactions.

[0033] S300: Construct a deep learning model, where the deep learning model is used to output control actions according to the input first-person perspective data of the driver and output them to the virtual human model, and the virtual human model controls the actions of the vehicle model in response to the input control actions.

[0034] In this embodiment, the deep learning model is similar to the brain of the virtual human model. Its specific architecture is designed according to the human brain structure. The deep learning model is used to control the virtual human model to execute the control instructions it outputs, so that the virtual human model can control the vehicle model to act in the driving simulation environment. The interaction information obtained in this way can be fed back for the training of the deep learning model, so that the finally obtained deep learning model can more accurately realize driving behavior prediction.

[0035] Specifically, the deep learning model includes a brain-inspired perception and decision-making network and a brain-inspired control network. Among them, the processing flow of the brain-inspired perception and decision-making network is as follows: using the driver's first-person perspective data as input data, a brain-inspired feature extraction network anatomically aligned with the brain's perception pathway is constructed through a convolutional neural network and a recurrent neural network structure to extract the activation response of the input data and encode the activation response into a first feature map. The brain-inspired control network includes several recurrent networks, each recurrent network is used to simulate the prefrontal cortex, motor cortex, sensory cortex, spinal cord, red nucleus, cerebellum, VLc, superior colliculus, and reticular nucleus. The processing flow of the brain-inspired control network is as follows: obtaining the first feature map, gradually extracting spatial and temporal features from the first feature map through multiple layers of convolution to obtain a second feature map, and obtaining a target Q value based on the second feature map. The target Q value is sequentially normalized, activated, feedback-regulated, and normalized, and a control action is output. The activation process uses a linear activation parameter including a slope parameter, and the slope parameter is used to process the negative value region. The feedback regulation includes: when the target Q value is not within the set threshold range, the first feature map is re-processed through multiple layers of convolution.

[0036] It should be noted that the role of the brain-inspired perception and decision-making network is to generate a preliminary decision based on the first-person perspective data (such as the image data collected by an in-vehicle camera) and is represented by the first feature map. The brain-inspired perception and decision-making network can be trained through corresponding image datasets. The brain-inspired control network is used to further optimize the preliminary decision generated by the brain-inspired perception and decision-making network. For example, deep feature extraction is performed on the first feature map to obtain a second feature map, and the target Q value is calculated through a Q-value function. Finally, after processing such as normalizing, activating, feedback-regulating, and normalizing the target Q value, a control action is output. In the subsequent step S500, the training of the deep learning model is mainly the training of the brain-inspired control network to enable it to output more accurate control actions, so as to obtain a deep learning model applicable to autonomous driving. For example, the control of the vehicle can be achieved through the control actions output by the trained deep learning model. Especially in complex environments, it can ensure that its control actions are more like the operations made by an experienced driver, thereby reducing the incidence of traffic accidents.

[0037] The design and processing flow of the brain-inspired perception and decision-making network and the brain-inspired control network in the deep learning model are introduced in detail below.

[0038] Figure 2This is the schematic diagram of the brain-inspired perception and decision-making network provided by the embodiments of this application to achieve brain-inspired perception and decision-making. The brain-inspired perception and decision-making network includes a brain-inspired perception network and a brain-inspired decision-making network. Taking the first-person perspective of the driver as the input, the brain will output the predicted identity label, and then taking this as the input, the brain will output the driving strategy. Referring to this model, this embodiment designs a brain-inspired perception network and a brain-inspired decision-making network with aligned brain pathways. The brain-inspired perception network consists of one convolutional neural network and three recurrent neural networks, and the brain-inspired decision-making network consists of one convolutional neural network and four recurrent neural networks. First, the brain-inspired perception network receives the first-person perspective information of the driver. After activation, this deep neural network will output brain-inspired perception information, and then taking the perception information as the input of the brain-inspired decision-making network, the driving strategy is finally obtained.

[0039] The processing flow of the brain-inspired perception network is as follows: First, obtain the first-person perspective data of the driver, and then construct a brain-inspired feature extraction network anatomically aligned with the brain perception pathway through the convolutional neural network and recurrent neural network structure to extract the activation response of the input data in the deep brain network, and finally encode the input end as feature data and IL, and use it as the input of the decision-making network.

[0040] The processing flow of the brain-inspired decision-making network is: Obtain the feature data and IL from the brain-inspired perception network and the measurement coding connection, construct a brain-inspired decision-making network aligned with the anatomical structure of the brain decision-making pathway through the convolutional neural network and recurrent neural network structure, extract the activation response of the input data in the deep brain network, and map it to continuous low-level actions.

[0041] In this embodiment, the design and processing flow of the brain-inspired control network are as follows: Brain-inspired control is mainly reflected in the control of hands and feet. For example, how a person controls the hand to turn the steering wheel, and how to control the foot to step on the brake and accelerator. The schematic diagram is as Figure 3As shown. High-level decisions first enter the prefrontal cortex, then are transmitted to the medial globus pallidus of the basal ganglia, then to VLo, and then back to area 6 in the motor cortex. The process of sensory cortex - pons - cerebellum - VLc - area 4 is for feedback regulation. The motor cortex transmits motor information to the spinal cord through the lateral pathway and the ventromedial pathway. The lateral pathway consists of the red nucleus and the corticospinal tract, and the ventromedial pathway consists of the superior colliculus and the reticular nucleus. By summarizing the motor loop of this human brain, this embodiment designs a brain-inspired control network based on brain pathway alignment. This network is mainly composed of an eight-layer recurrent network and a two-layer deep Q network. After the human brain motor loop is activated, it will output specific action control instructions, and the brain-inspired control network of this application will also output specific action control instructions after activation. Here, the brain-inspired network is compared with the human brain for neural measurement, and the corresponding action control is for behavioral measurement. The deep neural network is compared with the brain to obtain a brain-inspired control score to measure the quality of the brain-inspired control network.

[0042] The pathways for control include the corticospinal pathway from the cortex to the spinal cord and the pattern generator from the spinal cord to the muscles. The corticospinal pathway is the main pathway for transmitting motor commands between the brain and the body, and is divided into the lateral corticospinal tract and the anterior corticospinal tract. The former controls the fine movements of the limbs and trunk, and the latter mainly controls the gross movements of the trunk. The pattern generator (CPG) from the spinal cord to the muscles is a neural network mechanism responsible for generating and regulating periodic movement patterns such as gait and breathing, and can spontaneously generate regular movements without direct intervention from the brain.

[0043] In the generation of the action of turning the steering wheel with the arm, neurons in the primary motor cortex send commands, which are transmitted to the spinal cord through the corticospinal pathway, and then to the muscles of the hand and forearm (such as the biceps, flexors, extensors) through the lower motor neurons. The muscles contract alternately to generate rotational force and precisely control the turning of the steering wheel. Sensors in the hand and forearm sense the resistance and rotation angle, and feedback the information to the brain to help adjust the turning force or direction.

[0044] In the generation of the action of stepping on the brake and accelerator, when the driver steps on the accelerator or brake, the primary motor cortex issues commands, which are transmitted to the spinal cord through the corticospinal pathway, and then the signal is transmitted to the leg muscles, especially the ankle and toes, by the motor neurons. The spinal cord coordinates the rhythmic contraction of the lower limb muscles through the pattern generator to ensure smooth movement. When the sole of the foot pushes the pedal, the toes flex, and the ankle muscles are also involved in the movement. The proprioceptive organs in the foot sense the pedal pressure and contact situation and feedback to the brain to help adjust the force application and rhythm.

[0045] Among them, a deep Q-network is used to simulate the basal ganglia and VLo, and a recurrent network is used to simulate the prefrontal cortex, motor cortex, sensory cortex, spinal cord, red nucleus, cerebellum, VLc, superior colliculus, and reticular nucleus. The reinforcement learning mechanism of the deep Q-network (DQN) highly matches the functions of the basal ganglia. The basal ganglia learn optimal behaviors through a reward-driven mechanism (dopamine signal) to maximize long-term rewards, which is similar to DQN optimizing the decision-making process by learning the state-action value function (Q-value). In addition, DQN can handle high-dimensional state spaces, extract features in complex environments through deep neural networks, select optimal actions, simulate the decision-making, memory, and reinforcement learning functions of the basal ganglia in motor control, and thus achieve brain-like intelligent control. The recurrent neural network (RNN) can process time-series data. Through its recursive structure, the output depends on the current input and previous states, capturing temporal dynamics and long-term dependencies. This highly matches the dynamic feedback mechanism in motor control: the prefrontal cortex is responsible for motor planning and decision-making, the motor cortex executes specific instructions, and the sensory cortex receives and feeds back sensory information of the body. The spinal cord quickly executes movements and feeds back the state, the red nucleus coordinates the fine adjustment of movements, and the cerebellum ensures smooth and accurate movements. VLc regulates the initiation and inhibition of movements, the superior colliculus processes visual information, and the reticular nucleus regulates the movement rhythm and reactivity. RNN simulates the dynamic interaction and continuous feedback between neural systems through its recursive structure, accurately generates and adjusts movement instructions, and controls muscle actions, realizing functions similar to the biological motor control system. The bidirectional transmission of signals between each network also well reflects the feedback regulation mechanism of brain-like control, enabling it to better simulate the control network of the brain.

[0046] By referring to human control of these actions, the present invention is able to design a brain-like control network based on the alignment of human brain control pathways with high biological interpretability. The structure of the brain-like control network is as Figure 4 shown.

[0047] In this brain-inspired control network, first, the high-level decision output by the decision-making module is input, and then through three layers of convolution, temporal features are gradually extracted and the image dimension is reduced. Convolutional layer 1 uses 32 filters with a filter size of 8×8, a stride of 4, and a ReLU activation function. The main purpose of this layer is to extract the global features of the input state while reducing the dimension and computational complexity. Convolutional layer 2 uses 64 filters with a filter size of 4×4, a stride of 2, and a ReLU activation function. This layer further extracts finer-grained features while reducing the spatial dimension of the data, providing a more abstract representation for subsequent processing. Convolutional layer 3 uses 64 filters with a filter size of 3×3, a stride of 1, and a ReLU activation function. This layer mainly performs refined extraction of the feature map, focusing on more local feature patterns, and providing more detailed semantic information for the decision-making process. The output of the convolutional layer is flattened into a one-dimensional vector and enters the fully connected layer. The fully connected layer contains 512 neurons with a ReLU activation function. The role of this layer is to perform feature combination and further mapping on the feature map output by the convolutional layer and output the target Q value. The Q value is updated according to the Bellman equation, and the target Q value is: ; wherein, represents the target Q value, represents the immediate reward, is the discount factor (balancing current and future rewards), is the next state the maximum Q value under, calculated by the target network parameters . After the fully connected layer, a batch normalization layer is added to accelerate network convergence, reduce internal covariate shift, and improve training stability. Then a ReLU activation function is connected. In the present invention, a certain improvement is made to the traditional ReLU activation function, that is, a small slope parameter is introduced to handle the negative value region: ; wherein, is a small positive number, defined here as , so as to ensure that the gradient will not completely disappear. Then feedback adjustment is performed on it, and the less excellent data is re-input into convolutional layer 1. Finally, through another batch normalization layer, the data is standardized again to further enhance the training effect. Finally, the control action is output.

[0048] S400: Obtain a training data set according to the real driving data; wherein, the training data set includes multiple state-action pairs.

[0049] Exemplarily, in this embodiment, a high-precision motion capture system is used to collect the joint movements of the driver during driving. The motion capture system precisely records the limb movements by installing inertial sensors at the key joint parts of the driver, including the steering of the arm, gear shifting operations, the actions of stepping on the accelerator and brake with the legs, and the rotation of the head, etc. These data can describe in detail the posture and joint movements of the driver, thereby helping to analyze the limb behaviors in different driving tasks.

[0050] In terms of data processing, the collected kinematic data needs to be further processed to form a high-quality training dataset. The present invention adopts the DANNCE algorithm to process the collected kinematic data, especially the joint data of the human body. The DANNCE algorithm accurately extracts the key points of the human body (such as shoulders, elbows, knees, ankles, etc.) by analyzing the images captured by multi-view cameras or sensors, and generates the pose data of the human body in the 3D space. This process enables the accurate extraction of the specific position information of the joints, and then generates a complete joint movement trajectory. At the same time, by combining the first-person view data of the driver collected by a high-precision camera, a high-quality dataset for training, that is, multiple state-action pairs, is finally formed.

[0051] S500: Use the training dataset to train the deep learning model, so that the deep learning model maximizes the cumulative reward through two stages of imitation learning and reinforcement learning to obtain a policy function, and through interaction with the environmental information, continuously update the policy function using proximal policy optimization, and finally obtain the trained deep learning model; use the trained deep learning model to realize driving behavior prediction.

[0052] In this embodiment, the purpose of step 500 is to train the deep learning model, and its training purpose is mainly to obtain the optimal policy function. This policy function can obtain the target Q value through the second feature map, that is, this training process is mainly for the training of the brain-like control network.

[0053] Specifically, during the training process, in the imitation learning stage, the virtual human learns the behavior of the expert through behavior cloning. Behavior cloning maps the control actions of the expert to the state of the environment through supervised learning. By collecting real driving data, a set of state-action pairs is designed , where is the i-th state, is the action taken by the expert in state . The goal is to minimize the error between the predicted action and the actual action of the expert. This process can be achieved by minimizing the following loss function: ; In the formula, is the action predicted by the model, is the actual action demonstrated by the expert, are the model parameters, is the total number of demonstration data. By minimizing this loss function, the imitation agent can learn how to generate actions similar to those of the expert in the same environment.

[0054] In the reinforcement learning phase, the model learns the optimal policy through interaction with the environment. The goal of reinforcement learning is to maximize the cumulative reward. In the driving control task, the goal of the model is to select a series of actions to maximize the long-term reward. represents the expected cumulative reward for taking action in state . The goal in reinforcement learning is to learn an optimal Q function, i.e., ; where, is the immediate reward obtained at time t, is the discount factor, is the next action, is the next state. The model continuously updates through interaction with the environment to optimize the decision-making. For the update of the policy, the present invention adopts Proximal Policy Optimization (PPO) to update the policy by optimizing the following objective function: ; where, is the probability ratio, is the advantage function, is a hyperparameter that controls the magnitude of the policy update.

[0055] After completing imitation learning and reinforcement learning, the model can generate control commands such as throttle, brake, steering, etc. The control policy generation is based on the output of the decision-making module, and the decision is converted into continuous low-level control actions through a fully connected network. The output of the model is the amplitude of the throttle pedal , the braking force of the brake pedal and the swing angle of the steering wheel by hand , i.e., the control action can be expressed as: ; where, are the weights of the network, is the bias term, is the activation function (Sigmoid function), s t is the current state input (the speed of the vehicle, the change of direction, etc.). After being mapped by the neural network of the model, these control signals generate specific control actions.

[0056] In some embodiments, considering that the virtual human driving task involves multiple tasks (such as accelerating, braking, steering, obstacle avoidance, etc.), a multi-task reinforcement learning model is designed in this embodiment to optimize the strategies of multiple tasks through shared representations. Each task has a reward function of , and multiple tasks can be learned by sharing network weights: ; wherein, is the number of tasks, is the Q-value function of the i-th task, is the estimated Q-value obtained through the policy. Through multi-task learning, the virtual human can handle multiple driving tasks simultaneously and optimize the overall driving behavior.

[0057] Finally, the trained deep learning model controls the vehicle by controlling the output instructions (throttle, brake, steering wheel angle, etc.). Each control output needs to be fed back to the perception system during execution to update the environmental state and adjust the strategy according to real-time data.

[0058] In some embodiments, to improve the prediction performance (i.e., prediction accuracy) of the trained deep learning model, steps for training and validating the model are further included, and the specific steps are as follows: Input new data into the trained artificial neural network (ANN) to obtain the predicted control actions. Then, measure the similarity of driving actions. First, perform encoding analysis (EA): calculate the correlation between the input data and the activation values of each layer of the ANN to measure how the network encodes driving behavior: ; wherein, is the input data, is the activation value.

[0059] Next, perform representation similarity analysis (RSA) to compare the similarity of the driving behavior actions of the virtual human and the real human, using the cosine similarity method: ; wherein, is the behavior action of the virtual human (ANN), is the behavior action of the real human (biological), represents the similarity of the two actions.

[0060] The present invention improves the performance of the model by adjusting the architecture and hyperparameters of the neural network. First, adjust the number of layers of the encoder and decoder to optimize the expressive ability of the model and avoid overfitting. Next, use the ReLU function to process non-linear features and accelerate the training process. By adjusting the L2 regularization parameter, overfitting is prevented and the generalization ability is improved.

[0061] After model training, the similarity between the virtual human and the real human is evaluated based on the data obtained from the previous driving behavior action similarity analysis. If the similarity is lower than the preset threshold of 0.85, the model architecture and hyperparameters are adjusted through a feedback mechanism to optimize the accuracy and control ability of the model.

[0062] An embodiment of the present application further provides a driving behavior prediction device, which includes a controller configured to: Construct a virtual human model, where the virtual human model includes a joint model for simulating driving actions and postures; Construct a driving simulation environment, where the driving simulation environment includes a vehicle model and environmental information, and configure the virtual human model in the driving simulation environment so that the virtual human model interacts with the vehicle model and controls the actions of the vehicle model; Construct a deep learning model, which is used to output control actions based on the input first-person perspective data of the driver and output them to the virtual human model. The virtual human model controls the actions of the vehicle model in response to the input control actions. The deep learning model includes a brain-inspired perception decision-making network and a brain-inspired control network; wherein, the processing flow of the brain-inspired perception decision-making network is: using the first-person perspective data of the driver as input data, constructing a brain-inspired feature extraction network anatomically aligned with the brain perception pathway through a convolutional neural network and a recurrent neural network structure, extracting the activation response of the input data, and encoding the activation response as a first feature map; the brain-inspired control network includes several recurrent networks, and each recurrent network is used to simulate the prefrontal cortex, motor cortex, sensory cortex, spinal cord, red nucleus, cerebellum, VLc, superior colliculus, and reticular nucleus. The processing flow of the brain-inspired control network is: obtaining the first feature map, gradually extracting spatial and temporal features from the first feature map through multiple layers of convolution to obtain a second feature map, and obtaining a target Q value based on the second feature map. The target Q value is sequentially normalized, activated, feedback-regulated, and normalized to output control actions. The activation processing uses a linear activation parameter including a slope parameter, and the slope parameter is used to process the negative value region. The feedback regulation includes: when the target Q value is not within the set threshold range, re-performing multiple layers of convolution processing on the first feature map; Obtain a training data set according to real driving data; wherein, the training data set includes multiple state-action pairs; The deep learning model is trained using the training data set so that the deep learning model maximizes the cumulative reward through two stages of imitation learning and reinforcement learning to obtain a policy function, and the policy function is continuously updated using proximal policy optimization through interaction with environmental information to finally obtain a trained deep learning model; and the trained deep learning model is used to realize driving behavior prediction.

[0063] An embodiment of the present application provides an electronic device, which may include: a processor and a memory, wherein the processor and the memory may communicate with each other; illustratively, the processor and the memory communicate with each other via a communication bus.

[0064] The processor executes the computer execution instructions stored in the memory, so that the processor executes the scheme in the above embodiment. The processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit ASIC, a field programmable gate array FPGA or other programmable logic devices, discrete gates or transistor logic devices, and discrete hardware components.

[0065] The communication bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The system bus can be divided into an address bus, a data bus, a control bus, etc. The transceiver is used to implement communication between the database access device and other computers (such as clients, read-write libraries, and read-only libraries). The memory may include random access memory (RAM) and may also include non-volatile memory.

[0066] The electronic device provided in the embodiment of the present application may be the terminal device of the above embodiment.

[0067] An embodiment of the present application also provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed on a computer, the computer executes the technical solution of the driving behavior prediction method of the above embodiment.

[0068] The embodiments of the present application also provide a computer program product. The computer program product includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium. When the at least one processor executes the computer program, the technical solutions of the driving behavior prediction method in the above embodiments can be implemented.

[0069] In several embodiments provided by the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of modules is only a logical function division. In actual implementation, there may be other division methods. For example, multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces, indirect couplings or communication connections of devices or modules, which can be electrical, mechanical or other forms.

[0070] The modules described as separate components may or may not be physically separated. The components shown as modules may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to implement the solutions of this embodiment.

[0071] In addition, in each embodiment of the present application, the functional modules can be integrated in a processing unit, or each module can exist physically alone, or two or more modules can be integrated in one unit. The units formed by the above modules can be implemented in the form of hardware, or in the form of a combination of hardware and software functional units.

[0072] The above integrated modules implemented in the form of software functional modules can be stored in a computer-readable storage medium. The above software functional modules are stored in a storage medium and include several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute some steps of the methods in the various embodiments of the present application.

[0073] It should be understood that the above-mentioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the method disclosed in combination with the invention can be directly embodied as being executed and completed by a hardware processor, or can be executed and completed by a combination of hardware and software modules in the processor.

[0074] The memory may include high-speed RAM memory, and may also include non-volatile storage NVM, such as at least one disk memory, and can also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk, or an optical disc, etc.

[0075] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc.

[0076] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic memory, flash memory, a magnetic disk, or an optical disc. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0077] An exemplary storage medium is coupled to the processor, enabling the processor to read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an Application Specific Integrated Circuit (ASIC). Of course, the processor and the storage medium can also exist as discrete components in an electronic control unit or a master control device.

[0078] Those of ordinary skill in the art will understand that all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions. The foregoing program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments; and the foregoing storage medium includes: various media such as ROM, RAM, magnetic disks, or optical discs that can store program codes.

[0079] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application and are not intended to limit them. Although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A driving behavior prediction method, characterized in that The method includes: Constructing a virtual human model, where the virtual human model includes a joint model for simulating driving actions and postures; Constructing a driving simulation environment, where the driving simulation environment includes a vehicle model and environmental information, configuring the virtual human model in the driving simulation environment so that the virtual human model interacts with the vehicle model and controls the actions of the vehicle model; Constructing a deep learning model, where the deep learning model is used to output control actions based on the input first-person perspective data of the driver and output them to the virtual human model, and the virtual human model controls the actions of the vehicle model in response to the input control actions. The deep learning model includes a brain-inspired perception and decision-making network and a brain-inspired control network; where, the processing process of the brain-inspired perception and decision-making network is: using the first-person perspective data of the driver as input data, constructing a brain-inspired feature extraction network anatomically aligned with the brain perception pathway through a convolutional neural network and a recurrent neural network structure, extracting the activation response of the input data, and encoding the activation response as a first feature map; the brain-inspired control network includes several recurrent networks, and each recurrent network is used to simulate the prefrontal cortex, motor cortex, sensory cortex, spinal cord, red nucleus, cerebellum, VLc, superior colliculus, and reticular nucleus. The processing process of the brain-inspired control network is: obtaining the first feature map, gradually extracting spatial and temporal features from the first feature map through multiple layers of convolution to obtain a second feature map, and obtaining a target Q value based on the second feature map. The target Q value is successively normalized, activated, and feedback-regulated and normalized to output control actions. The activation process uses a linear activation parameter including a slope parameter, and the slope parameter is used to process the negative value region. The feedback regulation includes: when the target Q value is not within the set threshold range, re-performing multiple layers of convolution processing on the first feature map; Obtaining a training data set according to real driving data; where, the training data set includes multiple state-action pairs; Training the deep learning model using the training data set so that the deep learning model maximizes the cumulative reward through two stages of imitation learning and reinforcement learning to obtain a policy function, and continuously updates the policy function using proximal policy optimization through interaction with environmental information, and finally obtains a trained deep learning model; using the trained deep learning model to implement driving behavior prediction.

2. The driving behavior prediction method according to claim 1, wherein The calculation process of obtaining the target Q value based on the second feature map is: ； Wherein, represents the target Q value, represents the immediate reward, represents the discount factor, which is used to balance the current and future rewards, represents the next state the maximum Q value under, calculated by the target network parameters calculate, represents the next action.

3. The driving behavior prediction method according to claim 1, wherein Obtaining a training data set according to real driving data, including: Obtaining the kinematic data of the driver during driving, where the kinematic data includes the angles, speeds, and accelerations of the joints; Based on the kinematic data, extracting key points and generating pose data in 3D space; where, the key points include the shoulders, elbows, knees, and / or ankles, and the pose data in 3D space includes multiple data points, and each data point corresponds to a driving task, and the driving tasks include steering, braking, and accelerating; Generating state-action pairs according to the pose data in 3D space.

4. The driving behavior prediction method according to claim 1, characterized in that, Training the deep learning model using the training data set, so that the deep learning model maximizes the cumulative reward through two stages of imitation learning and reinforcement learning to obtain a policy function, and the method of continuously updating the policy function by interacting with environmental information using proximal policy optimization includes: In the imitation learning stage, with the goal of minimizing the error between the predicted action and the actual action of the expert, a loss function is constructed, and the training in the imitation learning stage is implemented based on the loss function. The loss function is expressed as: ； In the formula, represents the loss function, represents the action predicted by the model, represents the actual action of the expert, represents the model parameters, represents the total number of demonstration data, represents the serial number of the demonstration data, represents the th state; In the reinforcement learning stage, the policy function is learned through the following formula: ； wherein, represents the policy function, E represents the expectation of randomness in the environment, represents the immediate reward obtained at time t, represents the discount factor, represents at the next state select the one that maximizes for the next action , a represents the current action, s represents the current state; Through interaction with environmental information, the policy function is updated through the following objective function: ； In the formula, represents the objective function for policy optimization, represents the model parameters, represents the advantage function, represents the hyperparameter that controls the magnitude of policy update, and min represents the minimum value function, , represents the probability ratio, π represents pi, a t represents t the action at time s t represents t the state at time represents clipping the ratio to the interval , represents the probability that the new policy selects action at state , represents the probability that the old policy selects the same action at the same state.

5. The driving behavior prediction method according to claim 1, wherein After completing imitation learning and reinforcement learning, the trained deep learning model generates a control instruction, and the control instruction is expressed as: ； wherein, represents t the throttle amplitude at time represents t the braking force at time represents t the swing angle of the steering wheel at time s t represents t the state at time is the weight of the network, is the bias term.

6. The driving behavior prediction method according to claim 4, wherein When the deep learning model configures a multi-task reinforcement learning model and is involved in multiple driving tasks, multiple tasks are learned by sharing network weights. The learning process is expressed as: ； wherein, represents the total fitting error of all tasks, represents the number of tasks, represents the i Q-value function of the represents the estimated Q-value obtained through the policy, represents the expectation of the state and the action .

7. The driving behavior prediction method according to any one of claims 1 to 6, characterized in that, After finally obtaining the trained deep learning model, the method further includes: Measuring the similarity between the actions of real people and the actions output by the trained deep learning model, and adjusting the model architecture and hyperparameters through coding analysis and representation similarity analysis.

8. A driving behavior prediction device, characterized in that, The device includes a controller, and the controller is configured to: Construct a virtual human model, and the virtual human model includes a joint model for simulating driving actions and postures; Construct a driving simulation environment, and the driving simulation environment includes a vehicle model and environmental information. The virtual human model is configured in the driving simulation environment so that the virtual human model interacts with the vehicle model to control the actions of the vehicle model; Build a deep learning model, which is used to output control actions according to the input first-person perspective data of the driver and output them to the virtual human model. The virtual human model controls the actions of the vehicle model in response to the input control actions. The deep learning model includes a brain-inspired perception and decision-making network and a brain-inspired control network. Among them, the processing flow of the brain-inspired perception and decision-making network is as follows: using the first-person perspective data of the driver as input data, a brain-inspired feature extraction network anatomically aligned with the brain's perception pathway is constructed through a convolutional neural network and a recurrent neural network structure to extract the activation response of the input data and encode the activation response into a first feature map. The brain-inspired control network includes several recurrent networks, and each recurrent network is used to simulate the prefrontal cortex, motor cortex, sensory cortex, spinal cord, red nucleus, cerebellum, VLc, superior colliculus, and reticular nucleus. The processing flow of the brain-inspired control network is as follows: obtain the first feature map, gradually extract spatial and temporal features from the first feature map through multiple layers of convolution to obtain a second feature map, and obtain a target Q value based on the second feature map. The target Q value is sequentially normalized, activated, feedback-regulated, and normalized to output a control action. The activation process uses a linear activation parameter including a slope parameter, and the slope parameter is used to process the negative value region. The feedback regulation includes: when the target Q value is not within the set threshold range, the first feature map is reprocessed through multiple layers of convolution; Obtain a training data set according to real driving data; among them, the training data set includes multiple state-action pairs; Use the training data set to train the deep learning model, so that the deep learning model maximizes the cumulative reward through two stages of imitation learning and reinforcement learning to obtain a policy function, and through interaction with environmental information, continuously update the policy function using proximal policy optimization, and finally obtain a trained deep learning model; use the trained deep learning model to realize driving behavior prediction.

9. An electronic device, characterized in that, Including: A processor, and a memory communicatively connected to the processor; The memory stores computer execution instructions; The processor executes the computer execution instructions stored in the memory to implement the driving behavior prediction method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, Computer execution instructions are stored in the computer-readable storage medium, and when the computer execution instructions are executed by the processor, they are used to implement the driving behavior prediction method according to any one of claims 1-7.

Images