Driving behavior prediction method, device, equipment and storage medium

Abstract

The present application provides a driving behavior prediction method, device, equipment and storage medium. It relates to the field of autonomous driving technology. The method includes: constructing a virtual human model and a driving simulation environment, configuring the virtual human model in the driving simulation environment, allowing the virtual human model to interact with the vehicle model, controlling the vehicle model action, establishing a deep learning model to output control actions based on the input driver's first-person perspective data and output them to the virtual human model, using a training data set to train the deep learning model, and through the two stages of imitation learning and reinforcement learning, maximizing the cumulative reward to obtain a policy function, and through interaction with environmental information, using proximal policy optimization to continuously update the policy function to obtain a trained deep learning model; using the trained deep learning model to realize driving behavior prediction. The present application improves the decision-making ability of the autonomous driving system by simulating the decision-making process of human drivers.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to a driving behavior prediction method, device, equipment and storage medium. Background Art

[0002] With the development of autonomous driving technology and intelligent transportation systems, how to accurately predict and simulate driver behavior has become an important research direction. Traditional driving behavior prediction methods mostly rely on rules and sensor data, but these methods have limitations in complex road and traffic environments.

[0003] To address this issue, artificial intelligence, particularly deep learning and neural network technologies, has been widely applied to driving behavior prediction. Neural networks can be trained using massive amounts of driving data to automatically learn and extract characteristics and patterns in driver behavior, thereby enhancing the accuracy of driving behavior prediction. However, existing autonomous driving systems often struggle to respond quickly and effectively to dynamic traffic conditions and diverse driving behaviors. Traditional methods that rely on handcrafted rules and sensor data have limitations when handling complex scenarios. Summary of the Invention

[0004] The present application provides a driving behavior prediction method, device, equipment and storage medium, which improve the decision-making ability of the autonomous driving system by simulating the decision-making process of human drivers.

[0005] In a first aspect, the present application provides a driving behavior prediction method, comprising:

[0006] Constructing a virtual human model, the virtual human model including a joint model for simulating driving movements and postures;

[0007] Constructing a driving simulation environment, the driving simulation environment including a vehicle model and environmental information, and placing the virtual human model in the driving simulation environment so that the virtual human model interacts with the vehicle model to control the movement of the vehicle model;

[0008] A deep learning model is constructed, wherein the deep learning model is used to output control actions according to the input driver's first-person perspective data and output them to the virtual human model, and the virtual human model controls the vehicle model action in response to the input control actions. The deep learning model includes a brain-like perception decision network and a brain-like control network; wherein the processing flow of the brain-like perception decision network is as follows: using the driver's first-person perspective data as input data, a brain-like feature extraction network aligned with the brain's perception pathway anatomy is constructed through a convolutional neural network and a recurrent neural network structure, extracting the activation response of the input data, and encoding the activation response into a first feature map; the brain-like control network includes several recurrent networks, each of which is a brain-like feature extraction network. The recurrent network is used to simulate the prefrontal cortex, motor cortex, sensory cortex, spinal cord, red nucleus, cerebellum, VLc, superior colliculus, and reticular nucleus. The processing flow of the brain-like control network is as follows: obtaining the first feature map, gradually extracting spatial and temporal features from the first feature map through multi-layer convolution to obtain a second feature map, and obtaining a target Q value based on the second feature map. The target Q value is sequentially normalized, activated, and feedback-adjusted and normalized, and a control action is output. The activation process uses a linear activation parameter including a slope parameter, and the slope parameter is used to process negative value areas. The feedback adjustment includes: when the target Q value is not within a set threshold range, re-performing multi-layer convolution on the first feature map;

[0009] Acquire a training data set based on real driving data; wherein the training data set includes a plurality of state-action pairs;

[0010] The deep learning model is trained using the training data set so that the deep learning model maximizes the cumulative reward through two stages of imitation learning and reinforcement learning to obtain a policy function, and the policy function is continuously updated using proximal policy optimization through interaction with environmental information to ultimately obtain a trained deep learning model; and the trained deep learning model is used to realize driving behavior prediction.

[0011] In one possible design, the calculation process of obtaining the target Q value based on the second feature map is:

[0012] ;

[0013] Where, represents the target Q value, Indicates immediate reward, represents the discount factor used to weigh current and future rewards, Indicates the next state The maximum Q value is determined by the target network parameters calculate, Indicates the next action.

[0014] In one possible design, a training dataset is obtained based on real driving data, including:

[0015] Acquiring kinematic data of the driver during driving, wherein the kinematic data includes angles, velocities, and accelerations of joints;

[0016] Extracting key points based on the kinematic data to generate pose data in 3D space; wherein the key points include shoulders, elbows, knees, and / or ankles, and the pose data in 3D space includes multiple data points, each data point corresponding to a driving task, wherein the driving tasks include steering, braking, and acceleration;

[0017] According to the pose data in the 3D space, a state-action pair is generated.

[0018] In one possible design, the deep learning model is trained using the training dataset so that the deep learning model maximizes cumulative rewards through two stages of imitation learning and reinforcement learning to obtain a policy function, and the policy function is continuously updated using proximal policy optimization through interaction with environmental information. The method includes:

[0019] In the imitation learning stage, the goal is to minimize the error between the predicted action and the expert's actual action, and a loss function is constructed. The training of the imitation learning stage is carried out based on the loss function. The loss function is expressed as:

[0020] ;

[0021] Where, represents the loss function, represents the action predicted by the model, Indicates the actual actions of the experts. represents the model parameters, Indicates the total number of demonstration data, Indicates the serial number of the demonstration data, Indicates the status;

[0022] In the reinforcement learning stage, the policy function is learned through the following formula:

[0023] ;

[0024] Where, represents the policy function, E represents the expectation of randomness in the environment, represents the immediate reward obtained at time t, represents the discount factor, Indicates the next state The next choice can maximize Next move , a Indicates the current action. s Indicates the current state;

[0025] By interacting with environmental information, the policy function is updated through the following objective function:

[0026] ;

[0027] Where, represents the objective function of policy optimization, represents the model parameters, represents the advantage function, Represents a hyperparameter that controls the magnitude of the policy update, min represents the minimum function, , represents the probability ratio, π represents pi, a t express t The action of the moment, s t express t The state of the moment, Indicates clipping the ratio to the interval , Indicates that the new policy is in state Next select action The probability of represents the probability that the old policy chooses the same action in the same state.

[0028] In one possible design, after completing imitation learning and reinforcement learning, the trained deep learning model generates control instructions, which are expressed as:

[0029] ;

[0030] Where, express t The throttle amplitude at the moment, express t The braking force at each moment, express t The steering wheel's swing angle at that moment, s t express t The state of the moment, are the weights of the network, is the bias term.

[0031] In one possible design, a multi-task reinforcement learning model is configured in the deep learning model. When multiple driving tasks are involved, multiple tasks are learned by sharing network weights. The learning process is expressed as:

[0032] ;

[0033] Where, represents the sum of fitting errors of all tasks, Indicates the number of tasks, Indicates the i The Q-value function of a task, represents the estimated Q value obtained by the strategy, Indicates status and actions expectations.

[0034] In one possible design, after finally obtaining the trained deep learning model, the method further includes:

[0035] The similarity between real human movements and the movements output by the trained deep learning model is measured, and the model architecture and hyperparameters are adjusted through encoding analysis and representation similarity analysis.

[0036] In a second aspect, the present application provides a driving behavior prediction device, the device comprising a controller, the controller being configured to:

[0037] Constructing a virtual human model, the virtual human model including a joint model for simulating driving movements and postures;

[0038] Constructing a driving simulation environment, the driving simulation environment including a vehicle model and environmental information, and placing the virtual human model in the driving simulation environment so that the virtual human model interacts with the vehicle model to control the movement of the vehicle model;

[0039] A deep learning model is constructed, wherein the deep learning model is used to output control actions according to the input driver's first-person perspective data and output them to the virtual human model, and the virtual human model controls the vehicle model action in response to the input control actions. The deep learning model includes a brain-like perception decision network and a brain-like control network; wherein the processing flow of the brain-like perception decision network is as follows: using the driver's first-person perspective data as input data, a brain-like feature extraction network aligned with the brain's perception pathway anatomy is constructed through a convolutional neural network and a recurrent neural network structure, extracting the activation response of the input data, and encoding the activation response into a first feature map; the brain-like control network includes several recurrent networks, each of which is a brain-like feature extraction network. The recurrent network is used to simulate the prefrontal cortex, motor cortex, sensory cortex, spinal cord, red nucleus, cerebellum, VLc, superior colliculus, and reticular nucleus. The processing flow of the brain-like control network is as follows: obtaining the first feature map, gradually extracting spatial and temporal features from the first feature map through multi-layer convolution to obtain a second feature map, and obtaining a target Q value based on the second feature map. The target Q value is sequentially normalized, activated, and feedback-adjusted and normalized, and a control action is output. The activation process uses a linear activation parameter including a slope parameter, and the slope parameter is used to process negative value areas. The feedback adjustment includes: when the target Q value is not within a set threshold range, re-performing multi-layer convolution on the first feature map;

[0040] Acquire a training data set based on real driving data; wherein the training data set includes a plurality of state-action pairs;

[0041] The deep learning model is trained using the training data set so that the deep learning model maximizes the cumulative reward through two stages of imitation learning and reinforcement learning to obtain a policy function, and the policy function is continuously updated using proximal policy optimization through interaction with environmental information to ultimately obtain a trained deep learning model; and the trained deep learning model is used to realize driving behavior prediction.

[0042] In a third aspect, an embodiment of the present application provides an electronic device comprising: at least one processor and a memory; the memory stores computer-executable instructions; the at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the driving behavior prediction method described in the first aspect and various possible designs of the first aspect.

[0043] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the driving behavior prediction method described in the first aspect and various possible designs of the first aspect is implemented.

[0044] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the driving behavior prediction method described in the first aspect and various possible designs of the first aspect.

[0045] The driving behavior prediction method, device, equipment, and storage medium provided in this application have at least the following beneficial effects:

[0046] This application builds a virtual human model as an intelligent agent that responds to different driving scenarios through an internal artificial neural network. This model continuously learns and optimizes, gradually approaching the behavior of actual human drivers. This not only provides more reliable behavior predictions for the autonomous driving system, but also significantly improves the system's safety and adaptability in complex road scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0048] Figure 1 A flowchart of a driving behavior prediction method provided in an embodiment of the present application;

[0049] Figure 2 A schematic diagram of the brain-like perception and decision-making network provided in an embodiment of the present application to implement noise-like perception and decision-making;

[0050] Figure 3 A diagram of the brain-like control principle provided in an embodiment of the present application;

[0051] Figure 4 This is a structural diagram of the brain-inspired control network provided in an embodiment of the present application.

[0052] The above drawings illustrate specific embodiments of the present application, which will be described in more detail below. These drawings and the textual description are not intended to limit the scope of the present application in any way, but rather to illustrate the concepts of the present application to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION

[0053] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0054] The following specific embodiments describe in detail the technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0055] The embodiment of the present application provides a driving behavior prediction method, the purpose of which is to construct a model that can realize driving behavior prediction. The constructed model can be configured in the automatic driving system of the vehicle to perform driving behavior prediction functions, such as outputting vehicle control instructions based on the first-person perspective data observed by the driver (image data collected by the on-board camera), so as to realize automatic control of the vehicle. Figure 1 , which is a flow chart of a driving behavior prediction method provided in an embodiment of the present application, the driving behavior prediction method includes the following steps S100 - S500 .

[0056] S100: Constructing a virtual human model, wherein the virtual human model includes a joint model for simulating driving actions and postures.

[0057] In this embodiment, the constructed virtual human model is used to simulate a driver. This embodiment designs the virtual human model to appear seated in the driver's seat, with natural movements that align with the physiological characteristics of human driving. The modeling process begins with components such as the head, torso, and limbs, each constructed individually. A detailed skeletal system is created for the virtual human, including key joints such as the spine, shoulders, elbows, and knees, enabling natural movements such as turning the head, raising arms, and applying the accelerator and brakes. Automatic weight distribution technology is used to bind the skeleton to the model's mesh, ensuring that the movement of each bone drives the corresponding mesh, ensuring smooth and natural movements. Regarding physical properties, each body part in the virtual human model (such as arms, legs, and torso) is assigned a rigid body and associated with a collider to facilitate physical interaction with objects in the environment. To improve simulation accuracy, appropriate colliders (such as box and sphere colliders) are added to each rigid body part, and friction coefficients and elasticity parameters are adjusted to ensure realistic interaction between the virtual human and objects such as the seat, steering wheel, and pedals. Joint constraints were set for the hands, feet, and knees to ensure movement within the natural range of motion and avoid unnatural motion. Once the virtual human model was constructed, it was imported into the NVIDIA PhysX SDK for further physics simulation. The rigid bodies and collision bodies of each body part were linked in PhysX (the physics engine). Physical constraints such as spring and rotation constraints were used to ensure biological plausibility of joint movement (for example, knee flexion angles did not exceed physiological limits).

[0058] It should be noted that NVIDIA PhysX SDK is an open platform designed for virtual collaboration and real-time simulation with accurate physical properties.

[0059] In an exemplary embodiment, the specific method of constructing the virtual human model is:

[0060] The construction of the virtual human model begins with modeling using Blender to create the virtual human's base mesh, including components such as the head, torso, and limbs. Each component is modeled separately to facilitate subsequent skeletal rigging and motion control. A biomechanically correct skeletal system is designed for the virtual human, including key joints such as the spine, shoulders, elbows, knees, and ankles, with appropriate rotation ranges set to ensure natural and smooth movements. Using Blender's automatic weight distribution technology, the skeleton is bound to the mesh, ensuring that the movement of the bones drives the mesh of the body parts. To ensure the biomechanical plausibility of the movements, the model uses the physics engine PhysX to assign rigid bodies to each body part (such as arms and legs), and adds collision bodies (such as boxes or spheres) to simulate interaction with objects in the environment (such as seats, steering wheels, and accelerator and brake pedals).

[0061] It should be noted that Blender is a comprehensive 3D graphics software.

[0062] S200: Constructing a driving simulation environment, wherein the driving simulation environment includes a vehicle model and environmental information, and configuring the virtual human model in the driving simulation environment so that the virtual human model interacts with the vehicle model to control the movement of the vehicle model.

[0063] In this embodiment, the purpose of step S200 is to create a virtual driving simulation environment and configure the virtual human model constructed in step S100 in the driving simulation environment.

[0064] In an exemplary embodiment, after the avatar model is created, it is imported into CARLA, where a city scene is created as the simulation environment. The avatar model is then integrated with the vehicle. The specific steps include starting the CARLA server and connecting to the client, selecting and loading the simulation scene, creating and configuring the vehicle, setting the avatar model's position within the vehicle and its motion control (hands controlling the steering wheel, feet controlling the accelerator and brakes), and configuring sensors to provide environmental information.

[0065] It should be noted that CARLA is an open source simulator for autonomous driving research.

[0066] Specifically, for the physical simulation of the driving environment, the CARLA simulation platform was used to create the driving environment, including elements such as city roads, traffic signs, pedestrians, and vehicles. Town01 was selected as the simulation scene. Vehicle.audi.a2 was selected as the vehicle model, and its physical properties, such as speed, brake response, and steering system, were configured to ensure interaction with the virtual human model. The PhysX physics engine was used to configure the interaction between the virtual human model and objects within the vehicle, ensuring that the virtual human's hands could collide with the steering wheel and that its feet could interact with the accelerator and brake pedals. Friction coefficients and elasticity parameters were also set to simulate real-world physical reactions.

[0067] S300: Constructing a deep learning model, wherein the deep learning model is used to output a control action based on the input driver's first-person perspective data and output it to the virtual human model, and the virtual human model controls the vehicle model action in response to the input control action.

[0068] In this embodiment, the deep learning model is similar to the brain of the virtual human model, and its specific architecture is designed according to the structure of the human brain. The deep learning model is used to control the virtual human model to execute the control instructions it outputs, so that the virtual human model can control the vehicle model to move in the driving simulation environment. The interaction information obtained in this way can be fed back into the training of the deep learning model, so that the final deep learning model can more accurately predict driving behavior.

[0069] Specifically, the deep learning model includes a brain-like perception decision network and a brain-like control network; wherein, the processing flow of the brain-like perception decision network is: taking the driver's first-person perspective data as input data, constructing a brain-like feature extraction network that is anatomically aligned with the brain's perception pathway through a convolutional neural network and a recurrent neural network structure, extracting the activation response of the input data, and encoding the activation response into a first feature map; the brain-like control network includes several recurrent networks, each of which is used to simulate the prefrontal cortex, motor cortex, sensory cortex, spinal cord, red nucleus, cerebellum, VLc, superior colliculus and reticular nuclei, the processing flow of the brain-like control network is: obtaining the first feature map, gradually extracting spatial and temporal features from the first feature map through multi-layer convolution to obtain a second feature map, and obtaining a target Q value based on the second feature map, normalizing, activating, feedback-adjusting and normalizing the target Q value in turn, and outputting a control action. The activation processing adopts a linear activation parameter including a slope parameter, and the slope parameter is used to process the negative value area. The feedback adjustment includes: when the target Q value is not within the set threshold range, re-performing multi-layer convolution processing on the first feature map.

[0070] It should be noted that the brain-inspired perception and decision-making network generates preliminary decisions based on first-person perspective data (such as image data captured by an onboard camera), represented by a first feature map. The noise-inspired perception and decision-making network can be trained using a corresponding image dataset. The brain-inspired control network is used to further optimize the preliminary decisions generated by the brain-inspired perception and decision-making network. For example, deep feature extraction is performed on the first feature map to obtain a second feature map, and a target Q-value function is used to calculate the target Q-value. Finally, the target Q-value is normalized, activated, and feedback-adjusted, and normalized before outputting a control action. In the subsequent step S500, the training of the deep learning model primarily trains the brain-inspired control network to enable it to output more accurate control actions, thereby obtaining a deep learning model applicable to autonomous driving. For example, the control actions output by the trained deep learning model can be used to control the vehicle. Especially in complex environments, this ensures that the control actions are more similar to those of an experienced driver, thereby reducing the incidence of traffic accidents.

[0071] The following details the design and processing flow of the brain-inspired perception decision network and brain-inspired control network in the deep learning model.

[0072] Figure 2 The schematic diagram of the brain-like perception decision network provided in the embodiment of the present application to implement the brain-like perception decision. The brain-like perception decision network includes a brain-like perception network and a brain-like decision network. Taking the driver's first-person perspective as input, the brain will output its predicted identity label, and then use it as input, and the brain will output the driving strategy. Referring to this model, this embodiment designs a brain-like perception network and a brain-like decision network with aligned brain pathways. The brain-like perception network consists of a layer of convolutional neural network and a three-layer recurrent network, and the brain-like decision network consists of a layer of convolutional neural network and a four-layer recurrent network. First, the brain-like perception network receives the driver's first-person perspective information. After activation, the deep neural network will output brain-like perception information, and then use the perception information as input to the brain-like decision network to finally obtain the driving strategy.

[0073] The processing flow of the brain-like perception network is as follows: first, the driver's first-person perspective data is obtained, and then a brain-like feature extraction network is constructed that is aligned with the brain's perception pathway anatomy through convolutional neural networks and recurrent neural network structures. The activation response of the input data in the deep brain-like network is extracted, and finally the input end is encoded into feature data. and IL, and serve as the input of the decision network.

[0074] The processing flow of the brain-like decision network is: obtaining feature data from the brain-like perception network and measuring coding connections and IL, build a brain-like decision network that aligns with the anatomical structure of the brain's decision pathway through convolutional neural network and recurrent neural network structures, extract the activation response of the input data in the deep brain-like network, and map it to continuous low-level actions.

[0075] In this embodiment, the design and processing flow of the brain-inspired control network is as follows:

[0076] Brain-like control is mainly reflected in the control of hands and feet. For example, how do people control their hands to turn the steering wheel and how do they control their feet to step on the brakes and accelerators? The principle diagram is as follows Figure 3 As shown. High-level decisions first enter the prefrontal cortex, then pass to the medial pallidum of the basal ganglia, then to VLo, and then back to area 6 in the motor cortex. The process of sensory cortex-pons-cerebellum-VLc-area 4 is feedback regulation. The motor cortex transmits movement information to the spinal cord through the lateral pathway and the ventromedial pathway. The lateral pathway is composed of the red nucleus and the corticospinal tract, and the ventromedial pathway is composed of the superior colliculus and the reticular nucleus. By summarizing the motor circuit of the human brain, this embodiment designs a brain-like control network based on brain pathway alignment, which is mainly composed of an eight-layer recurrent network and a two-layer deep Q network. After activation, the human brain motor circuit will output specific action control instructions, and the brain-like control network of this application will also output specific action control instructions after activation. Here, the brain-like network is compared with the human brain for neural measurement, and the corresponding action control is measured behaviorally. The deep neural network is compared with the brain to obtain a brain-like control score to measure the pros and cons of the brain-like control network.

[0077] The control pathways include the corticospinal pathway from the cortex to the spinal cord and the pattern generator from the spinal cord to the muscles. The corticospinal pathway is the main pathway for transmitting movement commands between the brain and the body. It is divided into the lateral corticospinal tract and the anterior corticospinal tract. The former controls fine movements of the limbs and trunk, while the latter mainly controls gross movements of the trunk. The spinal cord-to-muscle pattern generator (CPG) is a neural network mechanism responsible for generating and regulating periodic movement patterns, such as gait and breathing, and can spontaneously produce regular movements without direct brain intervention.

[0078] To generate the arm's steering wheel movement, neurons in the primary motor cortex send commands, which are transmitted to the spinal cord via the corticospinal pathway. These signals are then transmitted via lower motor neurons to the muscles of the hand and forearm (such as the biceps, flexors, and extensors). These muscles contract alternately, generating rotational force and precisely controlling steering wheel rotation. Sensors in the hand and forearm sense resistance and rotation angle, feeding this information back to the brain to help adjust the force or direction of rotation.

[0079] When the driver presses the accelerator or brake pedal, the primary motor cortex issues a command, which is transmitted to the spinal cord via the cortical-spinal pathway. Motor neurons then transmit the signal to the leg muscles, particularly the ankles and toes. The spinal cord coordinates the rhythmic contraction of the lower limb muscles through pattern generators to ensure smooth movement. When the sole of the foot pushes on the pedal, the toes flex, and the ankle muscles also participate in the movement. The proprioceptive organs in the foot sense the pedal pressure and contact, providing feedback to the brain to help adjust force and rhythm.

[0080] A deep Q-network was used to simulate the basal ganglia and VLo, while a recurrent network was used to simulate the prefrontal cortex, motor cortex, sensory cortex, spinal cord, red nucleus, cerebellum, VLc, superior colliculus, and reticular nucleus. The reinforcement learning mechanism of the deep Q-network (DQN) closely aligns with the functions of the basal ganglia. The basal ganglia learn optimal behaviors to maximize long-term rewards through a reward-driven mechanism (dopamine signaling), similar to how the DQN optimizes decision-making by learning state-action-value functions (Q-values). Furthermore, the DQN can process high-dimensional state spaces. Using deep neural networks, it extracts features from complex environments and selects optimal actions, simulating the decision-making, memory, and reinforcement learning functions of the basal ganglia in motor control, thereby achieving brain-like intelligent control. Recurrent neural networks (RNNs) can process time series data. Through a recursive structure, their output depends on the current input and previous states, capturing temporal dynamics and long-term dependencies. This closely aligns with the dynamic feedback mechanism in motor control: the prefrontal cortex is responsible for movement planning and decision-making, the motor cortex executes specific commands, and the sensory cortex receives and provides feedback on sensory information. The spinal cord rapidly executes movements and provides feedback, the red nucleus coordinates fine-tuning of movements, and the cerebellum ensures smooth and accurate movement. The vertebral cortex regulates movement initiation and inhibition, the superior colliculus processes visual information, and the reticular nucleus regulates movement rhythm and responsiveness. Through its recursive structure, RNNs simulate the dynamic interaction and continuous feedback between neural systems, accurately generating and adjusting motor commands and controlling muscle movements, achieving functions similar to biological motor control systems. The bidirectional transmission of signals between these networks also embodies the feedback regulation mechanism of brain-inspired control, enabling it to better simulate the brain's control network.

[0081] By learning from human control of these actions, the present invention is able to design a brain-like control network that is highly biologically interpretable and aligned with the human brain control pathway. The structure of the brain-like control network is as follows: Figure 4 shown.

[0082] In this brain-inspired control network, the high-level decision output by the decision module is first input. It then undergoes three layers of convolution to gradually extract temporal features and reduce image dimensionality. Convolutional layer 1 uses 32 filters with a filter size of 8×8, a stride of 4, and a ReLU activation function. This layer primarily extracts global features from the input state while reducing dimensionality and computational complexity. Convolutional layer 2 uses 64 filters with a filter size of 4×4, a stride of 2, and a ReLU activation function. This layer further extracts finer-grained features while reducing the spatial dimensionality of the data, providing a more abstract representation for subsequent processing. Convolutional layer 3 uses 64 filters with a filter size of 3×3, a stride of 1, and a ReLU activation function. This layer primarily refines the feature map, focusing on more localized feature patterns and providing more detailed semantic information for the decision-making process. The output of the convolutional layer is flattened into a one-dimensional vector and fed into the fully connected layer, which contains 512 neurons and uses the ReLU activation function. The function of this layer is to combine and further map the feature maps output by the convolutional layer and output the target Q value. The Q value is updated according to the Bellman equation, and the target Q value is:

[0083] ;

[0084] Where, represents the target Q value, Indicates immediate reward, is the discount factor (weighing current and future rewards), For the next state The maximum Q value is determined by the target network parameters After the fully connected layer, a batch normalization layer is added to accelerate network convergence, reduce internal covariate shift, and improve training stability. Then the ReLU activation function is connected. In this invention, the traditional ReLU activation function is improved to a certain extent, that is, a small slope parameter is introduced. To handle negative areas:

[0085] ;

[0086] in, is a small positive number, defined here as This ensures that the gradient doesn't vanish completely. Feedback is then applied to the model, re-inputting suboptimal data into Convolutional Layer 1. Finally, a batch normalization layer normalizes the data again, further enhancing the training effect. Finally, the control action is output.

[0087] S400: Acquire a training data set based on real driving data; wherein the training data set includes a plurality of state-action pairs.

[0088] For example, this embodiment uses a high-precision motion capture system to collect the driver's joint movements while driving. By installing inertial sensors at the driver's key joints, the motion capture system accurately records their body movements, including arm movements such as steering and gear shifting, leg movements such as accelerator and brake pedaling, and head rotation. This data provides a detailed description of the driver's posture and joint movements, thereby facilitating analysis of their body movements during different driving tasks.

[0089] In terms of data processing, the collected kinematic data requires further processing to form a high-quality training dataset. This invention utilizes the DANNCE algorithm to process the collected kinematic data, particularly human joint data. The DANNCE algorithm analyzes images captured by multi-view cameras or sensors to accurately extract key points of the human body (such as shoulders, elbows, knees, ankles, etc.) and generate human pose data in 3D space. This process enables the precise extraction of specific joint position information, generating complete joint motion trajectories. Combined with the driver's first-person perspective data captured using high-precision cameras, this ultimately forms a high-quality training dataset, consisting of multiple state-action pairs.

[0090] S500: Using the training data set to train the deep learning model, so that the deep learning model maximizes the cumulative reward through two stages of imitation learning and reinforcement learning to obtain a policy function, and through interaction with environmental information, the policy function is continuously updated using proximal policy optimization to finally obtain a trained deep learning model; using the trained deep learning model to realize driving behavior prediction.

[0091] In this embodiment, the purpose of step 500 is to train the deep learning model, and the training purpose is mainly to obtain the optimal policy function, which can obtain the target Q value through the second feature map, that is, this training process is mainly aimed at training the brain-like control network.

[0092] Specifically, during the training process, in the imitation learning phase, the virtual human learns the expert's behavior through behavioral cloning. Behavioral cloning maps the expert's control actions to the state of the environment through supervised learning. By collecting real driving data, a set of state-action pairs is designed. ,in is the i-th state, Is in state The goal is to minimize the predicted action Actual action with experts This process can be achieved by minimizing the following loss function:

[0093] ;

[0094] Where, is the action predicted by the model, It is the actual action demonstrated by the experts. are model parameters, is the total number of demonstration data. By minimizing this loss function, the imitating agent can learn how to produce actions similar to those of the expert in the same environment.

[0095] During the reinforcement learning phase, the model learns the optimal policy through interaction with the environment. The goal of reinforcement learning is to maximize the cumulative reward. In the driving control task, the model's goal is to select a series of actions to maximize the long-term reward. Indicates that the status Take action Expected cumulative reward. The goal of reinforcement learning is to learn an optimal Q function, that is:

[0096] ;

[0097] Where, is the immediate reward obtained at time t, is the discount factor, It's the next move. is the next state. The model is constantly updated through interaction with the environment To optimize the decision. For the policy update, the present invention adopts proximal policy optimization (PPO) to update the policy by optimizing the following objective function:

[0098] ;

[0099] in, is the probability ratio, is the advantage function, is a hyperparameter that controls the magnitude of the policy update.

[0100] After completing imitation learning and reinforcement learning, the model can generate control instructions such as accelerator, brake, steering, etc. The control strategy generates outputs based on the decision module and converts the decision into continuous low-level control actions through the fully connected network. The output of the model is to control the amplitude of the accelerator. , brake force and the swing angle of the steering wheel , that is, the control action can be expressed as:

[0101] ;

[0102] in, are the weights of the network, is the bias term, is the activation function (Sigmoid function), s t The current state input (vehicle speed, direction change, etc.) is generated by the model’s neural network mapping, generating specific control actions.

[0103] In some embodiments, considering that the virtual human driving task involves multiple tasks (such as acceleration, braking, steering, obstacle avoidance, etc.), this embodiment designs a multi-task reinforcement learning model to optimize the strategies of multiple tasks by sharing representations. The reward function is , multiple tasks can be learned by sharing network weights:

[0104] ;

[0105] in, is the number of tasks, is the Q-value function of the i-th task, is the estimated Q value obtained through the strategy. Through multi-task learning, the virtual human can handle multiple driving tasks simultaneously and optimize the overall driving behavior.

[0106] Ultimately, the trained deep learning model controls the vehicle by outputting control commands (accelerator, brake, steering wheel angle, etc.). Each control output needs to be fed back to the perception system upon execution to update the environment state and adjust the strategy based on real-time data.

[0107] In some embodiments, to improve the prediction performance (i.e., prediction accuracy) of the trained deep learning model, the model training and verification steps are further included. The specific steps are as follows:

[0108] New data is fed into a trained artificial neural network (ANN) to obtain predicted control actions. The driving action similarity is then measured, starting with encoding analysis (EA): This calculates the correlation between the input data and the activation values ​​of each ANN layer to measure how the network encodes the driving behavior:

[0109] ;

[0110] in, For input data, is the activation value.

[0111] Next, we perform representation similarity analysis (RSA) to compare the similarity between the virtual human and the real human driving behaviors, using the cosine similarity method:

[0112] ;

[0113] in, is the behavior of the virtual person (ANN), It is the behavior of real people (organisms). Indicates the similarity of the two actions.

[0114] This paper improves model performance by adjusting the neural network architecture and hyperparameters. First, the number of encoder and decoder layers is adjusted to optimize the model's expressiveness and avoid overfitting. Next, the ReLU function is used to handle nonlinear features and accelerate the training process. Adjusting the L2 regularization parameter prevents overfitting and improves generalization.

[0115] After model training, the similarity between the virtual and real people is assessed using data from previous driving behavior similarity analysis. If the similarity falls below a preset threshold of 0.85, the model architecture and hyperparameters are adjusted through a feedback mechanism to optimize the model's accuracy and control capabilities.

[0116] An embodiment of the present application further provides a driving behavior prediction device, the driving behavior prediction device including a controller, wherein the controller is configured to:

[0117] Constructing a virtual human model, the virtual human model including a joint model for simulating driving movements and postures;

[0118] Constructing a driving simulation environment, the driving simulation environment including a vehicle model and environmental information, and placing the virtual human model in the driving simulation environment so that the virtual human model interacts with the vehicle model to control the movement of the vehicle model;

[0119] A deep learning model is constructed, wherein the deep learning model is used to output control actions according to the input driver's first-person perspective data and output them to the virtual human model, and the virtual human model controls the vehicle model action in response to the input control actions. The deep learning model includes a brain-like perception decision network and a brain-like control network; wherein the processing flow of the brain-like perception decision network is as follows: using the driver's first-person perspective data as input data, a brain-like feature extraction network aligned with the brain's perception pathway anatomy is constructed through a convolutional neural network and a recurrent neural network structure, extracting the activation response of the input data, and encoding the activation response into a first feature map; the brain-like control network includes several recurrent networks, each of which is a brain-like feature extraction network. The recurrent network is used to simulate the prefrontal cortex, motor cortex, sensory cortex, spinal cord, red nucleus, cerebellum, VLc, superior colliculus, and reticular nucleus. The processing flow of the brain-like control network is as follows: obtaining the first feature map, gradually extracting spatial and temporal features from the first feature map through multi-layer convolution to obtain a second feature map, and obtaining a target Q value based on the second feature map. The target Q value is sequentially normalized, activated, and feedback-adjusted and normalized, and a control action is output. The activation process uses a linear activation parameter including a slope parameter, and the slope parameter is used to process negative value areas. The feedback adjustment includes: when the target Q value is not within a set threshold range, re-performing multi-layer convolution on the first feature map;

[0120] Acquire a training data set based on real driving data; wherein the training data set includes a plurality of state-action pairs;

[0121] The deep learning model is trained using the training data set so that the deep learning model maximizes the cumulative reward through two stages of imitation learning and reinforcement learning to obtain a policy function, and the policy function is continuously updated using proximal policy optimization through interaction with environmental information to ultimately obtain a trained deep learning model; and the trained deep learning model is used to realize driving behavior prediction.

[0122] An embodiment of the present application provides an electronic device, which may include a processor and a memory, wherein the processor and the memory can communicate with each other; illustratively, the processor and the memory communicate with each other via a communication bus.

[0123] The processor executes the computer-executable instructions stored in the memory, so that the processor implements the solutions in the above embodiments. The processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0124] The communication bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. System buses can be categorized as address buses, data buses, and control buses. Transceivers facilitate communication between the database access device and other computers (e.g., clients, read-write libraries, and read-only libraries). Memory may include random access memory (RAM) or non-volatile memory.

[0125] The electronic device provided in the embodiment of the present application may be the terminal device of the above embodiment.

[0126] An embodiment of the present application also provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed on a computer, the computer executes the technical solution of the driving behavior prediction method of the above embodiment.

[0127] An embodiment of the present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium, and when at least one processor executes the computer program, it can implement the technical solution of the driving behavior prediction method in the above embodiment.

[0128] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of modules is only a logical function division. In actual implementation, there may be other division methods, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or module, which can be electrical, mechanical or other forms.

[0129] Modules described as separate components may or may not be physically separate, and components shown as modules may or may not be physical units, that is, they may be located in one place or distributed across multiple network elements. Some or all of these modules may be selected to implement the solution of this embodiment based on actual needs.

[0130] In addition, the functional modules in the various embodiments of the present application may be integrated into a single processing unit, or each module may exist physically separately, or two or more modules may be integrated into a single unit. The above-mentioned modules may be implemented in the form of hardware or hardware plus software functional units.

[0131] The integrated modules implemented in the form of software function modules can be stored in a computer-readable storage medium. The software function modules stored in a storage medium include a number of instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute some of the steps of the methods of various embodiments of the present application.

[0132] It should be understood that the processor described above may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the present invention may be directly executed by a hardware processor or by a combination of hardware and software modules within the processor.

[0133] The memory may include a high-speed RAM memory, and may also include non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk or an optical disk.

[0134] A bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. Buses can be divided into address buses, data buses, and control buses.

[0135] The storage medium may be implemented by any type of volatile or non-volatile memory device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium may be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0136] An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the storage medium can also exist as discrete components in an electronic control unit or a main control device.

[0137] Those skilled in the art will appreciate that all or part of the steps in the above-described method embodiments can be implemented using hardware associated with program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0138] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A driving behavior prediction method, characterized in that: The method comprises: Constructing a virtual human model, the virtual human model including a joint model for simulating driving movements and postures; Constructing a driving simulation environment, the driving simulation environment including a vehicle model and environmental information, and placing the virtual human model in the driving simulation environment so that the virtual human model interacts with the vehicle model to control the movement of the vehicle model; A deep learning model is constructed, wherein the deep learning model is used to output control actions based on the input driver's first-person perspective data and output them to the virtual human model. The virtual human model controls the vehicle model action in response to the input control actions. The deep learning model includes a brain-like perception decision network and a brain-like control network; wherein the processing flow of the brain-like perception decision network is as follows: using the driver's first-person perspective data as input data, a brain-like feature extraction network aligned with the brain's perception pathway anatomy is constructed through a convolutional neural network and a recurrent neural network structure, extracting the activation response of the input data, and encoding the activation response into a first feature map; the brain-like control network includes several recurrent neural networks and a deep Q network , each recurrent neural network is used to simulate the prefrontal cortex, motor cortex, sensory cortex, spinal cord, red nucleus, cerebellum, VLc, superior colliculus and reticular nucleus. The processing flow of the brain-like control network is: obtaining the first feature map, gradually extracting spatial and temporal features from the first feature map through multi-layer convolution to obtain a second feature map, and obtaining a target Q value based on the second feature map, normalizing, activating, and feedback-adjusting the target Q value in sequence, and outputting a control action. The activation processing uses a linear activation parameter including a slope parameter, and the slope parameter is used to process the negative value area. The feedback adjustment includes: when the target Q value is not within the set threshold range, re-performing multi-layer convolution processing on the first feature map; Acquire a training data set based on real driving data; wherein the training data set includes a plurality of state-action pairs; The deep learning model is trained using the training data set so that the deep learning model maximizes the cumulative reward through two stages of imitation learning and reinforcement learning to obtain a policy function, and the policy function is continuously updated using proximal policy optimization through interaction with environmental information to ultimately obtain a trained deep learning model; and the trained deep learning model is used to realize driving behavior prediction.

2. The driving behavior prediction method according to claim 1, characterized in that: The calculation process of obtaining the target Q value based on the second feature map is: ； Where, represents the target Q value, Indicates immediate reward, represents the discount factor used to weigh current and future rewards, Indicates the next state The maximum Q value is determined by the target network parameters calculate, Indicates the next action.

3. The driving behavior prediction method according to claim 1, characterized in that: Obtain training data sets based on real driving data, including: Acquiring kinematic data of the driver during driving, wherein the kinematic data includes angles, velocities, and accelerations of joints; Extracting key points based on the kinematic data to generate pose data in 3D space; wherein the key points include shoulders, elbows, knees, and / or ankles, and the pose data in 3D space includes multiple data points, each data point corresponding to a driving task, wherein the driving tasks include steering, braking, and acceleration; According to the pose data in the 3D space, a state-action pair is generated.

4. The driving behavior prediction method according to claim 1, characterized in that: The deep learning model is trained using the training data set so that the deep learning model maximizes the cumulative reward through two stages of imitation learning and reinforcement learning to obtain a policy function, and the policy function is continuously updated using proximal policy optimization through interaction with environmental information. The method includes: In the imitation learning stage, the goal is to minimize the error between the predicted action and the expert's actual action, and a loss function is constructed. The training of the imitation learning stage is carried out based on the loss function. The loss function is expressed as: ； Where, represents the loss function, represents the action predicted by the model, Indicates the actual actions of the experts. represents the model parameters, Indicates the total number of demonstration data, Indicates the serial number of the demonstration data, Indicates the status; In the reinforcement learning stage, the policy function is learned through the following formula: ； Where, represents the policy function, E represents the expectation of randomness in the environment, represents the immediate reward obtained at time t, represents the discount factor, Indicates the next state The next choice can maximize Next move , a Indicates the current action. s Indicates the current state; By interacting with environmental information, the policy function is updated through the following objective function: ； Where, represents the objective function of policy optimization, represents the model parameters, represents the advantage function, Represents a hyperparameter that controls the magnitude of the policy update, min represents the minimum function, , represents the probability ratio, π represents pi, a t express t The action of the moment, s t express t The state of the moment, Indicates clipping the ratio to the interval , Indicates that the new policy is in state Next select action The probability of represents the probability that the old policy chooses the same action in the same state.

5. The driving behavior prediction method according to claim 1, characterized in that: After completing imitation learning and reinforcement learning, the trained deep learning model generates control instructions, which are expressed as: ； Where, express t The throttle amplitude at the moment, express t The braking force at each moment, express t The steering wheel's swing angle at that moment, s t express t The state of the moment, are the weights of the network, is the bias term.

6. The driving behavior prediction method according to claim 4, characterized in that: The multi-task reinforcement learning model is configured in the deep learning model. When multiple driving tasks are involved, multiple tasks are learned by sharing network weights. The learning process is expressed as: ； Where, represents the sum of fitting errors of all tasks, Indicates the number of tasks, Indicates the i The Q-value function of a task, represents the estimated Q value obtained by the strategy, Indicates status and actions expectations.

7. The driving behavior prediction method according to any one of claims 1 to 6, characterized in that: After finally obtaining the trained deep learning model, the method further includes: The similarity between real human movements and the movements output by the trained deep learning model is measured, and the model architecture and hyperparameters are adjusted through encoding analysis and representation similarity analysis.

8. A driving behavior prediction device, characterized in that: The apparatus includes a controller configured to: Constructing a virtual human model, the virtual human model including a joint model for simulating driving movements and postures; Constructing a driving simulation environment, the driving simulation environment including a vehicle model and environmental information, and placing the virtual human model in the driving simulation environment so that the virtual human model interacts with the vehicle model to control the movement of the vehicle model; A deep learning model is constructed, wherein the deep learning model is used to output control actions based on the input driver's first-person perspective data and output them to the virtual human model. The virtual human model controls the vehicle model action in response to the input control actions. The deep learning model includes a brain-like perception decision network and a brain-like control network; wherein the processing flow of the brain-like perception decision network is as follows: using the driver's first-person perspective data as input data, a brain-like feature extraction network aligned with the brain's perception pathway anatomy is constructed through a convolutional neural network and a recurrent neural network structure, extracting the activation response of the input data, and encoding the activation response into a first feature map; the brain-like control network includes several recurrent neural networks and a deep Q network , each recurrent neural network is used to simulate the prefrontal cortex, motor cortex, sensory cortex, spinal cord, red nucleus, cerebellum, VLc, superior colliculus and reticular nucleus. The processing flow of the brain-like control network is: obtaining the first feature map, gradually extracting spatial and temporal features from the first feature map through multi-layer convolution to obtain a second feature map, and obtaining a target Q value based on the second feature map, normalizing, activating, and feedback-adjusting the target Q value in sequence, and outputting a control action. The activation processing uses a linear activation parameter including a slope parameter, and the slope parameter is used to process the negative value area. The feedback adjustment includes: when the target Q value is not within the set threshold range, re-performing multi-layer convolution processing on the first feature map; Acquire a training data set based on real driving data; wherein the training data set includes a plurality of state-action pairs; The deep learning model is trained using the training data set so that the deep learning model maximizes the cumulative reward through two stages of imitation learning and reinforcement learning to obtain a policy function, and the policy function is continuously updated using proximal policy optimization through interaction with environmental information to ultimately obtain a trained deep learning model; and the trained deep learning model is used to realize driving behavior prediction.

9. An electronic device, characterized in that: include: a processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the driving behavior prediction method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the driving behavior prediction method according to any one of claims 1 to 7.

Images