Intelligent agent driving training method and system for robot vehicles based on assisted driving

Abstract

The present invention provides a driving training method and system for a robot vehicle based on assisted driving by an intelligent agent, which relates to the field of intelligent driving technology, including obtaining multi-sensor fusion perception data of the robot vehicle, inputting the multi-sensor fusion perception data into a preset deep reinforcement learning model, and generating driving guidance subtasks in three dimensions: lateral control driving guidance, longitudinal control driving guidance, and traffic rule driving guidance according to the driving decision output by the deep reinforcement learning model and in combination with a preset safe driving rule library; perceiving the execution state and the environment of the robot vehicle in real time, inputting the perception results into the hierarchical finite state machine model, thereby triggering corresponding state transfer conditions, and outputting corresponding driving guidance action sequences according to the current state nodes until the corresponding driving guidance subtasks are completed, and converting the driving guidance action sequences into specific vehicle control instructions.

Description

Technical Field

[0001] The present invention relates to intelligent driving technology, and in particular to a driving training method and system for a robot vehicle by an intelligent agent based on assisted driving. Background Art

[0002] In order to achieve fully assisted driving, the intelligent agent needs to have the ability to drive the vehicle independently in a complex and ever-changing traffic environment. At present, assisted driving technology mainly uses the following two methods to train intelligent agents: one is to use machine learning algorithms to train intelligent agents to master driving strategies through a large amount of driving data; the other is to train intelligent agents to learn driving skills in a simulation environment through reinforcement learning and other methods.

[0003] However, the above methods have certain limitations. First, relying solely on real road data for training has problems such as high data collection costs and unbalanced samples, making it difficult to cover all driving scenarios. Second, there are differences between the simulation environment and the real world, which leads to poor performance of the trained intelligent agent in the actual road environment. In addition, the intelligent agent often requires a lot of trial and error during the training process, which poses a safety hazard. Summary of the invention

[0004] The embodiments of the present invention provide a driving training method and system for a robot vehicle by an intelligent agent based on assisted driving, which can solve the problems in the prior art.

[0005] According to a first aspect of the embodiments of the present invention,

[0006] Provides a driving training method for a robot vehicle based on assisted driving by an intelligent agent, including:

[0007] Acquire multi-sensor fusion perception data of a robot vehicle, wherein the multi-sensor fusion perception data includes vehicle surrounding environment information collected by an on-board laser radar, millimeter-wave radar, ultrasonic radar, and panoramic camera, and vehicle self-motion state information collected by an inertial measurement unit and a wheel speed encoder, and the robot vehicle includes a vehicle with an assisted driving function; input the multi-sensor fusion perception data into a preset deep reinforcement learning model, wherein the deep reinforcement learning model is trained by robot vehicle driving data and human driver driving behavior data in a large-scale real road environment, and can output the optimal driving decision for the current scene in real time;

[0008] According to the driving decision output by the deep reinforcement learning model, combined with the preset safe driving rule library, three-dimensional driving guidance subtasks of lateral control driving guidance, longitudinal control driving guidance and traffic rule driving guidance are generated; for each driving guidance subtask, an independent hierarchical finite state machine model is constructed, each hierarchical finite state machine model includes multiple state nodes and state transition conditions, the state nodes correspond to specific driving actions, and the state transition conditions include vehicle state conditions and environmental state conditions;

[0009] The execution state and environment of the robot vehicle are perceived in real time, and the perception results are input into the hierarchical finite state machine model, thereby triggering the corresponding state transfer conditions, and outputting the corresponding driving guidance action sequence according to the current state node until the corresponding driving guidance subtask is completed; the driving guidance action sequence is converted into specific vehicle control instructions, including throttle, brake, and steering control signals, which are sent to the vehicle actuator through the vehicle controller to realize real-time driving guidance and control of the robot vehicle.

[0010] In an optional embodiment,

[0011] The multi-sensor fusion perception data is input into a preset deep reinforcement learning model, which is trained by robot vehicle driving data and human driver driving behavior data in a large-scale real road environment, and can output the optimal driving decision for the current scene in real time, including:

[0012] The deep reinforcement learning model consisting of an action network and a judgment network and built based on a deep deterministic policy gradient is trained using a pre-acquired training data set. The action network parameters are optimized by minimizing the mean square error between the output actions of the action network and the actions of human drivers. At the same time, the instant reward is calculated based on the vehicle state change and the predefined reward function. The judgment network parameters are optimized by minimizing the mean square error between the estimated cumulative reward of the judgment network and the actual cumulative reward.

[0013] The driving decision samples of human drivers in complex scenarios are introduced as additional supervisory signals into the training of deep reinforcement learning models. The cross entropy loss between the output actions of the action network and the driving decision samples is minimized, and the behavior cloning method is used to accelerate the convergence of deep reinforcement learning model training.

[0014] The action network is used to output the action values ​​of multiple alternative driving actions according to the training data set, and the judgment network is used to determine the long-term cumulative reward value according to the training data set and the action value generated by the action network. The alternative driving actions generated by the action network are input into the pre-built state transition model to predict the vehicle state change and environmental feedback after executing each alternative driving action, and quantify them into corresponding immediate reward values;

[0015] The long-term cumulative reward and the immediate reward output by the judgment network are weightedly summed to obtain a comprehensive score for each alternative driving action. The alternative driving action with the highest comprehensive score is selected as the optimal driving decision for the current scenario. The optimal driving decision is then checked for safety in combination with the preset safe driving rule library, and driving decisions that do not meet safety constraints are pruned and replaced. Finally, a safe and executable optimal driving decision for the current scenario is output.

[0016] In an optional embodiment,

[0017] The action network is used to output the action values ​​of multiple alternative driving actions according to the training data set. The judgment network is used to determine the long-term cumulative reward value according to the training data set and the action value generated by the action network. The alternative driving actions generated by the action network are input into the pre-built state transition model to predict the vehicle state changes and environmental feedback after executing each alternative driving action, and quantify them into corresponding immediate reward values ​​including:

[0018] Determining the long-term cumulative reward value includes:

[0019] ;

[0020] in, G t represents the long-term cumulative reward starting from time t, r t Indicates execution of action a t After Moment t The immediate reward value, represents the discount factor used to attenuate the impact of future rewards, represents the instant reward value at time t+k, k Indicates the count value;

[0021] The alternative driving actions generated by the action network are input into the pre-built state transition model to predict the vehicle state changes and environmental feedback after executing each alternative driving action, and quantify them into corresponding immediate reward values ​​including:

[0022] ;

[0023] ;

[0024] Among them, r t Indicates execution of action a t After Moment t The immediate reward value, s t represents the state input at time t, a t represents the driving action generated at time t, f(s t ,at ) represents the state transition function of a state-action pair, represents an action network with parameters θ.

[0025] In an optional embodiment,

[0026] For each driving guidance subtask, an independent hierarchical finite state machine model is constructed. Each hierarchical finite state machine model contains multiple state nodes and state transition conditions. The state nodes correspond to specific driving actions. The state transition conditions include vehicle state conditions and environmental state conditions, including:

[0027] The hierarchical finite state machine model includes a three-layer structure, which is a task layer state machine, a decision layer state machine and an execution layer state machine from top to bottom;

[0028] The task layer state machine divides different driving stages according to the semantic information of the driving task. Each driving stage corresponds to a task layer state node. The transition conditions between adjacent task layer state nodes depend on the vehicle state parameters and environmental state parameters perceived in real time. When a specific state transition condition is met, the decision layer state machine is triggered to perform corresponding driving subtask planning.

[0029] The decision-layer state machine receives the driving phase information output by the task-layer state machine, performs behavior decision-making and path planning in combination with the vehicle state and the environment state, and outputs a series of local path points and corresponding motion control parameters to be executed of the target speed, acceleration, and steering angle. Each decision-layer state node corresponds to a decision-making execution action in a specific scenario.

[0030] The execution layer state machine is responsible for converting the local path and motion control parameters output by the decision layer state machine into low-level control instructions for throttle, brake, and steering, achieving smooth and accurate motion control with reference to the vehicle dynamics model and controller characteristics, and adaptively adjusting the controller parameters based on the feedback information from the vehicle chassis sensor, where each execution layer state node corresponds to a basic control operation.

[0031] In an optional embodiment,

[0032] The driving guidance action sequence is converted into specific vehicle control instructions, including throttle, brake, and steering control signals, which are sent to the vehicle actuator through the vehicle controller to achieve real-time driving guidance and control of the robot vehicle, including:

[0033] Parsing the driving instruction action sequence to extract semantic information of each driving instruction action sequence, wherein the semantic information includes an action type, a target value, and a duration;

[0034] Generate corresponding vehicle control instructions according to the semantic information of each driving guidance action sequence, wherein the vehicle control instructions include throttle control instructions, brake control instructions and steering control instructions, wherein the throttle control instructions and brake control instructions are calculated by a preset control algorithm according to the current speed and expected speed of the vehicle, and the steering control instructions are calculated by a preset control algorithm according to the current position, heading and expected trajectory of the vehicle;

[0035] Smoothing the generated vehicle control instructions, using a preset smoothing algorithm to filter the throttle control instructions, brake control instructions and steering control instructions, so as to remove high-frequency disturbances and mutations in the vehicle control instructions and obtain smooth vehicle control instructions;

[0036] The smooth vehicle control instructions are sent to the vehicle actuator through the vehicle controller, which controls the vehicle actuator to output the corresponding physical control quantity, thereby realizing real-time control of the vehicle.

[0037] In an optional embodiment,

[0038] The method further comprises:

[0039] Constructing a dynamic model describing the lateral and longitudinal coupled motion characteristics of the vehicle, wherein the dynamic model includes a lateral position error, a heading angle error, and a sideslip angle of the center of mass representing the lateral motion state of the vehicle, a longitudinal position and a vehicle speed representing the longitudinal motion state of the vehicle, and a front wheel steering angle and a driving / braking force as control variables;

[0040] Constructing an integrated objective function that considers both lateral control performance and longitudinal control performance, the integrated objective function includes lateral position error, heading angle error, center of mass sideslip angle, longitudinal position error and vehicle speed error, as well as front wheel steering angle change and driving / braking force change, and weighting each error and control amount change by setting a weight matrix;

[0041] On the basis of vehicle physical constraints and controller performance constraints, tire adhesion constraints are introduced to establish joint constraints. The tire adhesion constraints restrict the tire from side slipping and sliding under complex working conditions through the relationship between tire vertical load, longitudinal force and lateral force.

[0042] The dynamic model, the integrated objective function and the joint constraint conditions are integrated into a quadratic optimization problem, and a numerical optimization algorithm is used to solve the quadratic optimization problem to obtain an optimal control sequence of the front wheel steering angle and the driving / braking force; the solved optimal control sequence is updated in real time according to the rolling optimization strategy of the model predictive control, the first control quantity is executed in each control cycle, and the remaining control quantities are used as initial values ​​for optimization in the next cycle, so as to realize coordinated tracking control of the vehicle's lateral and longitudinal motions.

[0043] According to a second aspect of the embodiments of the present invention,

[0044] Provides a driving training system for robot vehicles based on intelligent assisted driving, including:

[0045] The first unit is used to obtain multi-sensor fusion perception data of a robot vehicle, wherein the multi-sensor fusion perception data includes vehicle surrounding environment information collected by an on-board laser radar, millimeter-wave radar, ultrasonic radar and panoramic camera, and vehicle self-motion state information collected by an inertial measurement unit and a wheel speed encoder, and the robot vehicle includes a vehicle with an assisted driving function; the multi-sensor fusion perception data is input into a preset deep reinforcement learning model, which is trained by robot vehicle driving data and human driver driving behavior data in a large-scale real road environment, and can output the optimal driving decision for the current scene in real time;

[0046] The second unit is used to generate driving guidance subtasks in three dimensions, namely, lateral control driving guidance, longitudinal control driving guidance, and traffic rule driving guidance, based on the driving decision output by the deep reinforcement learning model and in combination with a preset safe driving rule library; for each driving guidance subtask, an independent hierarchical finite state machine model is constructed, each hierarchical finite state machine model includes multiple state nodes and state transition conditions, the state nodes correspond to specific driving actions, and the state transition conditions include vehicle state conditions and environmental state conditions;

[0047] The third unit is used to perceive the execution status and environment of the robot vehicle in real time, input the perception results into the hierarchical finite state machine model, thereby triggering the corresponding state transfer conditions, and output the corresponding driving guidance action sequence according to the current state node until the corresponding driving guidance subtask is completed; the driving guidance action sequence is converted into specific vehicle control instructions, including throttle, brake, and steering control signals, which are sent to the vehicle actuator through the vehicle controller to realize real-time driving guidance and control of the robot vehicle.

[0048] According to a third aspect of the embodiments of the present invention,

[0049] An electronic device is provided, comprising:

[0050] processor;

[0051] a memory for storing processor-executable instructions;

[0052] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0053] According to a fourth aspect of the embodiments of the present invention,

[0054] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0055] This application can build a complete assisted driving decision control system through the organic combination of three levels of state machines. The task layer is responsible for the macro control of the driving process and the understanding of task semantics, the decision layer is responsible for the behavior decision and path planning in the local scene, and the execution layer is responsible for parsing the decision instructions into control operations that can be executed by the underlying hardware. The three-layer state machine works together to achieve rapid response and adaptation to dynamic traffic scenes.

[0056] Applying continuous and smooth control instructions to the vehicle actuator can enable the robot vehicle to accurately execute the planned driving actions, complete complex driving behaviors such as acceleration and deceleration, following the vehicle, changing lanes, and respond to changes in the surrounding environment in real time. The control execution method proposed in this application is universal and can be integrated into an assisted driving system with end-to-end learning, imitation learning, reinforcement learning, etc. as the "last mile" from top-level decision output to bottom-level execution. While ensuring vehicle stability and passenger comfort, it provides key support for the engineering implementation of the assisted driving system. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 It is a flowchart of a method for training a robot vehicle by an intelligent agent based on assisted driving according to an embodiment of the present invention;

[0058] Figure 2 It is a schematic diagram of the structure of a driving training system for a robot vehicle based on assisted driving by an intelligent agent according to an embodiment of the present invention. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0060] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0061] Figure 1 FIG. 1 is a flow chart of a method for training a robot vehicle by an intelligent agent based on assisted driving according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0062] S101. Acquire multi-sensor fusion perception data of the robot vehicle, the multi-sensor fusion perception data including vehicle surrounding environment information collected by vehicle-mounted laser radar, millimeter-wave radar, ultrasonic radar and panoramic camera, and vehicle self-motion state information collected by inertial measurement unit and wheel speed encoder, the robot vehicle including a vehicle with assisted driving function; input the multi-sensor fusion perception data into a preset deep reinforcement learning model, the deep reinforcement learning model is trained by robot vehicle driving data and human driver driving behavior data in a large-scale real road environment, and can output the optimal driving decision for the current scene in real time;

[0063] S102. Generate driving guidance subtasks in three dimensions: lateral control driving guidance, longitudinal control driving guidance, and traffic rule driving guidance, based on the driving decision output by the deep reinforcement learning model and in combination with a preset safe driving rule library; construct an independent hierarchical finite state machine model for each driving guidance subtask, each hierarchical finite state machine model comprising a plurality of state nodes and state transition conditions, the state nodes corresponding to specific driving actions, and the state transition conditions comprising vehicle state conditions and environmental state conditions;

[0064] S103. Perceive the execution status and environment of the robot vehicle in real time, input the perception results into the hierarchical finite state machine model, thereby triggering the corresponding state transfer conditions, and output the corresponding driving guidance action sequence according to the current state node until the corresponding driving guidance subtask is completed; convert the driving guidance action sequence into specific vehicle control instructions, including throttle, brake, and steering control signals, and send them to the vehicle actuator through the vehicle controller to realize real-time driving guidance and control of the robot vehicle.

[0065] For example, multi-sensor fusion perception: The robot vehicle is equipped with multiple sensors such as laser radar, millimeter-wave radar, ultrasonic radar, panoramic camera, etc., which can comprehensively collect environmental information around the vehicle. At the same time, the vehicle is also equipped with an inertial measurement unit and a wheel speed encoder to collect the vehicle's own motion state information. Multi-sensor fusion perception technology can provide real and rich data support for intelligent driving teaching. By analyzing these data, students can deeply understand the interactive relationship between the vehicle and the environment and master the perception and decision-making methods in complex driving scenarios.

[0066] Deep reinforcement learning model: This solution uses large-scale real-world robot vehicle driving data and human driver driving behavior data to train a deep reinforcement learning model. The model can output the optimal driving decision for the current scenario in real time. Applying deep reinforcement learning to intelligent driving teaching can enable students to learn more intelligent and efficient driving strategies. At the same time, the deep reinforcement learning model can also be used as a teaching evaluation standard to judge whether the student's driving behavior is reasonable and safe.

[0067] Driving guidance subtasks: Based on the driving decisions output by the deep reinforcement learning model and combined with the preset safe driving rule library, driving guidance subtasks in three dimensions, namely lateral control, longitudinal control and traffic rules, are generated. This hierarchical decomposition teaching method can help students master driving skills in a more systematic and comprehensive manner. Through special training for each subtask, students can receive targeted reinforcement in lateral control (such as lane keeping), longitudinal control (such as vehicle distance control), traffic rules (such as traffic light recognition), etc., and gradually improve their driving ability.

[0068] Hierarchical finite state machine model: For each driving guidance subtask, an independent hierarchical finite state machine model is constructed. Each model contains multiple state nodes (corresponding to specific driving actions) and state transition conditions (including vehicle state and environment state). This state machine-based modeling method can divide complex driving behaviors into several discrete states and clarify the transition logic between states. In intelligent driving teaching, the hierarchical finite state machine model can be used to describe the behavior patterns in typical driving scenarios, helping students understand the relationship between driving decisions and vehicle and environment states, so as to master the basic strategies and laws of driving.

[0069] Real-time driving guidance and control: By sensing the vehicle state and environment state in real time, the corresponding state transition conditions in the hierarchical finite state machine model are triggered, and the corresponding driving guidance action sequence is output according to the current state node. The driving guidance action is further converted into specific control instructions such as throttle, brake, and steering, and sent to the vehicle for execution to achieve real-time driving guidance and control. Applying this method to intelligent driving teaching can achieve real-time monitoring and correction of students' driving behavior. When students make mistakes or unsafe operations, the system can give voice or visual prompts in time to guide students to perform standardized operations. At the same time, during the actual vehicle teaching process, teachers can remotely monitor the driving status of students and make safety interventions when necessary.

[0070] Assisted driving function: The robot vehicle used in this solution has an assisted driving function, which can automatically provide safety assistance when the trainee operates improperly, such as automatic braking, lane departure warning, etc. The introduction of assisted driving function can reduce the safety risks of intelligent driving teaching, enhance the confidence of trainees in the actual vehicle training process, and deepen the trainees' understanding and trust in the intelligent driving system through human-machine collaboration.

[0071] In an optional embodiment,

[0072] The multi-sensor fusion perception data is input into a preset deep reinforcement learning model, which is trained by robot vehicle driving data and human driver driving behavior data in a large-scale real road environment, and can output the optimal driving decision for the current scene in real time, including:

[0073] The deep reinforcement learning model consisting of an action network and a judgment network and built based on a deep deterministic policy gradient is trained using a pre-acquired training data set. The action network parameters are optimized by minimizing the mean square error between the output actions of the action network and the actions of human drivers. At the same time, the instant reward is calculated based on the vehicle state change and the predefined reward function. The judgment network parameters are optimized by minimizing the mean square error between the estimated cumulative reward of the judgment network and the actual cumulative reward.

[0074] The driving decision samples of human drivers in complex scenarios are introduced as additional supervisory signals into the training of deep reinforcement learning models. The cross entropy loss between the output actions of the action network and the driving decision samples is minimized, and the behavior cloning method is used to accelerate the convergence of deep reinforcement learning model training.

[0075] The action network is used to output the action values ​​of multiple alternative driving actions according to the training data set, and the judgment network is used to determine the long-term cumulative reward value according to the training data set and the action value generated by the action network. The alternative driving actions generated by the action network are input into the pre-built state transition model to predict the vehicle state change and environmental feedback after executing each alternative driving action, and quantify them into corresponding immediate reward values;

[0076] The long-term cumulative reward and the immediate reward output by the judgment network are weightedly summed to obtain a comprehensive score for each alternative driving action. The alternative driving action with the highest comprehensive score is selected as the optimal driving decision for the current scenario. The optimal driving decision is then checked for safety in combination with the preset safe driving rule library, and driving decisions that do not meet safety constraints are pruned and replaced. Finally, a safe and executable optimal driving decision for the current scenario is output.

[0077] Exemplarily, the present application proposes an assisted driving decision-making method based on deep reinforcement learning, which outputs the optimal driving decision for the current scene in real time by inputting multi-sensor fusion perception data into a pre-trained deep reinforcement learning model. The deep reinforcement learning model is trained by large-scale robot vehicle driving data and human driver driving behavior data in real road environments, and can effectively learn driving strategies in complex traffic scenarios. A deep deterministic policy gradient algorithm is introduced to construct a deep reinforcement learning architecture, model training is guided by the behavioral cloning method, and a state transition model is used to assist in driving decision evaluation, and finally an executable optimal decision is output in combination with a safe driving rule base. The specific implementation steps are as follows:

[0078] First, a large-scale training data set is collected, including vehicle driving data (such as speed, acceleration, heading angle, etc.) in real road environments and human driver driving behavior data (such as steering wheel angle, throttle, brake, etc.). Then, a deep reinforcement learning model consisting of an action network (Actor) and a critic network (Critic) is constructed based on deep deterministic policy gradient (DDPG). Among them, the action network receives the current state (such as vehicle state, environmental information) and outputs continuous driving actions (such as steering wheel angle, acceleration); the critic network receives the state and action, and outputs the long-term cumulative reward value of the action.

[0079] During the training process, the action network parameters are optimized by minimizing the mean square error between the output actions of the action network and the actions of the human driver. For example, if the current driver's turning angle is 5 degrees and the action network output is 8 degrees, the optimization goal is to minimize (8-5)^2. At the same time, the immediate reward is calculated based on the vehicle state change (such as the degree of deviation from the reference trajectory) and the predefined reward function (such as the greater the distance to the obstacle, the higher the reward), and the judgment network parameters are optimized by minimizing the mean square error between the estimated cumulative reward of the judgment network and the actual cumulative reward.

[0080] In order to further improve the training effect and convergence speed of the deep reinforcement learning model, this step introduces the driving decision samples of human drivers in complex scenarios (such as intersections and construction sections) as additional supervisory signals into the model training. By minimizing the cross entropy loss between the output action of the action network and the driving decision samples, the model output is close to human driving decisions. For example, in the intersection scenario, the driving decision samples are [slow down, brake, observe, start, accelerate], and if the action network output is [slow down, observe, accelerate], the model output is corrected by minimizing the cross entropy of the two sequences. In addition, using the behavior cloning method, the action network is first trained with driving decision samples to enable it to quickly learn human driving behavior strategies, and then fine-tuned in combination with reinforcement learning to accelerate the convergence of model training.

[0081] This step uses the trained deep reinforcement learning model to evaluate and select driving decisions. First, the current vehicle state and environmental information are input into the action network to generate multiple alternative driving actions (such as steering wheel angles of -5 degrees, 0 degrees, +5 degrees) and their corresponding action values. Then, the judgment network is used to estimate the long-term cumulative reward based on the current state and alternative action values. Next, the alternative driving actions are input into a pre-built state transition model (such as a physical model, a kinematic model) to predict the changes in vehicle state (such as speed, position) and environmental feedback (such as distance from obstacles) after executing each action, and quantify them into immediate reward values ​​based on the preset reward function.

[0082] Finally, the long-term cumulative reward and the immediate reward output by the judgment network are weighted and summed to obtain the comprehensive score of each alternative driving action. For example, the long-term cumulative reward of an alternative action is 5 points, the immediate reward is 2 points, and the weights are 0.8 and 0.2 respectively. The comprehensive score of the action is 5*0.8+2*0.2=4.4 points. The driving action with the highest comprehensive score is selected as the optimal decision output in the current scenario.

[0083] To ensure that the output driving decision meets the safety constraints, this step introduces a preset safe driving rule base to perform a secondary check on the optimal driving decision. The rule base contains a series of IF-THEN rules based on expert experience, such as "IF the distance to the front vehicle is less than 5m THEN slow down to less than 30km / h". For decisions that do not meet the rule constraints, a pruning strategy is adopted to eliminate or replace them to ensure that the final output driving decision is safe and executable. For example, if the optimal decision is "maintain the current speed through the intersection", but the rule base contains the provision that "the vehicle must stop when the red light is on", the decision will be corrected to "slow down to the stop line".

[0084] After the above steps, the method of the present application can realize continuous, smooth and safe assisted driving decisions in complex traffic scenarios, provide reliable upper-level input for unmanned vehicle environmental perception and motion planning, and has good intelligent decision-making ability and practical application value.

[0085] In an optional embodiment,

[0086] The action network is used to output the action values ​​of multiple alternative driving actions according to the training data set. The judgment network is used to determine the long-term cumulative reward value according to the training data set and the action value generated by the action network. The alternative driving actions generated by the action network are input into the pre-built state transition model to predict the vehicle state changes and environmental feedback after executing each alternative driving action, and quantify them into corresponding immediate reward values ​​including:

[0087] Determining the long-term cumulative reward value includes:

[0088] ;

[0089] in, G t represents the long-term cumulative reward starting from time t, r t Indicates execution of action a t After Moment t The immediate reward value, represents the discount factor used to attenuate the impact of future rewards, represents the instant reward value at time t+k, k Indicates the count value;

[0090] The alternative driving actions generated by the action network are input into the pre-built state transition model to predict the vehicle state changes and environmental feedback after executing each alternative driving action, and quantify them into corresponding immediate reward values ​​including:

[0091] ;

[0092] ;

[0093] Among them, r t Indicates execution of action a t After Moment t The immediate reward value, s t represents the state input at time t, a t represents the driving action generated at time t, f(s t ,a t ) represents the state transition function of a state-action pair, represents an action network with parameters θ.

[0094] For example, by introducing a deep reinforcement learning model, the optimal driving strategy in complex traffic scenarios can be effectively learned. The deep reinforcement learning model is trained using massive real road environment data and human driving behavior samples, enabling it to grasp vehicle dynamics characteristics and traffic regulations, output continuous, smooth, and safe driving decisions, and improve the environmental adaptability and intelligent decision-making level of unmanned vehicles.

[0095] The action-critic network architecture is constructed using a deep deterministic policy gradient algorithm, which can take into account both the exploratory and exploitative nature of the strategy. The action network is responsible for generating alternative driving actions, and the critic network is responsible for evaluating the long-term cumulative rewards of the actions. The two networks are optimized alternately and iteratively, which ensures the convergence of the strategy while avoiding falling into the local optimum too early, and improves the generalization ability and robustness of the decision-making model. Human driving decision samples are integrated into the reinforcement learning training process, and prior knowledge is fully utilized to guide model learning. The action network is pre-trained by behavioral cloning, and then fine-tuned with the goal of minimizing the cross entropy loss, which can significantly accelerate the strategy search and model convergence, reduce the development difficulty and iteration cycle of the intelligent driving system, and has high engineering practical value.

[0096] The state transition model is introduced to dynamically evaluate the immediate reward of driving decisions, enhancing the interpretability and controllability of decisions. Based on the current vehicle state and environmental information, the physical model is used to predict the consequences of alternative decisions, quantitatively analyze the pros and cons of decisions, and avoid short-sighted or risky operations, which reflects the safety and reliability requirements of the assisted driving system.

[0097] Integrate preset safe driving rules into the decision output link, build a hybrid architecture of "reinforcement learning + rule base", and take into account the flexibility and safety of decision-making. Through secondary verification and correction of driving decisions output by deep reinforcement learning models, eliminate or replace dangerous decisions that do not comply with traffic rules and safety constraints, thereby maximizing the safety of vehicles and traffic participants and meeting regulatory requirements in the field of assisted driving.

[0098] In an optional embodiment,

[0099] For each driving guidance subtask, an independent hierarchical finite state machine model is constructed. Each hierarchical finite state machine model contains multiple state nodes and state transition conditions. The state nodes correspond to specific driving actions. The state transition conditions include vehicle state conditions and environmental state conditions, including:

[0100] The hierarchical finite state machine model includes a three-layer structure, which is a task layer state machine, a decision layer state machine and an execution layer state machine from top to bottom;

[0101] The task layer state machine divides different driving stages according to the semantic information of the driving task. Each driving stage corresponds to a task layer state node. The transition conditions between adjacent task layer state nodes depend on the vehicle state parameters and environmental state parameters perceived in real time. When a specific state transition condition is met, the decision layer state machine is triggered to perform corresponding driving subtask planning.

[0102] The decision-layer state machine receives the driving phase information output by the task-layer state machine, performs behavior decision-making and path planning in combination with the vehicle state and the environment state, and outputs a series of local path points and corresponding motion control parameters to be executed of the target speed, acceleration, and steering angle. Each decision-layer state node corresponds to a decision-making execution action in a specific scenario.

[0103] The execution layer state machine is responsible for converting the local path and motion control parameters output by the decision layer state machine into low-level control instructions for throttle, brake, and steering, achieving smooth and accurate motion control with reference to the vehicle dynamics model and controller characteristics, and adaptively adjusting the controller parameters based on the feedback information from the vehicle chassis sensor, where each execution layer state node corresponds to a basic control operation.

[0104] Exemplarily, the present application proposes a hierarchical finite state machine (HFSM) model for realizing task decomposition and behavioral decision-making in complex driving scenarios. For each driving guidance subtask, an independent HFSM model is constructed, which includes multiple state nodes and state transition conditions. Among them, the state node corresponds to a specific driving action, and the state transition condition includes two types of parameters: vehicle state and environment state. By reasonably dividing the state space and transfer logic, dynamic modeling and behavioral decision-making of the driving process can be realized. Specifically, the present application adopts a three-layer structure to organize the HFSM model, which is the task layer, decision layer and execution layer from top to bottom, respectively responsible for driving task semantic understanding, behavioral decision planning and underlying motion control. The following is a detailed description of the construction steps and process of the HFSM model.

[0105] The task layer state machine is responsible for dividing the entire driving task into several consecutive driving phases, and switching between phases according to the semantic information of vehicle state parameters (such as vehicle speed and acceleration) and environmental state parameters (such as road type and traffic flow density). For example, the task of "automatically merging into the main road on the highway ramp" is divided into "ramp deceleration phase", "ramp detection phase", "merge preparation phase", "lane change merging phase" and "cruise recovery phase".

[0106] Each driving phase corresponds to a state node in the task layer state machine. The transition between adjacent state nodes depends on whether the state parameters perceived in real time meet the predefined transition conditions. When a specific transition condition is triggered, the task layer state machine will send the corresponding driving phase switching instruction to the decision layer state machine of the next layer, requesting to update the decision and plan.

[0107] For example, if the current vehicle speed is 60km / h and the distance to the vehicle in front is 50m, the task layer state machine will remain in the "constant speed following stage". Once the vehicle speed drops to 40km / h, the distance to the vehicle in front is shortened to 30m, and the duration exceeds 3s, the transition condition to enter the "deceleration and parking stage" is met, and the task layer state machine will switch to this stage node.

[0108] The decision-making layer state machine receives the driving phase information output by the task layer state machine, and combines the vehicle state and the environment state to perform local path planning and behavior decision-making. Each decision-making layer state node corresponds to a decision execution action in a specific scenario, such as "go straight through", "turn left and wait", "change lanes and overtake", etc. The decision-making layer state machine outputs a series of local path points (waypoints) to be executed and the corresponding motion control parameters such as target speed, acceleration, steering angle, etc. at the current driving phase node.

[0109] For example, in the "left turn at intersection" scenario, the decision-making state machine may include state nodes such as "straight deceleration", "waiting area", "turning area", and "turning recovery". Initially, the decision-making state machine is located at the "straight deceleration" node, and outputs a speed curve that decelerates to 10m / s. When the vehicle enters the waiting area in the center of the intersection, it meets the transfer condition of "no obstacles in the area & TTC of vehicles on the left>6s", and the decision-making state machine switches to the "turning area" node and outputs a turning instruction with a steering angular velocity of 5deg / s and a duration of 2s. After the left turn is completed, it switches to the "turning recovery" node and outputs a path to restore to the original heading.

[0110] It should be noted that due to the dynamic changes in the traffic environment, the decision-making state machine needs to have the ability to re-plan. When encountering an emergency (such as a pedestrian intruding) that makes the current decision plan no longer safe, it should be able to adjust the state nodes and transfer strategies in time, and take emergency avoidance and other response measures in time to ensure driving safety.

[0111] The execution layer state machine is responsible for converting the local path and motion parameters output by the decision layer into executable low-level control instructions, such as throttle opening, brake pressure, steering wheel angle, etc. Referring to the vehicle dynamics characteristics and the low-level controller model, the execution layer will generate a series of continuous and smooth operation instruction sequences, and make adaptive adjustments based on the real-time feedback from the chassis sensors to achieve precise motion tracking control.

[0112] For example, when receiving the "stop at 100m" instruction from the decision layer, the execution layer state machine first switches to the "deceleration and braking" node, and outputs a pressure curve from the current brake pressure (such as 20kPa) to the target value (such as 100kPa) according to the braking force model. Then it switches to the "stop and hold" node, outputs a constant 100kPa brake pressure, and monitors the speed change at the same time. If the speed does not drop to 0 within the predetermined time (such as 5s), the "emergency brake" node is triggered to increase the brake pressure to the maximum (such as 150kPa) until the vehicle stops completely.

[0113] During the control process, the execution layer state machine also needs to adjust the controller parameters in real time according to chassis feedback such as wheel speed and acceleration. Taking the engine speed as an example, assuming that the current speed is 1200rpm, the target speed is 1500rpm, and the controller outputs a throttle opening of 40%. If the actual speed only reaches 1350rpm after 1s, it means that the actual acceleration response is slower than expected, and the controller needs to increase the throttle opening accordingly (such as to 50%) to ensure that the target speed is reached as soon as possible. Similarly, if the actual speed exceeds 1500rpm within 0.5s, the throttle opening needs to be reduced to avoid overshoot.

[0114] Through the organic combination of the above three levels of state machines, a complete assisted driving decision control system can be constructed. The task layer is responsible for the macro control of the driving process and the semantic understanding of the task, the decision layer is responsible for the behavioral decision and path planning in the local scene, and the execution layer is responsible for parsing the decision instructions into control operations that can be executed by the underlying hardware. The three-layer state machine works together to achieve rapid response and adaptation to dynamic traffic scenes. At the same time, since the transfer conditions of the state machine nodes at each layer can be defined modularly, the entire HFSM model has good reconfigurability and generalization capabilities, and can be easily applied to different driving scenarios.

[0115] In an optional embodiment,

[0116] The driving guidance action sequence is converted into specific vehicle control instructions, including throttle, brake, and steering control signals, which are sent to the vehicle actuator through the vehicle controller to achieve real-time driving guidance and control of the robot vehicle, including:

[0117] Parsing the driving instruction action sequence to extract semantic information of each driving instruction action sequence, wherein the semantic information includes an action type, a target value, and a duration;

[0118] Generate corresponding vehicle control instructions according to the semantic information of each driving guidance action sequence, wherein the vehicle control instructions include throttle control instructions, brake control instructions and steering control instructions, wherein the throttle control instructions and brake control instructions are calculated by a preset control algorithm according to the current speed and expected speed of the vehicle, and the steering control instructions are calculated by a preset control algorithm according to the current position, heading and expected trajectory of the vehicle;

[0119] Smoothing the generated vehicle control instructions, using a preset smoothing algorithm to filter the throttle control instructions, brake control instructions and steering control instructions, so as to remove high-frequency disturbances and mutations in the vehicle control instructions and obtain smooth vehicle control instructions;

[0120] The smooth vehicle control instructions are sent to the vehicle actuator through the vehicle controller, which controls the vehicle actuator to output the corresponding physical control quantity, thereby realizing real-time control of the vehicle.

[0121] For example, this application converts the driving guidance action sequence output by the upper-level decision module into control instructions that can be executed by the vehicle's underlying controller, and finally applies it to the vehicle through the actuator to achieve real-time driving control of the robot vehicle. The core is to establish a mapping mechanism between driving action semantics and control instructions, and use smoothing and other methods to ensure the continuity and stability of control instructions. The following are detailed technical implementation steps.

[0122] The driving guidance action sequence output by the decision layer is semantically parsed to extract the semantic information of each atomic driving action. An atomic driving action refers to a smallest indivisible driving operation, such as "accelerate to 60km / h", "stop at 50m", "turn left at an angular velocity of 5deg / s", etc. Each atomic driving action can be represented by a triple (action_type, target_value, duration), corresponding to the action type, target value and duration respectively.

[0123] Taking the driving instruction action of "reducing the vehicle speed from 60km / h to 40km / h within 100m" as an example, it can be decomposed into two atomic driving actions:

[0124] (decelerate, 40km / h, 8s) / / Assume that the average deceleration time for 100m is 8s;

[0125] (uniform_speed, 40km / h, 0) / / Maintain vehicle speed after deceleration. Duration is unknown, recorded as 0;

[0126] Here, action_type is a set of predefined action types, such as {accelerate, decelerate, uniform_speed, stop, turn_left, turn_right, lane_change}, etc.; target_value represents physical quantities such as target speed and target angular velocity according to different action_type; duration represents the duration of the action, and if the duration is uncertain, it is recorded as 0.

[0127] Through the above parsing process, a complex driving guidance action sequence can be decomposed into several semantically clear atomic driving actions, providing a basis for the generation of subsequent control instructions.

[0128] According to the parsed atomic driving action semantics, the corresponding vehicle control instructions are generated using predefined mapping rules. The control instructions mainly include three categories: throttle instructions, brake instructions and steering instructions, which correspond to the longitudinal acceleration and deceleration control and lateral steering control of the vehicle respectively. Specifically:

[0129] For (accelerate, v_target, t) and (decelerate, v_target, t) actions, the accelerator opening or brake pressure change curve within a period of time t is calculated based on the vehicle's current speed v_current and target speed v_target through the preset speed control algorithm. Common speed control algorithms include PID control, optimal control, adaptive cruise control (ACC), etc.

[0130] For example, assuming that PID control is used, the speed deviation value (v_target - v_current) is used as the controller input, the throttle opening is used as the output, and the proportional, integral, and differential coefficients are set to Kp=0.8, Ki=0.1, and Kd=0.2. If the current vehicle speed is 50km / h and it is expected to accelerate to 70km / h for 5s, the throttle opening change curve within 5s is as follows:

[0131] [0s: 35%, 1s: 50%, 2s: 62%, 3s: 70%, 4s: 75%, 5s: 76%];

[0132] For actions such as (turn_left, omega, t) and (turn_right, omega, t), the steering wheel angle change curve within a period of time t is calculated through the preset steering control algorithm according to the vehicle's current position, heading and expected steering angular velocity omega. Commonly used steering control algorithms include Stanley method and Pure Pursuit method.

[0133] For example, assuming the vehicle's current heading is 30 degrees, it needs to turn left to 45 degrees within 3 seconds, and the steering angular velocity is set to 5 degrees / s. The angular velocity can be converted to the front wheel angle phi using the lateral kinematics formula:

[0134] phi = arctan(omega * L / v);

[0135] Where L is the wheelbase of the vehicle and v is the current speed. Assuming L=2.5m, v=10m / s, we can get:

[0136] phi = arctan(5 * pi / 180 * 2.5 / 10) = 0.22 rad = 12.6 deg;

[0137] Then, the Pure Pursuit method is used to calculate the steering wheel angle change from 0 to 12.6 degrees within 3 seconds:

[0138] [0s: 0 deg, 0.5s: 3.2 deg, 1s: 6.3 deg, 1.5s: 8.7 deg, 2s: 10.5 deg, 3s: 12.6 deg];

[0139] Through a similar method, all atomic driving actions can be mapped into specific sequences of throttle, brake and steering control instructions.

[0140] Since vehicle control has high requirements for stability and comfort, the generated control instructions need to be smoothed. Digital filtering and other methods are used to remove the possible spike noise and high-frequency jitter in the control instruction sequence to obtain a smooth and continuous control instruction curve.

[0141] Commonly used smoothing algorithms include moving average filtering, low-pass filtering, Kalman filtering, etc. Taking Kalman filtering as an example, the system state is defined as the throttle opening, brake pressure and steering wheel angle, and the observed value is the ideal control instruction calculated. The Kalman filter estimates a smooth and close to real state change curve.

[0142] Assume that the initial value of the throttle opening is 30%, the standard deviation is 5%, the process noise is 1%, and the measurement noise is 2%. If the throttle openings observed in three consecutive cycles (such as 0.1s) are [35%, 37%, 34%] respectively, the smoothed value estimated by the Kalman filter may be:

[0143] [30% ->32% ->34% ->35%];

[0144] It can be seen that the control command curve after smoothing changes more smoothly, which helps to improve driving safety and ride comfort.

[0145] The smoothed control instructions are sent to the vehicle's actuators (such as electronic throttle, hydraulic brake pump, electric power steering) through the vehicle controller (such as CAN bus), controlling the actuators to output corresponding physical control quantities (such as throttle opening, brake pressure, steering assist torque), and finally applied to the vehicle's accelerator pedal, brake pedal, steering wheel and other control devices to achieve longitudinal and lateral motion control of the vehicle.

[0146] At the same time, the bottom-level controller performs closed-loop correction on the control instructions based on the state feedback of the actuator (such as actual throttle opening and actual steering angle), and reports the corrected state quantity to the upper-level decision-making control module, forming a closed loop of planning, decision-making, and control.

[0147] For example, if the planned throttle opening command at time t is 40%, and the actual throttle opening is 38%, it means that the actual acceleration effect is weaker than expected. The controller feeds back the actual throttle opening to the upper layer, and the planning module adjusts the throttle opening command (such as 45%) at the next time (t+1) accordingly until the actual acceleration meets expectations. Similarly, closed-loop control of lateral position can also be achieved through steering angle feedback.

[0148] Through the above four steps, a complete mapping mechanism from abstract driving action semantics to specific control execution can be established. Applying continuous and smooth control instructions to the vehicle actuator can enable the robot vehicle to accurately execute the planned driving actions, complete complex driving behaviors such as acceleration and deceleration, following the vehicle, changing lanes, and respond to changes in the surrounding environment in real time. The control execution method proposed in this application is universal and can be integrated into an assisted driving system with end-to-end learning, imitation learning, reinforcement learning, etc. as the "last mile" from top-level decision output to bottom-level execution. While ensuring vehicle stability and passenger comfort, it provides key support for the engineering implementation of the assisted driving system.

[0149] In an optional embodiment,

[0150] The method further comprises:

[0151] Constructing a dynamic model describing the lateral and longitudinal coupled motion characteristics of the vehicle, wherein the dynamic model includes a lateral position error, a heading angle error, and a sideslip angle of the center of mass representing the lateral motion state of the vehicle, a longitudinal position and a vehicle speed representing the longitudinal motion state of the vehicle, and a front wheel steering angle and a driving / braking force as control variables;

[0152] Constructing an integrated objective function that considers both lateral control performance and longitudinal control performance, the integrated objective function includes lateral position error, heading angle error, center of mass sideslip angle, longitudinal position error and vehicle speed error, as well as front wheel steering angle change and driving / braking force change, and weighting each error and control amount change by setting a weight matrix;

[0153] On the basis of vehicle physical constraints and controller performance constraints, tire adhesion constraints are introduced to establish joint constraints. The tire adhesion constraints restrict the tire from side slipping and sliding under complex working conditions through the relationship between tire vertical load, longitudinal force and lateral force.

[0154] The dynamic model, the integrated objective function and the joint constraint conditions are integrated into a quadratic optimization problem, and a numerical optimization algorithm is used to solve the quadratic optimization problem to obtain an optimal control sequence of the front wheel steering angle and the driving / braking force; the solved optimal control sequence is updated in real time according to the rolling optimization strategy of the model predictive control, the first control quantity is executed in each control cycle, and the remaining control quantities are used as initial values ​​for optimization in the next cycle, so as to realize coordinated tracking control of the vehicle's lateral and longitudinal motions.

[0155] Exemplarily, this application introduces an integrated control method that takes into account the lateral and longitudinal coupling characteristics of the vehicle. Traditional assisted driving control often decouples lateral control (such as steering tracking) and longitudinal control (such as speed tracking), ignoring the mutual influence between the two during vehicle motion. This paper adopts the idea of ​​Model Predictive Control (MPC), describes the lateral and longitudinal coupling characteristics of the vehicle through a dynamic model, constructs an integrated objective function to simultaneously optimize the lateral and longitudinal control performance, and solves the optimal control sequence to achieve coordinated control of the vehicle, thereby obtaining higher trajectory tracking accuracy and driving stability. The following are detailed controller design steps.

[0156] In order to accurately describe the motion characteristics of the vehicle under complex conditions such as steering, acceleration and deceleration, a dynamic model that includes both lateral and longitudinal state variables is first constructed. The state variables of the model are as follows:

[0157] Lateral position error e_y: represents the lateral deviation of the vehicle's center of mass relative to the reference path, in meters;

[0158] Heading angle error e_psi: indicates the deviation of the vehicle heading angle relative to the tangent direction of the reference path, in rad;

[0159] Center of mass side slip angle beta: represents the deviation between the vehicle speed direction and the heading angle, in rad;

[0160] Longitudinal position s: represents the longitudinal position of the vehicle's center of mass on the reference path, in m;

[0161] Vehicle speed v: represents the total velocity at the center of mass of the vehicle, in m / s.

[0162] The control variables of the model are:

[0163] Front wheel turning angle delta_f: indicates the turning angle of the front wheel of the vehicle relative to the longitudinal axis of the vehicle body, in rad;

[0164] Driving / braking force F_x: represents the resultant force acting on the center of mass of the vehicle. Driving force is positive and braking force is negative. The unit is N.

[0165] Under the Newtonian mechanics and simplified small angle assumptions, a nonlinear dynamic model can be established to describe the differential equation relationship between the state quantity and the control quantity. The specific equation form will not be repeated. However, it should be noted that the dynamic model needs to be reasonably simplified to balance the control accuracy and real-time calculation efficiency. For example, the tire cornering stiffness can be assumed to be a constant and the nonlinear characteristics of the tire can be ignored; the roll and pitch rotation of the suspension can also be assumed to be small angle motions.

[0166] The core of MPC control is to optimize the control quantity at the current moment according to the predicted value of the system state quantity in the future (i.e., the prediction time domain) to minimize a certain objective function in the prediction time domain. Therefore, the selection of the objective function directly determines the performance of the controller. In order to take into account both the lateral control accuracy and longitudinal driving quality of the vehicle, this paper constructs an integrated objective function J that comprehensively considers the lateral error, longitudinal error, and control quantity changes:

[0167] J = (e_y)^T * Q_y * e_y + (e_psi)^T * Q_psi * e_psi + beta^T * Q_beta* beta

[0168] + (e_s)^T * Q_s * e_s + (e_v)^T * Q_v * e_v

[0169] + (delta_f)^T * R_delta * delta_f + (F_x)^T * R_F * F_x;

[0170] Among them, e_y: lateral position error vector, which represents the lateral deviation of the vehicle's center of mass relative to the reference path at each moment in the prediction time domain. For example, e_y = [0.1, 0.2, 0.3] means that the lateral deviations at the next three moments are 0.1m, 0.2m and 0.3m respectively.

[0171] Q_y: The weight matrix of the lateral position error, which is used to adjust the controller's attention to the lateral deviation. Q_y is usually a diagonal matrix. The larger the element in the matrix, the more attention is paid to the lateral error at the corresponding moment. For example, Q_y = diag(100,100,100) means that the lateral error weights at the three prediction moments are the same.

[0172] e_psi: heading angle error vector, which indicates the deviation of the vehicle heading angle relative to the tangent direction of the reference path at each moment in the prediction time domain. For example, e_psi = [0.02, 0.03, 0.01] means that the heading angle deviations at the next three moments are 0.02rad, 0.03rad, and 0.01rad respectively.

[0173] Q_psi: The weight matrix of the heading angle error, which is used to adjust the controller's attention to the heading deviation. For example, Q_psi = diag(500,500,500) means that the heading angle error weights at the three prediction moments are the same, and the weight value is higher than Q_y, indicating that more attention is paid to heading tracking.

[0174] beta: The center of mass slip angle vector, which represents the angle between the vehicle speed direction and the vehicle head direction at each moment in the prediction time domain. For example, beta = [0.005, 0.003, 0.008] means that the slip angles at the next three moments are 0.005rad, 0.003rad, and 0.008rad respectively.

[0175] Q_beta: The weight matrix of the center of mass slip angle, which is used to adjust the controller's attention to the slip motion. For example, Q_beta = diag(1000,1000,1000) means that the slip angle weights at the three prediction moments are the same and the weight is the highest, indicating that the vehicle's driving stability is emphasized.

[0176] e_s: longitudinal position error vector, which indicates the longitudinal deviation of the vehicle's center of mass position relative to the expected position at each moment in the prediction time domain. For example, e_s = [0.5, 1.0, 1.5] means that the longitudinal position deviations at the next three moments are 0.5m, 1.0m, and 1.5m respectively.

[0177] Q_s: The weight matrix of the longitudinal position error, which is used to adjust the controller's attention to the longitudinal position deviation. For example, Q_s = diag(10,10,10) means that the longitudinal position error weights at the three prediction moments are the same, and the weights are small, indicating that the lateral and heading tracking are relatively more concerned.

[0178] e_v: Vehicle speed error vector, which indicates the deviation between the actual vehicle speed and the expected speed at each moment in the prediction time domain. For example, e_v = [1.0, 0.8, 1.2] indicates that the vehicle speed errors at the next three moments are 1.0m / s, 0.8m / s, and 1.2m / s respectively.

[0179] Q_v: The weight matrix of the vehicle speed error, which is used to adjust the controller's attention to the vehicle speed deviation. For example, Q_v = diag(50,50,50) means that the vehicle speed error weights at the three prediction moments are the same.

[0180] delta_f: Front wheel steering angle change vector, indicating the change in the front wheel steering angle at each moment in the prediction time domain relative to the current moment. For example, delta_f = [0.02, 0.01, -0.015] means that the front wheel steering angle will change by 0.02rad, 0.01rad and -0.015rad at the next three moments respectively.

[0181] R_delta: Weight matrix of the front wheel angle change, used to constrain the drastic change of the front wheel angle. The larger the R_delta value, the less allowed the front wheel angle is to change rapidly, thus obtaining a smooth and comfortable steering response. For example, R_delta = diag(5000,5000,5000).

[0182] F_x: driving / braking force change vector, indicating the change in the resultant force at each moment in the prediction time domain relative to the current moment. Driving force is positive and braking force is negative. For example, F_x = [500, -200, 300] means that the vehicle resultant force at the next three moments will change by 500N (driving), 200N (braking) and 300N (driving) respectively.

[0183] R_F: The weight matrix of driving / braking force change, used to constrain the drastic change of driving / braking force. The larger the R_F value, the less rapid changes in the resultant force are allowed, so as to obtain a smooth acceleration and deceleration experience. For example, R_F = diag(0.001,0.001,0.001).

[0184] By adjusting the weight matrix, the emphasis of the tracking control task on lateral performance and longitudinal performance can be flexibly configured. For example, the risk brought by lateral error is greater when driving at high speed, so the weights of Q_y and Q_psi are increased; while when passing through construction areas at low speed, longitudinal speed control is more critical, so the weights of Q_s and Q_v are increased. In addition, the weights of R_delta and R_F affect the smoothness and comfort of control. Experience shows that too low weights of the two will cause drastic changes in the control amount, while too high weights will lead to increased tracking errors. Reasonable values ​​need to be determined in combination with actual vehicle tests.

[0185] For example, assume that the prediction time domain is 10 time steps in the future, and each time step is 0.1s. The current vehicle deviates from the reference path by 0.5m, the speed is 20m / s, and the expected speed is 22m / s. Take Q_y=10, Q_psi=100, Q_beta=50, Q_s=1, Q_v=5, R_delta=500, R_F=0.1, and substitute them into the integrated objective function to calculate the control target cost value that comprehensively considers the lateral and longitudinal performance under the current working conditions. The smaller the value, the more ideal the vehicle's motion state and control effect in the prediction time domain. The MPC controller will minimize the integrated objective function as the optimization goal and solve the optimal control sequence.

[0186] In addition to the physical constraints inherent in the vehicle dynamics model, such as the tire slip angle and center of mass slip angle cannot be too large, the controller must also meet the performance constraints of the actuator, such as the front wheel angular velocity and driving / braking force rate cannot exceed the limit. In addition, the adhesion constraints between the tire and the ground must also be considered under complex working conditions.

[0187] The establishment of tire adhesion constraints requires comprehensive consideration of the tire vertical load F_z, longitudinal force F_x and lateral force F_y. In general, F_z can be approximated as a constant value (small angle assumption), while F_x and F_y satisfy the friction ellipse equation:

[0188] (F_x / u_x)^2 + (F_y / u_y)^2 = 1;

[0189] Among them, u_x and u_y are the peak adhesion coefficients of the tire in the longitudinal and lateral directions, respectively, which are related to the road material and tire condition. To ensure that the vehicle does not skid or slip, the MPC controller needs to limit the (F_x, F_y) of each tire within the friction ellipse during the optimization process.

[0190] Take the icy road as an example, assuming u_x=0.2, u_y=0.25. If the vertical load of a tire is 5000N at this moment, and the longitudinal force has reached 800N (braking), the maximum allowable lateral force can be obtained using the friction ellipse equation:

[0191] F_y_max = u_y * sqrt(1 - (F_x / 5000 / u_x)^2) = 1031 N;

[0192] The MPC controller uses the above maximum lateral force value as a constraint to optimize the front wheel steering angle to ensure that the tire will not slip under complex working conditions. Similarly, other tires are also subject to this constraint.

[0193] The vehicle dynamics model, integrated objective function and constraints are integrated to form a complete MPC problem. Further, the objective function is expanded by Taylor J and the terms above the second order are ignored, and the constraints are linearized to transform the original problem into a quadratic optimization problem.

[0194] Quadratic optimization problems can be solved efficiently using numerical optimization algorithms, such as the interior point method, the effective set method, and sequential quadratic programming. The solution is the optimal control sequence for the lateral and longitudinal motion of the vehicle in the future prediction time domain. However, the MPC controller does not directly execute the entire sequence, but adopts a rolling optimization strategy: each control cycle only executes the first control quantity of the sequence, and the remaining control quantities are used as the initial values ​​for the next cycle optimization. In this way, the prediction error can be corrected in time after the arrival of new measurement values, and it is more adaptable to the changing traffic environment.

[0195] Specifically, assuming that the prediction time domain is 1s and the control cycle is 0.1s, a single planning can obtain 10 optimal control quantities including (delta_f, F_x). The controller only executes (delta_f_k, F_x_k) at time k, that is, the front wheel angle is adjusted to delta_f_k and the driving / braking force is adjusted to F_x_k through the actuator. At time k+1, the controller remeasures the vehicle state, replaces the control quantity originally planned at time k+1 with (delta_f_k, F_x_k) as the initial value, and re-optimizes under the new vehicle state constraints, and repeats this cycle. By updating the controlled object information in real time and rolling optimization, MPC control can adaptively adjust the vehicle motion under the premise of satisfying physical constraints, and finally realize the coordinated tracking control of the vehicle's lateral position, heading and longitudinal speed.

[0196] The advantages of this method are: the lateral and longitudinal movements are coupled through the dynamic model, overcoming the modeling error introduced by the simplified model; the lateral and longitudinal control indicators are uniformly considered through the integrated objective function, overcoming the local optimum caused by artificial trade-offs; the risk of slipping and instability under complex road conditions is overcome through tire adhesion constraints; and the uncertainty of open-loop control is overcome through MPC rolling optimization.

[0197] Figure 2 FIG. 1 is a schematic diagram of a driving training system for a robot vehicle based on assisted driving by an intelligent agent according to an embodiment of the present invention. Figure 2 As shown, the system comprises:

[0198] The first unit is used to obtain multi-sensor fusion perception data of a robot vehicle, wherein the multi-sensor fusion perception data includes vehicle surrounding environment information collected by an on-board laser radar, millimeter-wave radar, ultrasonic radar and panoramic camera, and vehicle self-motion state information collected by an inertial measurement unit and a wheel speed encoder, and the robot vehicle includes a vehicle with an assisted driving function; the multi-sensor fusion perception data is input into a preset deep reinforcement learning model, which is trained by robot vehicle driving data and human driver driving behavior data in a large-scale real road environment, and can output the optimal driving decision for the current scene in real time;

[0199] The second unit is used to generate driving guidance subtasks in three dimensions, namely, lateral control driving guidance, longitudinal control driving guidance, and traffic rule driving guidance, based on the driving decision output by the deep reinforcement learning model and in combination with a preset safe driving rule library; for each driving guidance subtask, an independent hierarchical finite state machine model is constructed, each hierarchical finite state machine model includes multiple state nodes and state transition conditions, the state nodes correspond to specific driving actions, and the state transition conditions include vehicle state conditions and environmental state conditions;

[0200] The third unit is used to perceive the execution status and environment of the robot vehicle in real time, input the perception results into the hierarchical finite state machine model, thereby triggering the corresponding state transfer conditions, and output the corresponding driving guidance action sequence according to the current state node until the corresponding driving guidance subtask is completed; the driving guidance action sequence is converted into specific vehicle control instructions, including throttle, brake, and steering control signals, which are sent to the vehicle actuator through the vehicle controller to realize real-time driving guidance and control of the robot vehicle.

[0201] According to a third aspect of the embodiments of the present invention,

[0202] An electronic device is provided, comprising:

[0203] processor;

[0204] a memory for storing processor-executable instructions;

[0205] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0206] A fourth aspect of the embodiments of the present invention is:

[0207] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0208] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0209] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A driving training method for a robot vehicle based on assisted driving by an intelligent agent, characterized in that: include: Acquire multi-sensor fusion perception data of a robot vehicle, wherein the multi-sensor fusion perception data includes vehicle surrounding environment information collected by an on-board laser radar, millimeter-wave radar, ultrasonic radar, and panoramic camera, and vehicle self-motion state information collected by an inertial measurement unit and a wheel speed encoder, and the robot vehicle includes a vehicle with an assisted driving function; input the multi-sensor fusion perception data into a preset deep reinforcement learning model, wherein the deep reinforcement learning model is trained by robot vehicle driving data and human driver driving behavior data in a large-scale real road environment, and can output the optimal driving decision for the current scene in real time; According to the driving decision output by the deep reinforcement learning model, combined with the preset safe driving rule library, three-dimensional driving guidance subtasks of lateral control driving guidance, longitudinal control driving guidance and traffic rule driving guidance are generated; for each driving guidance subtask, an independent hierarchical finite state machine model is constructed, each hierarchical finite state machine model includes multiple state nodes and state transition conditions, the state nodes correspond to specific driving actions, and the state transition conditions include vehicle state conditions and environmental state conditions; Real-time perception of the execution state and environment of the robot vehicle, inputting the perception results into the hierarchical finite state machine model, thereby triggering the corresponding state transition conditions, and outputting the corresponding driving guidance action sequence according to the current state node until the corresponding driving guidance subtask is completed; converting the driving guidance action sequence into specific vehicle control instructions, including throttle, brake and steering control signals, and sending them to the vehicle actuator through the vehicle controller to realize real-time driving guidance and control of the robot vehicle; The multi-sensor fusion perception data is input into a preset deep reinforcement learning model, which is trained by robot vehicle driving data and human driver driving behavior data in a large-scale real road environment, and can output the optimal driving decision for the current scene in real time, including: The deep reinforcement learning model consisting of an action network and a judgment network and built based on a deep deterministic policy gradient is trained using a pre-acquired training data set. The action network parameters are optimized by minimizing the mean square error between the output actions of the action network and the actions of human drivers. At the same time, the instant reward is calculated based on the vehicle state change and the predefined reward function. The judgment network parameters are optimized by minimizing the mean square error between the estimated cumulative reward of the judgment network and the actual cumulative reward. The driving decision samples of human drivers in complex scenarios are introduced as additional supervisory signals into the training of deep reinforcement learning models. The cross entropy loss between the output actions of the action network and the driving decision samples is minimized, and the behavior cloning method is used to accelerate the convergence of deep reinforcement learning model training. The action network is used to output the action values ​​of multiple alternative driving actions according to the training data set, and the judgment network is used to determine the long-term cumulative reward value according to the training data set and the action value generated by the action network. The alternative driving actions generated by the action network are input into the pre-built state transition model to predict the vehicle state change and environmental feedback after executing each alternative driving action, and quantify them into corresponding immediate reward values; The long-term cumulative reward and the immediate reward output by the judgment network are weighted and summed to obtain the comprehensive score of each alternative driving action. The alternative driving action with the highest comprehensive score is selected and output as the optimal driving decision in the current scenario. The optimal driving decision is then checked for safety in combination with the preset safe driving rule library, and driving decisions that do not meet safety constraints are pruned and replaced. Finally, a safe and executable optimal driving decision for the current scenario is output; For each driving guidance subtask, an independent hierarchical finite state machine model is constructed. Each hierarchical finite state machine model contains multiple state nodes and state transition conditions. The state nodes correspond to specific driving actions. The state transition conditions include vehicle state conditions and environmental state conditions, including: The hierarchical finite state machine model includes a three-layer structure, which is a task layer state machine, a decision layer state machine and an execution layer state machine from top to bottom; The task layer state machine divides different driving stages according to the semantic information of the driving task. Each driving stage corresponds to a task layer state node. The transition conditions between adjacent task layer state nodes depend on the vehicle state parameters and environmental state parameters perceived in real time. When a specific state transition condition is met, the decision layer state machine is triggered to perform corresponding driving subtask planning. The decision-layer state machine receives the driving phase information output by the task-layer state machine, performs behavior decision-making and path planning in combination with the vehicle state and the environment state, and outputs a series of local path points and corresponding target speeds and accelerations as well as steering angle motion control parameters to be executed. Each decision-layer state node corresponds to a decision-making execution action in a specific scenario. The execution layer state machine is responsible for converting the local path and motion control parameters output by the decision layer state machine into low-level control instructions for throttle, brake and steering, achieving smooth and accurate motion control with reference to the vehicle dynamics model and controller characteristics, and adaptively adjusting the controller parameters according to the feedback information from the vehicle chassis sensor, where each execution layer state node corresponds to a basic control operation.

2. The method according to claim 1, characterized in that The action network is used to output the action values ​​of multiple alternative driving actions according to the training data set. The judgment network is used to determine the long-term cumulative reward value according to the training data set and the action value generated by the action network. The alternative driving actions generated by the action network are input into the pre-built state transition model to predict the vehicle state changes and environmental feedback after executing each alternative driving action, and quantify them into corresponding immediate reward values ​​including: Determining the long-term cumulative reward value includes: ； in, G t represents the long-term cumulative reward starting from time t, r t Indicates execution of action a t After Moment t The immediate reward value, represents the discount factor used to attenuate the impact of future rewards, represents the instant reward value at time t+k, k Indicates the count value; The alternative driving actions generated by the action network are input into the pre-built state transition model to predict the vehicle state changes and environmental feedback after executing each alternative driving action, and quantify them into corresponding immediate reward values ​​including: ； ； Among them, r t Indicates execution of action a t After Moment t The immediate reward value, s t represents the state input at time t, a t represents the driving action generated at time t, f(s t ,a t ) represents the state transition function of a state-action pair, represents an action network with parameters θ.

3. The method according to claim 1, characterized in that The driving guidance action sequence is converted into specific vehicle control instructions, including throttle, brake and steering control signals, which are sent to the vehicle actuator through the vehicle controller to achieve real-time driving guidance and control of the robot vehicle, including: Parsing the driving instruction action sequence to extract semantic information of each driving instruction action sequence, wherein the semantic information includes an action type, a target value, and a duration; Generate corresponding vehicle control instructions according to the semantic information of each driving guidance action sequence, wherein the vehicle control instructions include throttle control instructions, brake control instructions and steering control instructions, wherein the throttle control instructions and brake control instructions are calculated by a preset control algorithm according to the current speed and expected speed of the vehicle, and the steering control instructions are calculated by a preset control algorithm according to the current position, heading and expected trajectory of the vehicle; Smoothing the generated vehicle control instructions, using a preset smoothing algorithm to filter the throttle control instructions, brake control instructions and steering control instructions, so as to remove high-frequency disturbances and mutations in the vehicle control instructions and obtain smooth vehicle control instructions; The smooth vehicle control instructions are sent to the vehicle actuator through the vehicle controller, which controls the vehicle actuator to output the corresponding physical control quantity, thereby realizing real-time control of the vehicle.

4. The method according to claim 1, characterized in that: The method further comprises: Constructing a dynamic model describing the lateral and longitudinal coupled motion characteristics of the vehicle, wherein the dynamic model includes a lateral position error, a heading angle error, and a sideslip angle of the center of mass representing the lateral motion state of the vehicle, a longitudinal position and a vehicle speed representing the longitudinal motion state of the vehicle, and a front wheel steering angle and a driving / braking force as control variables; Constructing an integrated objective function that considers both lateral control performance and longitudinal control performance, the integrated objective function includes lateral position error, heading angle error, center of mass sideslip angle, longitudinal position error and vehicle speed error, as well as front wheel steering angle change and driving / braking force change, and weighting each error and control amount change by setting a weight matrix; On the basis of vehicle physical constraints and controller performance constraints, tire adhesion constraints are introduced to establish joint constraints. The tire adhesion constraints restrict the tire from side slipping and sliding under complex working conditions through the relationship between tire vertical load, longitudinal force and lateral force. The dynamic model, the integrated objective function and the joint constraint conditions are integrated into a quadratic optimization problem, and a numerical optimization algorithm is used to solve the quadratic optimization problem to obtain an optimal control sequence of the front wheel steering angle and the driving / braking force; the solved optimal control sequence is updated in real time according to the rolling optimization strategy of the model predictive control, the first control quantity is executed in each control cycle, and the remaining control quantities are used as initial values ​​for optimization in the next cycle, so as to realize coordinated tracking control of the vehicle's lateral and longitudinal motions.

5. A driving training system for a robot vehicle based on assisted driving by an intelligent agent, used to implement the method described in any one of claims 1 to 4, characterized in that: include: The first unit is used to obtain multi-sensor fusion perception data of a robot vehicle, wherein the multi-sensor fusion perception data includes vehicle surrounding environment information collected by an on-board laser radar, millimeter-wave radar, ultrasonic radar and panoramic camera, and vehicle self-motion state information collected by an inertial measurement unit and a wheel speed encoder, and the robot vehicle includes a vehicle with an assisted driving function; the multi-sensor fusion perception data is input into a preset deep reinforcement learning model, which is trained by robot vehicle driving data and human driver driving behavior data in a large-scale real road environment, and can output the optimal driving decision for the current scene in real time; The second unit is used to generate driving guidance subtasks in three dimensions, namely, lateral control driving guidance, longitudinal control driving guidance, and traffic rule driving guidance, based on the driving decision output by the deep reinforcement learning model and in combination with a preset safe driving rule library; for each driving guidance subtask, an independent hierarchical finite state machine model is constructed, each hierarchical finite state machine model includes multiple state nodes and state transition conditions, the state nodes correspond to specific driving actions, and the state transition conditions include vehicle state conditions and environmental state conditions; The third unit is used to perceive the execution state and environment of the robot vehicle in real time, input the perception results into the hierarchical finite state machine model, thereby triggering the corresponding state transfer conditions, and output the corresponding driving guidance action sequence according to the current state node until the corresponding driving guidance subtask is completed; the driving guidance action sequence is converted into specific vehicle control instructions, including throttle, brake and steering control signals, which are sent to the vehicle actuator through the vehicle controller to realize real-time driving guidance and control of the robot vehicle.

6. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method described in any one of claims 1 to 4.

7. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images