An intelligent vehicle integrated decision system based on human-machine co-driving concept

Abstract

本发明公开了一种基于人机共驾理念的智能车辆集成式决策系统，包括：外界环境感知模块，用于将外界传感信息进行处理，输出感知目标结果Senv；座舱感知模块，用于将座舱内感知信息进行处理，输出感知结果Scab；决策模块，用于将混合感知状态S＝[Senv,Scab]作为输入，并根据深度强化学习决策算法输出决策动作A＝[aenv,acab]，决策动作A其中一路输出为针对车辆的行驶行为决策指令aenv，另一路输出为座舱控制指令acab；行驶行为控制模块，根据aenv对车辆的行驶状态进行适应性调整；舱内控制模块，根据acab对舱内执行设备进行适应性调整。本发明利用舱内信息指导车辆的道路行驶决策，提升系统的鲁棒性和安全性，同时利用外界行驶环境信息提升舱内乘客用户体验。

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving of automobiles, and in particular to an intelligent vehicle integrated decision system based on a man-machine co-driving concept. BACKGROUND

[0002] With the improvement of the automatic driving level of automobiles, the requirements of functional scenarios on the autonomous decision-making ability of vehicles in complex and variable scenarios are also increasing. The proportion of traditional decision-making methods based on rules and artificial design will gradually decrease. The academic field has proposed many AI algorithms based on data driving to build decision modules, but such end-to-end methods have a large demand for data volume and have the characteristics of being uninterpretable and unpredictable. For example, patent publication CN113920484A discloses an end-to-end automatic driving decision method based on monocular RGB_D features and reinforcement learning. This patent makes end-to-end automatic driving decisions based on camera features and reinforcement learning, and directly outputs vehicle control action signals such as brakes, throttles, and steering. This method first considers only the information of a monocular camera as input, and directly outputs control instructions in an end-to-end manner, which is risky and has poor practicality. For another example, patent publication CN112348201A discloses an intelligent decision implementation method for automatic driving of a group of vehicles based on federated deep reinforcement learning. This patent introduces more sensors and environmental information as inputs to the model, but does not consider adaptation to man-machine co-driving scenarios and does not have optimal game decision-making ability between multiple vehicles in a global sense. Patent publication CN113602284A discloses a man-machine co-driving mode decision method, device, equipment, and storage medium, which collects the physiological characteristics of the driver through a watch to provide a basis for mode downgrading and switching of vehicle functions, but does not involve behavior decision-making of vehicles in complex scenarios. Therefore, there is an urgent need to develop a more comprehensive intelligent vehicle integrated decision system and decision method based on the man-machine co-driving concept to better make decisions. SUMMARY

[0003] The present application aims to provide an intelligent vehicle integrated decision system based on the man-machine co-driving concept. The present application realizes the use of cabin information to guide the road driving decision of the vehicle in the L2-L3 level automatic driving man-machine co-driving scenario, improves the robustness and safety of the system, and uses external driving environment information to improve the user experience of passengers in the cabin.

[0004] The technical solution of the present application is an intelligent vehicle integrated decision system based on the man-machine co-driving concept, which comprises:

[0005] An external environment perception module for processing current vehicle external environment sensing information and outputting a perception target result S env ;

[0006] a cockpit perception module for processing information of the cockpit internal perception system and outputting a perception result S cab ;

[0007] a decision module for taking the mixed perception state S = [S env , S cab ] as input and outputting a deep fusion decision action A = [a env , a cab ] according to a deep reinforcement learning decision algorithm, the decision action A being output in two paths, one of which is a driving behavior decision instruction a env for the vehicle and the other is a cockpit control instruction a cab ;

[0008] a driving behavior control module for adaptively adjusting the driving state of the vehicle according to the driving behavior decision instruction a env ;

[0009] a cockpit control module for adaptively adjusting the cockpit execution equipment according to the cockpit control instruction a cab .

[0010] In the foregoing intelligent vehicle integrated decision system based on the man-machine co-driving concept, the decision system further comprises a planning module, the planning module being configured to process the driving behavior decision instruction a env , plan a trajectory point according to a env , and then output the trajectory point to the driving behavior control module to control the driving state of the vehicle.

[0011] In the foregoing intelligent vehicle integrated decision system based on the man-machine co-driving concept, the decision system further comprises a positioning module, the positioning module being configured to output the positioning information of the vehicle to the decision module, the planning module, and the driving behavior control module, respectively.

[0012] In the foregoing intelligent vehicle integrated decision system based on the man-machine co-driving concept, the framework of the deep reinforcement learning decision algorithm in the decision module includes but is not limited to one of DQN, DDPG, A3C, PPO, and SAC reinforcement learning frameworks.

[0013] In the foregoing intelligent vehicle integrated decision system based on the man-machine co-driving concept, the use process of the deep reinforcement learning decision algorithm in the decision module includes: inputting the mixed perception state S into a neural network N with current network parameters w, thereby outputting a decision value V, and determining a final output decision action A according to the decision value V.

[0014] In the aforementioned intelligent vehicle integrated decision-making system based on the concept of human-machine co-driving, the decision value V = [driving decision v1, cabin decision (air conditioning v2, lighting v3, music v4, windows v5)] includes 5 types of decision objects from v1 to v5. In each type of decision object, the decision corresponding to the largest value is selected as the final decision action output A = [a1, a2, a3, a4, a5] according to the greedy strategy.

[0015] In the aforementioned intelligent vehicle integrated decision-making system based on the human-machine co-driving concept, the training process of the deep reinforcement learning decision-making algorithm in the decision-making module includes the following steps:

[0016] Step 1: Initialize the exploration threshold E and initialize the network parameters w of the neural network N;

[0017] Step 2: Set the current vehicle's hybrid perception state S = [S env S cab The input is fed into the neural network N, generating a random number e.

[0018] If e≥E, then the decision corresponding to the maximum value is selected as the final output decision action A according to the greedy strategy;

[0019] If e < E, then randomly generate decision action A;

[0020] Step 3: Execute decision action A on the vehicle currently in mixed perception state S, thereby obtaining a new mixed perception state S' and reward R. Use the new mixed perception state S' to determine:

[0021] If a collision occurs, a termination flag (Flag = 1) is generated.

[0022] If no collision occurs, then Flag = 0;

[0023] Step 4: Store the set of data {S, A, S', R, Flag} into the data pool;

[0024] Step 5: Assign S' to S, the vehicle enters a new hybrid perception state S', and at the same time assign E*0.99999 to E, then proceed to step 6.

[0025] Step 6: Collect n sets of sample data {S} from the data pool. n A n S n ', R n Flag n For each set of sample data, calculate the current decision objective value V. target :

[0026] If Flag = 1, then V target =R;

[0027] If Flag = 0, then V target =R+γmax N(S′, w), where γmax N(S′, w) means that the new hybrid sensing state S' is input into the neural network N with the current network parameters w, the maximum decision value is output according to the greedy policy, and multiplied by a fixed discount value γ.

[0028] Step 7: Calculate the n decision objective values ​​V using the mean square error function. target Losses:

[0029] Step 8: Use backpropagation of neural network gradients to update the network parameters w of neural network N, return to step 2, and perform iterative training.

[0030] In the aforementioned intelligent vehicle integrated decision-making system based on the human-machine co-driving concept, the reward R function in step 3 is: R = A * safety + B * efficiency + C * comfort + D * passenger experience, where A, B, C, and D represent the weights of each evaluation indicator, safety is calculated by collision penalty, efficiency is measured by vehicle speed, comfort is derived from lane change penalty and acceleration change, and passenger experience is provided by the in-cabin perception system.

[0031] In the aforementioned intelligent vehicle integrated decision-making system based on the concept of human-machine co-driving, the fixed discount value γ = 0.999 in step 6.

[0032] Compared with the prior art, the beneficial effects of the present invention are reflected in:

[0033] I. This invention improves upon existing end-to-end autonomous driving decision-making schemes based on reinforcement learning, which suffer from problems such as large data transfer gap from simulation to reality, difficulty in training convergence, and lack of interpretability. It also differs from methods that are entirely based on manually designed rules. The decision-making system proposed in this invention has a certain ability to autonomously cope with complex scenarios.

[0034] Second, this invention utilizes a deep reinforcement learning framework to integrate L2-L3 level autonomous driving systems with cockpit systems into a human-machine co-driving system, creating a vehicle-level global decision-making function that includes both the internal and external aspects of the vehicle, thereby improving safety and the user experience of passengers in the cockpit. In particular, it enables the use of in-cabin information to guide the vehicle's road driving decisions, improving the robustness and safety of the system, while also using external driving environment information to enhance the user experience of passengers in the cockpit.

[0035] Third, the present invention trains state features through a specially designed deep reinforcement learning framework, which improves the generalization and consistency of the algorithm, narrows the gap between AI model training in simulation and deployment in real vehicles, and enables a large amount of low-cost data from the simulation environment to be well applied to real vehicles. At the same time, the present invention combines federated learning and reinforcement learning to provide the system with more valuable data and improve the efficiency of technology iteration. Attached Figure Description

[0036] Figure 1 This is a schematic diagram of the system framework of the present invention combined with the decision-making process;

[0037] Figure 2 This is one vehicle driving scenario in the embodiment;

[0038] Figure 3 This is a network model diagram of a deep reinforcement learning decision-making algorithm. Detailed Implementation

[0039] The present invention will be further described below with reference to the accompanying drawings and embodiments, but this should not be construed as limiting the present invention.

[0040] Example: An integrated decision-making system for intelligent vehicles based on the concept of human-machine co-driving, system structure referenced Figure 1 As shown, it includes an external environment perception module, a cockpit perception module, a decision-making module, a driving behavior control module, an in-cabin control module, a planning module, and a positioning module, among which:

[0041] The external environment perception module is used to process the current vehicle's external environment sensing information and output the perception target result S. env .

[0042] The cockpit perception module is used to process information from the in-cockpit perception system and output the perception results S. cab .

[0043] That is, the state variable of reinforcement learning in this invention is: the perceived target result S obtained after the input from the external environment sensor is processed by the perception module. env And the perception results of the in-cabin perception system (including the status of passengers, the driver, etc.) cab splicing [S env S cab ].

[0044] To achieve the above data collection, the human-machine co-driving system includes sensors that are already integrated into existing autonomous driving systems, such as lidar, millimeter-wave radar, cameras, and ultrasonic radar. It also covers in-vehicle cabin perception systems such as DMS and OMS to fully collect road information outside the vehicle and the current status of the driver and passengers inside the vehicle.

[0045] The decision module is used to process the hybrid sensing state S = [S env S cab As input, and based on the deep reinforcement learning decision algorithm, output a deeply fused decision action A = [a] env a cab The decision action A is output in two ways, one of which is the decision instruction a for the vehicle's driving behavior. env Actions such as lane changing, going straight, turning left, turning right, and stopping are further interpreted and executed by the downstream planning module and driving behavior control module; the other output is the cockpit control command a. cab Such as the power of the air conditioner, the light switch, and the playback of various types of music.

[0046] For example, if heavy traffic or bad weather is detected, the ambient lights can be changed to red, and the behavioral decisions output will be more conservative, avoiding overtaking and lane changes as much as possible; if the driver is detected to be drowsy, invigorating music can be played in the cabin, and the vehicle speed can be reduced.

[0047] This invention features a special design for the state variables in reinforcement learning. Instead of using the raw sensor input or the feature vector after feature extraction as input, it retains the upstream sensing module (see...). Figure 1 The system uses the target result output by the perception algorithm as part of the state variable, and simultaneously accesses the state information of the driver and passengers in the cockpit to make comprehensive and targeted decisions.

[0048] The deep reinforcement learning decision-making algorithm in the decision-making module adopts a framework including but not limited to one of the following reinforcement learning frameworks: DQN, DDPG, A3C, PPO, and SAC.

[0049] The driving behavior control module can make driving behavior decision commands based on the instructions a. env Adaptive adjustments are made to the vehicle's driving status.

[0050] The in-cabin control module can respond to cockpit control commands a cab Adaptive adjustments were made to the equipment used in the cabin.

[0051] The planning module is used to process driving behavior decision instructions. env According to a env The trajectory points are planned and then output to the driving behavior control module to control the vehicle's driving status.

[0052] The positioning module is used to output the vehicle's positioning information to the decision-making module, planning module, and driving behavior control module, respectively.

[0053] When the reinforcement learning decision algorithm model is trained in a simulation environment, the same mixed-sensor state input [S] is used. env S cab This means that the target data processed by the perception module is used as input, avoiding deviations caused by differences between the simulation and the original real-world environment. The reward function is designed as follows: R = A * Safety + B * Efficiency + C * Comfort + D * Passenger Experience, where A, B, C, and D represent the weights of each evaluation indicator. Safety is calculated by collision penalties, efficiency is measured by vehicle speed, comfort is derived from lane-changing penalties, and passenger experience is provided by the in-cabin perception system.

[0054] Whenever the decision system is based on the current hybrid sensing state S = [S env S cab ], take decision action A = [a env a cab When a reward R is received and a hybrid perception state S' is entered into the next decision cycle, a set of experiences [S, A, R, S'] is collected. All intelligent vehicles that have deployed the decision-making system of this invention periodically upload their own experiences to a cloud-based experience pool for sharing.

[0055] Each intelligent vehicle can periodically upload its reinforcement learning decision-making algorithm model to a cloud server, train and iterate the model through horizontal federated learning, and then download it locally for model replacement and updates. This maximizes the use of empirical data without violating laws and regulations.

[0056] The following example demonstrates the usage and training process of the deep reinforcement learning decision algorithm in the decision system of this invention.

[0057] The process of using decision-making algorithms:

[0058] The decision-making algorithm framework proposed in this invention circumvents the gap between simulation training and actual deployment by using specially designed state variables (hybrid sensing state S) as input: that is, it no longer requires raw sensor signals as input to the algorithm (because simulation environments are difficult to completely reproduce real-world scenarios, resulting in significant deviations between the signals output by simulation sensors and reality, making it difficult to apply AI algorithm models trained based on sensor signals in simulations in practice). The decision-making algorithm in this invention selects the target and results processed by the sensing module as input.

[0059] For example, when car A is driving on the highway, refer to Figure 2 If the sensor can detect surrounding vehicles B, C, D, and E, then the external perception result S is the input to the decision algorithm. envThe data would be [relative position of car B, speed of car B, acceleration of car B, orientation of car B, relative position of car C, speed of car C, acceleration of car C, orientation of car C, relative position of car D, speed of car D, acceleration of car D, orientation of car D, relative position of car E, speed of car E, acceleration of car E, orientation of car E]. If, at this moment, it is detected that the driver inside the cockpit is experiencing low attention due to drowsiness, feels cold, and is in a bad mood, then the perceived result S inside the cockpit at this time... cab =[driver attention level, perceived temperature, mood index], the input to the decision-making algorithm will be S = [S env S cab ].

[0060] The network model for the decision-making algorithm can be adopted as follows: Figure 3 The fully connected network structure shown has a number of input nodes and a hybrid sensing state quantity [S]. env S cab The matrix dimensions are related; for example, there are 19 input nodes in this case (for better display of the attached diagram, ...). Figure 3 This only illustrates the working principle of the network model and does not show all 19 input nodes, which correspond to [Relative position of car B, speed of car B, acceleration of car B, orientation of car B, relative position of car C, speed of car C, acceleration of car C, orientation of car C, relative position of car D, speed of car D, acceleration of car D, orientation of car D, relative position of car E, speed of car E, acceleration of car E, orientation of car E, driver attention level, perceived temperature, mood index]. The output decision value V = [Driving decision v1, cabin decision (air conditioning v2, lighting v3, music v4, windows v5)], details:

[0061]

[0062] For v1, since it is a high-speed scenario, there is no need to consider turning decision behavior. However, in urban scenarios, decisions such as turning and U-turns can be added.

[0063]

[0064] For v2, options such as increasing or decreasing power can also be added.

[0065]

[0066] For v3, other colors and ambient lighting modes can also be added.

[0067]

[0068] For v4, drivers can also add their preferred music presets.

[0069]

[0070] In this example, the output decision value V will have a total of 18 values, covering 5 decision objects from v1 to v5. Finally, within each decision object category, according to a greedy strategy, only the decision with the largest v value is selected as the final decision action, outputting A = [a1, a2, a3, a4, a5]. Assuming the mixed-sensor state input in this example:

[0071]

[0072]

[0073]

[0074]

[0075]

[0076] Based on the input mixed-sensing state quantity, the complete and reasonable decision action output value should be A = [decelerate, increase air conditioning temperature, switch ambient light to yellow, turn on refreshing music, close windows]. The downstream planning module and driving behavior control module execute the corresponding actions according to the deceleration decision command, and the actuators in the cabin (audio, air conditioning, lights, windows, etc.) are also controlled according to the corresponding decision commands.

[0077] Training process of decision-making algorithm:

[0078] The above has introduced the process of using deep reinforcement learning decision-making algorithms, including: inputting the mixed-sensor state S into the neural network N with the current network parameter w, thereby outputting the decision value V, i.e., V = N(S, w), and determining the final output decision action A based on the decision value V.

[0079] The training process based on the above algorithm includes the following steps:

[0080] Step 1: Initialize the exploration threshold E = 0.8 (can be adjusted freely), and initialize the network parameters w of the neural network N.

[0081] Step 2: Set the current vehicle's hybrid perception state S = [S env S cab (Data can come from a simulation environment or be collected during actual driving) is input into the neural network N to generate a random number e.

[0082] Assuming e = 0.9, then e ≥ E, and the decision-making behavior follows the previous greedy strategy, directly choosing the decision action A with the largest v value;

[0083] If e < E, then a decision action A is generated randomly.

[0084] Step 3: Execute decision action A on the vehicle currently in a mixed perception state S, thereby obtaining a new mixed perception state S' and a reward R. The reward function is designed as follows: R = A * Safety + B * Efficiency + C * Comfort + D * Passenger Experience, where A, B, C, and D represent the weights of each evaluation indicator. Safety is calculated by collision penalty (e.g., safety = -100 if a collision occurs), efficiency is measured by vehicle speed (e.g., efficiency = current speed), comfort is derived from lane-changing penalty and acceleration changes (e.g., comfort = -10 * number of lane-changing actions - 10 * acceleration change), and passenger experience is provided by the in-cabin perception system (e.g., passenger experience = passenger mood index * 5). The new mixed perception state S' and reward R are used to determine:

[0085] If a collision occurs, the termination flag F1ag = 1 is obtained;

[0086] If no collision occurs, then F1ag = 0.

[0087] Step 4: Store the set of data {S, A, S', R, Flag} into the data pool.

[0088] Step 5: Assign S' to S, the vehicle enters a new hybrid perception state S', and at the same time assign E*0.99999 to E (to gradually reduce the exploration threshold E), then proceed to step 6.

[0089] Step 6: Collect n sets of sample data {S} from the data pool. n A n S n ′, R n Flag n For each set of sample data, calculate the current decision objective value V. target :

[0090] If Flag = 1, then V target =R;

[0091] If Flag = 0, then V target =R+γmax N(S′, w), where γmax N(S′, w) means that the new mixed sensing state S' is input into the neural network N with the current network parameter w, the maximum decision value is output according to the greedy policy, and multiplied by a fixed discount value γ, which is generally a decimal close to 1. In this example, γ = 0.999.

[0092] Step 7: Calculate the n decision objective values ​​V using the mean square error function. target Losses:

[0093] Step 8: Use backpropagation of neural network gradients to update the network parameters w of neural network N, return to step 2, and perform iterative training.

[0094] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. An integrated decision-making system for intelligent vehicles based on the concept of human-machine co-driving, characterized in that: include: The external environment perception module processes the sensor information of the current external environment of the vehicle and outputs the perception target result. ; The cockpit perception module processes information from the in-cabin perception system and outputs the perception results. ; The decision module is used to process the hybrid sensing state S= And outputs a deeply fused decision action A based on the deep reinforcement learning decision algorithm. Decision action A is output in two ways, one of which is a decision command for the vehicle's driving behavior. The other output is the cockpit control command. ; The driving behavior control module, based on driving behavior decision instructions Adaptive adjustments are made to the vehicle's driving status; The in-cabin control module, based on cockpit control commands... Adaptive adjustments were made to the in-cabin operating equipment; The process of using the deep reinforcement learning decision algorithm in the decision module includes: inputting the mixed sensing state S into the neural network N with the current network parameters w, thereby outputting the decision value V, and determining the final output decision action A based on the decision value V; The decision value V = [driving decision v1, cabin decision (air conditioning v2, lighting v3, music v4, windows v5)] includes 5 types of decision objects from v1 to v5. In each type of decision object, the decision corresponding to the largest value is selected as the final decision action output A = [a1, a2, a3, a4, a5] according to the greedy strategy. The training process of the deep reinforcement learning decision-making algorithm in the decision-making module includes the following steps: Step 1: Initialize the exploration threshold E and initialize the network parameters w of the neural network N; Step 2: Set the current vehicle's hybrid perception state S= The input is fed into the neural network N, which generates a random number e. If e≥E, then the decision corresponding to the maximum value is selected as the final output decision action A according to the greedy strategy; If e < E, then randomly generate decision action A; Step 3: Execute decision action A on the vehicle currently in mixed perception state S, thereby obtaining a new mixed perception state S' and reward R. Use the new mixed perception state S' to determine: If a collision occurs, a termination flag (Flag=1) is generated. If no collision occurs, then Flag=0; The function for reward R in step 3 is: R = A Security +B Efficiency +C Comfort +D Passenger experience is defined by A, B, C, and D, which represent the weights of various evaluation indicators. Safety is calculated by collision penalties, efficiency is measured by vehicle speed, and comfort is derived from lane change penalties and acceleration changes. Passenger experience is provided by the in-cabin perception system. Step 4: Store the set of data {S, A, S', R, Flag} into the data pool; Step 5: Assign S' to S, the vehicle enters a new hybrid perception state S', and simultaneously set E... Assign 0.99999 to E, then proceed to step 6; Step 6: Collect n sets of sample data from the data pool. For each set of sample data, calculate the current decision target value. : If Flag=1, then ; If Flag=0, then ,in This means that the new hybrid sensing state S' is input into the neural network N with current network parameters w, and the maximum decision value is output according to the greedy policy, multiplied by a fixed discount value. ; Step 7: Calculate the n decision objective values ​​using the mean square error function. Losses: ； Step 8: Use backpropagation of neural network gradients to update the network parameters w of neural network N, return to step 2, and perform iterative training.

2. The intelligent vehicle integrated decision-making system based on the human-machine co-driving concept according to claim 1, characterized in that: The decision-making system also includes a planning module, which is used to process driving behavior decision instructions. ,according to The trajectory points are planned and then output to the driving behavior control module to control the vehicle's driving status.

3. The intelligent vehicle integrated decision-making system based on the human-machine co-driving concept according to claim 2, characterized in that: The decision-making system also includes a positioning module, which outputs the vehicle's positioning information to the decision-making module, the planning module, and the driving behavior control module, respectively.

4. The intelligent vehicle integrated decision-making system based on the human-machine co-driving concept according to claim 1, characterized in that: The deep reinforcement learning decision-making algorithm in the decision-making module adopts a framework including but not limited to one of the following reinforcement learning frameworks: DQN, DDPG, A3C, PPO, and SAC.

5. The intelligent vehicle integrated decision-making system based on the human-machine co-driving concept according to claim 1, characterized in that: The fixed discount value in step 6 =0.999.

Images