Human-in-the-loop autonomous driving training and testing method and system based on digital twinning

Abstract

The application discloses a human-in-the-loop automatic driving training and testing method and system based on digital twinning, and the method comprises the following steps: obtaining driving data of a human-in-the-loop driver based on digital twinning, inputting the driving data into a pre-constructed human-in-the-loop driver social value orientation identification model, outputting a weight representing the preference selection of the human-in-the-loop driver between safety and efficiency, and converting the weight into an angle for evaluating the social value orientation of the human-in-the-loop driver; regarding a self vehicle of the human-in-the-loop driver as a leader, pre-constructing a virtual environment vehicle as a follower, and judging whether the environment vehicle and the self vehicle have a game according to the distance between the vehicles; when the game occurs, outputting a behavior strategy of the environment vehicle according to a pre-constructed game model; planning a trajectory of the environment vehicle according to an optimal solution, and outputting a control amount of the environment vehicle. The application can realize real-time interaction between an automatic driving vehicle and a simulation driving environment, and efficiently realize human-in-the-loop automatic driving training and testing based on digital twinning.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving, in particular to a human-in-the-loop automatic driving training and testing method and system based on digital twinning. BACKGROUND

[0002] I. About inverse reinforcement learning:

[0003] In reinforcement learning, Markov Decision Process (MDP) is usually chosen as the theoretical framework under random conditions. In order to obtain the optimal strategy, the agent will make decisions according to the current state to achieve the purpose of maximizing the reward.

[0004] Define a Markov Decision Process as: , where represents a finite state set, denotes the action space, is the state transition probability from the current state to the next state, is the reward function based on the state action space, represents the decay factor. In a complete Markov decision process, the strategy is the probability of taking action for a state , which is only related to the current state and has nothing to do with historical information:

[0005]

[0006] At the same time, introduce the state value function and the action value function , and expect that for any MDP, there is always an optimal strategy that can obtain the corresponding optimal state value function and optimal action value function, and the corresponding Bellman optimal equation is:

[0007]

[0008] In inverse reinforcement learning (IRL), it is expected to use the idea of IRL to learn a reward function that can guide the automatic driving car to make reasonable planning from the decision-making behavior of the driver. Given a set of expert demonstrations , where each expert demonstration trajectory is . Therefore, it can be assumed that the expert tries to optimize an unknown reward function, and the goal of inverse reinforcement learning IRL is to find the optimal strategy so that it is as good as the expert demonstration. Assuming that the optimal strategy is Then we have

[0009]

[0010] At this time, according to the Bellman equation:

[0011]

[0012] To obtain the reward function , the equation is rewritten as:

[0013]

[0014] Therefore, inverse reinforcement learning has the ability to learn and extract implicit behavior rules from expert example behaviors, and is suitable for complex traffic scenarios.

[0015] II. On the basis of game theory:

[0016] Game theory, also known as game theory, is a theory about the interaction of strategies, and is suitable for solving the decision-making problem of multi-agent system. The existing classical game model can be classified in different ways from different angles, and can be divided into three categories according to the model elements: game participants, cooperative relationship and game strategy space.

[0017] According to the number of game participants (N), the existing game model can be divided into self-game, two-person game and multi-person game. Among them, self-game refers to the game between the intelligent agent and its own copy or its own historical state when N = 1; two-person game and multi-person game refer to the game between multiple intelligent agents when N ≥ 2. According to the relationship between competition and cooperation among game participants, von Neumann and Oscar Morganstein divided the game into two frameworks of cooperative game and non-cooperative game in their co-authored book “Game Theory and Economic Behavior”. Among them, cooperative game refers to the game in which players can form alliances to maximize collective benefits through cooperation; non-cooperative game refers to the game in which players (or players) do not cooperate with each other, including not allowing to pass messages, sign alliances, exchange structures, take care of each other, and not allowing mandatory arrangements between each other during the game. Non-cooperative game requires participants to independently choose strategies and strive for the best results for themselves. Further, non-cooperative game can be divided into complete information game and incomplete information game from the perspective of the degree of knowledge of players about game-related information. Among them, complete information game requires that 1) each player in the game knows the strategy set and payoff function of other players, and 2) each player also knows that other players know that he knows the strategy set and payoff function of other players; incomplete information game refers to non-cooperative game in which some players do not know the payoff of others. In addition, from the perspective of the order of strategy selection of game players during the game, non-cooperative game can be divided into static game and dynamic game. Among them, static game refers to a game in which all players simultaneously, or can be considered simultaneously, select strategies; dynamic game refers to a non-cooperative game in which the strategy selection and action of players have a sequence, and the latter selection and action of players can see the decision-making behavior of other players before decision-making. For dynamic game, from the perspective of the understanding of game process by game players, it can be divided into perfect information game and imperfect information game. Among them, perfect information dynamic game refers to a dynamic game in which all players are fully aware of the game process; and imperfect information dynamic game refers to a dynamic game in which there are players who are not fully aware of the game process. From the perspective of the number of strategies, the game can be divided into finite game and infinite game. Among them, finite game refers to a game in which the strategy set of all players contains a finite number of strategies; infinite game refers to a game in which at least one player's strategy set contains an infinite number of strategies.

[0018] For static game in non-cooperative game, strategy or norm is usually used to express. For dynamic game in non-cooperative game, a virtual player called "nature" is introduced to simulate the randomness and uncertainty in the process of dynamic game, and extended form is usually used to express. Extended form can be converted with strategy form, and the specific conversion method is not described here. The following introduces the elements of strategy form and extended form. Strategy form is usually in the form of matrix, which has three elements, namely player set N, strategy set Si of player i and payoff function Pi of player i. Extended form has six elements, namely player set N, action order or game order of player, information set Ii of player i, strategy set Si of player i, payoff function Pi of player i and probability distribution of virtual player "nature". For cooperative game, coalition form is usually used to express, which has two elements, namely player set N and characteristic function v(S). Among them, for the subset S of players, the characteristic function v(S) represents the maximum value that the members of the coalition S can create.

[0019] III. Human-in-the-loop reinforcement learning

[0020] With the application of machine learning algorithms in practical scenarios, humans have higher requirements for the accuracy, safety and cost of algorithms, and users in different application fields are often non-machine learning technical personnel. Therefore, how to let non-machine learning technical personnel participate in the training and deployment process of algorithms has become the core problem of human-in-the-loop machine learning. Reinforcement learning is a kind of machine learning, which mainly learns in the interaction process between agent and environment. Many studies apply the idea of human-in-the-loop machine learning to the field of reinforcement learning.

[0021] Human-in-the-loop machine learning is a set of strategies that combine human and machine intelligence in artificial intelligence applications. The goal is usually to perform one or more of the following strategies: improve the accuracy of machine learning models, achieve the accuracy of machine learning models faster, combine human and machine intelligence to maximize accuracy, and improve efficiency by assisting human tasks through machine learning; the general human-in-the-loop machine learning architecture is an iterative process that combines human and machine components.

[0022] Annotation and active learning are core technologies in human-in-the-loop machine learning. Annotation is the process of labeling raw data to transform it into training data for machine learning. Most data scientists spend more time on preparing and labeling datasets than building machine learning models. Quality control of manual annotation relies on more complex statistics than most machine learning models, making it crucial to invest the necessary time in learning how to create high-quality training data. Active learning is the process of deciding which data to sample for manual annotation. No single algorithm, architecture, or set of parameters can make a machine learning model accurate in all situations, and no single active learning strategy is optimal for all use cases and datasets. Transfer learning retrains a portion of a neural model to adapt to a new task (pre-trained model) or uses the parameters of one neural model as input to another, applying an existing model to a new task instead of training from scratch, thus avoiding cold starts.

[0023] Reinforcement learning is based on the idea of ​​an agent learning through continuous interaction with its environment. The agent acquires information such as rewards and states from the environment, and the environment responds by changing its actions based on the agent's choices. Furthermore, assuming the agent has a goal to achieve, it learns how to reach that goal by observing the impact of its interactions with the environment.

[0024] When using reinforcement learning algorithms to handle a decision problem, the problem is usually first modeled as a Markov Decision Process (MDP). An MDP is an optimization model of an agent that performs actions in a stochastic environment, consisting of tuples. Define it, where Represents a set of states, To represent a set of actions, This represents the state transition function, indicating the state transition function in the given state. Execution of actions Later arrival state The probability, i.e. Let $R(s, a)$ be the reward function, where $R(s, a)$ represents the reward function when the agent is in state $s$. Execution of actions Instant rewards obtained at that time; A discount factor defines the agent's preference for immediate or long-term rewards. The reward function defines the agent's objective in the environment, while the transition function reflects the effect of the agent's chosen action in each specific state.

[0025] Reinforcement learning algorithms have been proven to be effective in scenarios with well-designed reward functions and easy interaction with the environment. However, in some real-world applications, explicit reward functions do not exist, and in some application scenarios, the environment is related to specific hardware devices, and frequent interaction between the agent and the environment will result in expensive costs, and even catastrophic failures in some scenarios with high safety requirements.

[0026] To solve the above problems, human-in-the-loop reinforcement learning provides reward functions and task knowledge by introducing human interaction with reinforcement learning agents, reduces the frequency of interaction between the agent and the environment, and safely shapes the behavior of the agent. Human-in-the-loop reinforcement learning tailors specific elements of reinforcement learning algorithms to improve their performance or generate appropriate strategies for specific tasks.

[0027] The advantage of human-in-the-loop reinforcement learning algorithms is that they integrate prior knowledge of the task on the basis of the existing performance of reinforcement learning algorithms, and are more effective than fully autonomous reinforcement learning algorithms in some specific application scenarios. Humans obtain interaction data generated during the interaction between the agent and the environment, and guide the agent through feedback, mainly in the modification of rewards and actions. Therefore, the core of human-in-the-loop reinforcement learning algorithms is human feedback, which includes prior knowledge about the real world and evaluation of the agent's behavior. After obtaining feedback, the reward and action are modified through reward shaping and policy shaping methods to improve the learning efficiency of the reinforcement learning agent and the safety during deployment.

[0028] Currently, human-in-the-loop autonomous driving training based on deep reinforcement learning faces two major bottlenecks:

[0029] (1) Environment vehicle behavior is not realistic, lacking adaptation to different driving styles. In existing virtual simulation platforms (Carla, SUMO, SMARTS, etc.), environment vehicles are mostly controlled by fixed mathematical models (such as IDM+MOBIL) or simple ML models, with single and conservative behavior patterns, making it difficult to reproduce the diversity and prediction of "human-human" driving interactions. As a result, the distribution of training data and human driving data in real road scenarios is severely disconnected, leading to a significant decline in safety and adaptability when migrating agents with good performance in simulation to reality.

[0030] (2) Lack of a soft and hardware digital twin platform that coordinates with environment vehicle control methods. Even with improved environment vehicle control algorithms, it is difficult to truly reflect the "human-vehicle-environment" closed-loop interaction in pure software simulation. The lack of a soft and hardware integrated digital twin system that can access the human-machine cockpit, motion platform, and real vehicle perception in real time makes human-in-the-loop training unable to obtain high-fidelity, low-risk extreme scenario data. SUMMARY

[0031] The present application aims at the problems of high cost, difficulty in obtaining dangerous or extreme scenes and safety risk in real road data collection, and the problems of too simple decision logic of environment vehicles in existing virtual simulation, large gap between behavior intention and real human drivers, and inconsistency between simulation feedback data and real road data leading to difficulty in direct landing of models, and proposes a human-in-the-loop automatic driving training and testing system based on digital twinning, to realize real-time interaction between automatic driving vehicles and simulated driving environment.

[0032] The technical scheme adopted by the present application is:

[0033] A human-in-the-loop automatic driving training and testing method based on digital twinning is provided, comprising the following steps:

[0034] Obtain the driving data of the human-in-the-loop driver based on digital twinning, and input it into the pre-constructed human-in-the-loop driver social value orientation identification model to output the preference selection weight between safety and efficiency of the human-in-the-loop driver, and then convert the weight into the angle for evaluating the social value orientation of the human-in-the-loop driver;

[0035] The ego vehicle of the human-in-the-loop driver is taken as the leader, and the virtual environment vehicle is pre-constructed as the follower, and whether the environment vehicle and the ego vehicle game occurs is judged according to the distance between the vehicles;

[0036] When the game occurs, the behavior strategy of the environment vehicle is output according to the pre-constructed game model, the game model is specifically based on the angle of the social value orientation of the human-in-the-loop driver of the ego vehicle to construct the game comprehensive cost function of the environment vehicle related to the ego vehicle within a certain range, and the game comprehensive cost function is constructed based on the horizontal and vertical behaviors of the leader and the follower; the behavior strategy of the environment vehicle with the minimum cost is taken as the optimal solution by solving the game comprehensive cost function;

[0037] The trajectory of the environment vehicle is planned according to the optimal solution, and the control amount of the environment vehicle is output through the vehicle controller.

[0038] According to the above technical scheme, the specific construction process of the human-in-the-loop driver social value orientation identification model is:

[0039] Use the existing digital twinning hardware platform to conduct multiple experiments, collect driving trajectories to generate a trajectory set;

[0040] A human driving reward function model based on inverse reinforcement learning and a human driving behavior probability distribution are used to construct a target function of the trajectory set; the solution of the target function is the reward parameter of the human driving reward function model, corresponding to the preference selection weight of the driving efficiency and social safety of the driver;

[0041] Solve the target function to obtain the two preference selection weights under the optimal reward parameter.

[0042] According to the technical solution, the weight ratio is specifically mapped to an angle value of the driver's social value orientation in the loop.

[0043] According to the technical solution, the game comprehensive cost function includes a safety cost function and an efficiency cost function, wherein the safety cost is measured by the speed difference and the distance from the nearest vehicle on the target lane, and the additional risk caused by the lateral offset and the heading deviation; and the efficiency cost is measured by the reciprocal of the square difference between the current speed and the maximum speed.

[0044] According to the technical solution, when solving the game comprehensive cost function, the optimal behavior strategy is solved as a double-layer optimization problem, the decision behavior of the ego vehicle is taken as the optimal solution of the inner game, the decision behavior of the environment vehicle is discretized within the range of the vehicle motion dynamics constraint, for each pair of discretized actions, the game comprehensive cost of the environment vehicle is calculated, and the behavior strategy that minimizes the game comprehensive cost is selected as the final game result behavior of the environment vehicle, wherein the decision behavior includes the lane changing behavior and the longitudinal speed of the corresponding vehicle.

[0045] According to the technical solution, the trajectory of the environment vehicle is specifically planned by a quintic polynomial based on the optimal solution.

[0046] According to the technical solution, the vehicle controller is an MPC or LQR controller.

[0047] According to the technical solution, if it is determined that the environment vehicle and the ego vehicle will not game according to the distance between the vehicles, the environment vehicle is controlled based on a pre-constructed lateral and longitudinal driver model.

[0048] The application also provides a human-in-the-loop automatic driving training and testing system based on digital twinning, which comprises a digital twinning hardware platform and an environment vehicle controller based on social game, wherein the digital twinning hardware platform is used to realize the closed-loop interaction of “human-vehicle-environment” and automatic driving training, and specifically, the sensor continuously collects the state of the entity vehicle and the external environment data to build an interactive environment for the driver and the virtual scene; the environment vehicle controller based on social game is used to generate and control the environment vehicle in the virtual scene according to the method described in the above technical solution, so that the interaction process between the environment vehicle and the driver is close to the interaction process between the automatic driving vehicle and other vehicles in the environment in the real world.

[0049] The application also provides a computer storage medium, which stores a computer program that can be executed by a processor, and the computer program executes the human-in-the-loop automatic driving training and testing method based on digital twinning described in the above technical solution.

[0050] The beneficial effects generated by the present application are: based on the online acquisition of the in-loop driver driving data based on digital twinning, the in-loop driver's preference weight between safety and efficiency in the perspective of social value orientation is obtained in real time for evaluating the in-loop driver, and then a game model of the surrounding virtual environment vehicles is constructed based on the driver's social value orientation, when the game between the ego vehicle and the environment vehicles occurs, the game model is used to drive the environment vehicles to make interactive behavior strategies close to real humans, and then the trajectory of the environment vehicles is planned according to the behavior strategies, and the control amount of the environment vehicles is output through the vehicle controller, so that the interaction process between the environment vehicles and the driver is closer to the interaction process between the autonomous vehicles and other vehicles in the environment in the real world, the representativeness of the training sample and the generalization ability of the model in the real scene are significantly improved, and the human-in-the-loop autonomous driving training and testing are efficiently completed.

[0051] The human-in-the-loop autonomous driving training and testing system based on digital twinning of the present application comprises a digital twinning hardware platform and an environment vehicle controller based on social game, realizes the closed-loop interaction of "human-vehicle-environment" and the autonomous driving training, effectively eliminates the differences between the simulation environment and the real road in terms of data distribution, decision logic and human-vehicle interaction through the deep cooperation of the software and hardware platform and the control algorithm, provides a real and controllable training and verification environment for the high-level autonomous driving system, and greatly improves the safety and adaptability of the model in the actual scene.

[0052] Of course, implementing any product of the present application does not necessarily require achieving all the advantages described above at the same time. BRIEF DESCRIPTION OF DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0054] Figure 1 is a flowchart of the human-in-the-loop autonomous driving training and testing method based on digital twinning of an embodiment of the present application;

[0055] Figure 2 is an architecture diagram of the human-in-the-loop autonomous driving training and testing system based on digital twinning of an embodiment of the present application;

[0056] Figure 3 is a basic digital twinning system architecture diagram of an embodiment of the present application;

[0057] Figure 4 is a human-in-the-loop driving simulation system diagram for digital twinning of an embodiment of the present application;

[0058] Figure 5 is a social value orientation calculation flowchart of an embodiment of the present application;

[0059] Figure 6 is a social game flowchart of an embodiment of the present application;

[0060] Figure 7 is an environmental vehicle control flowchart based on a social game of an embodiment of the present application;

[0061] Figure 8 is a technical architecture diagram of an environmental vehicle controller based on a social game of an embodiment of the present application;

[0062] Figure 9 is an environmental vehicle control quantity calculation flowchart of an embodiment of the present application. DETAILED DESCRIPTION

[0063] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0064] It should be noted that the diagrams provided in the embodiments of the present application only schematically illustrate the basic concepts of the present application, and therefore only the components related to the present application are shown in the diagrams, rather than the number, shape and size of the components when actually implemented. The actual implementation of each component can be arbitrarily changed in terms of shape, number and proportion, and the layout pattern of the components can also be more complex.

[0065] In the present application, it should also be noted that, if terms such as "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer" and the like appear, the indicated orientation or positional relationship is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the present application and simplifying the description, and therefore cannot be understood as indicating or implying that the indicated device or element must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application. In addition, if the terms "first" and "second" appear, they are only for description and distinction purposes, and cannot be understood as indicating or implying relative importance.

[0066] In addition, it should also be noted that the features of various embodiments of the present application can be combined or integrated partially or entirely, and can interact and operate in different ways as understood by those skilled in the art. Each embodiment can be implemented independently of each other, or in an associated relationship.

[0067] The application mainly constructs a "human-vehicle-environment" closed loop, uses the online estimated driver social value orientation (SVO) and game to drive the environment vehicle to make interactive decisions close to real humans, and through the cooperation of the software and hardware platform and the control algorithm, the distribution gap between the simulation environment and the real road is greatly reduced, and the representativeness of the training sample and the generalization ability of the model in the real scene are significantly improved.

[0068] As shown in Figure 1 The embodiment of the application based on digital twin human-in-the-loop automatic driving training and testing method includes the following steps:

[0069] S1, obtain the driving data of the in-the-loop driver based on digital twin, and input it into the pre-constructed in-the-loop driver social value orientation identification model, output the preference selection weight of the in-the-loop driver between safety and efficiency, and then convert the weight into the angle for evaluating the social value orientation of the in-the-loop driver;

[0070] S2, taking the in-the-loop ego vehicle of the driver as the leader and pre-constructing a virtual environment vehicle as the follower, judging whether the environment vehicle and the ego vehicle have a game according to the distance between the vehicles;

[0071] S3, when the game occurs, output the behavior strategy of the environment vehicle according to the pre-constructed game model, the game model is specifically based on the angle of the in-the-loop driver social value orientation of the ego vehicle to construct a game comprehensive cost function of the environment vehicle related to the ego vehicle within a certain range, and the game comprehensive cost function is constructed based on the horizontal and vertical behaviors of the leader and the follower; solving the game comprehensive cost function, taking the behavior strategy of the environment vehicle with the minimum cost as the optimal solution;

[0072] S4, planning the trajectory of the environment vehicle according to the optimal solution, and outputting the control amount of the environment vehicle through the vehicle controller.

[0073] In order to realize the above method, the application proposes a human-in-the-loop automatic driving training and testing system based on digital twin, which is used to realize the real-time interaction of the automatic driving vehicle and the simulation driving environment, collect training data, and further evaluate the performance of the model. In view of the problems that the data collection cost of the real road is high, the dangerous or extreme scene is difficult to obtain and there is a safety risk, and the decision logic of the environment vehicle in the existing virtual simulation is too simple, the behavior intention is far away from the real human driver, and the simulation feedback data is inconsistent with the real road data, which leads to the problem that the model is difficult to directly land, as shown in Figure 2 An automatic driving training and testing system containing a digital twin software and hardware platform and an environment vehicle controller based on social game is designed.

[0074] The digital twin software and hardware platform is mainly used for providing a software and hardware basic environment for the automatic driving algorithm training process of the application, realizing "person-vehicle-environment" closed-loop interaction and automatic driving training, continuously collecting entity vehicle state and external environment data through sensors, and building an interactive environment for the driver and the virtual scene. Specifically, the digital twin software and hardware platform includes a hardware system and a software system. The hardware system includes an intelligent networked line control passenger vehicle development platform, a wireless bridge, a server, a driving simulator (including a three-degree-of-freedom motion platform and an actual vehicle cockpit), a 3D loop screen and the like. The software system mainly includes a virtual scene model, a virtual vehicle, an environment vehicle controller based on social game and the like.

[0075] The application builds a basic digital twin system based on existing hardware as shown in the figure. Figure 3 The figure mainly presents the system composition from the abstract architecture level (while Figure 2 is from the hardware implementation level, both of which jointly illustrate the digital twin training and testing platform built by the application from different levels). From a higher system function module level, the digital twin system abstractly defines three core collaborative subsystems supporting the entire training process, including a physical entity data acquisition platform, a human-in-the-loop driving simulation system and a data interaction communication platform, to support "person-vehicle-environment" closed-loop interaction and automatic driving training. The physical entity data acquisition platform is mainly realized by the intelligent networked line control passenger vehicle and its sensors (cameras, laser radars and the like) in Figure 2 , and is responsible for collecting point cloud, video, pose and vehicle state data of the real world. The human-in-the-loop driving simulation system is mainly composed of the driving simulator, the 3D loop screen, the server and the software system (the virtual scene model, the virtual vehicle and the environment vehicle controller) in Figure 2 , and provides a simulation environment for the driver to interact with the high-fidelity virtual environment. The data interaction communication platform is mainly realized by the wireless bridge (including the bridge 1 and the bridge 2) in Figure 2 , and is responsible for establishing a reliable and real-time bidirectional data communication channel between the real vehicle, the driving simulator (the person) and the virtual environment, and transmitting perception data and control instructions.

[0076] Figure 2 The specific hardware devices listed in Figure 3 and the associated software modules (such as the environment vehicle controller) are the specific implementation carriers of the three abstract function subsystems (the physical entity data acquisition platform, the human-in-the-loop simulation and the data interaction communication) defined in Figure 3 . The architecture of Figure 2The specific technical facilities required to support these abstract functions are then demonstrated. The two are closely integrated and together constitute a complete "human-vehicle-environment" closed-loop interaction system, providing a solid hardware and software foundation environment for subsequent automatic driving algorithm training, testing and verification.

[0077] Specifically, first, the intelligent networked drive-by-wire passenger car platform is used as a physical entity data collection platform, providing real vehicle entity and environmental perception capabilities, built-in drive-by-wire domain controller and chassis power system, and continuously collecting vehicle state and external environment data through GPS, IMU and other sensors. Second, the human-in-the-loop driving simulation system is composed of a driving simulator, a 3D loop screen and a local server, which builds a high-fidelity interactive environment for the driver and the virtual scene. As shown in Figure 4 The driving simulator includes a three-degree-of-freedom motion platform and a real vehicle cockpit integrated into two parts. The real vehicle cockpit is equipped with a force feedback steering wheel, an accelerator pedal and a gear controller, which are connected to the hardware (such as dSPACE) through a serial port (such as DB9) and output real CAN control signals. The three-degree-of-freedom motion platform interfaces with CAN messages and reproduces vehicle attitude through servo motors. The 180° loop screen display system (3D loop screen) combines with high-performance servers to render and project virtual driving scenes to the instrument panel and rearview mirror display in real time. Finally, the wireless bridge is used as a data interaction communication platform to realize bidirectional data transmission between the virtual world, the cockpit and the remote real vehicle on the IP / TCP channel, and finally issue control instructions under the CAN protocol to ensure that the virtual vehicle state is highly consistent with the real vehicle motion. The platform parts work closely together to provide a reliable physical basis for subsequent automatic driving algorithm training and verification.

[0078] It can be seen that the present application is based on the concept of digital twinning, which integrates real drive-by-wire passenger cars, three-degree-of-freedom cockpits, loop screen displays, motion platforms and high-performance simulation environments to form a "human-vehicle-environment" closed-loop interactive hardware and software integrated system. This platform not only supports human-in-the-loop participatory training, but also seamlessly compatible with non-human-in-the-loop (pure simulation) agent training, greatly improving the flexibility and efficiency of training deployment, reducing the safety risks and costs of field testing, and providing reliable protection for the rapid iteration and performance evaluation of automatic driving algorithms.

[0079] The virtual scene model and the virtual vehicle in the software system are used to provide a basic space and a controlled object. The environmental vehicle controller based on social game is used to control the environmental vehicle interacting with the driver, so that the interaction process between the environmental vehicle and the driver is closer to the interaction process between the automatic driving vehicle and other vehicles in the environment in the real world. By narrowing the distribution gap between simulation and reality, the algorithm landing is improved, and through the deep cooperation of the software and hardware platform and the control algorithm, the invention effectively eliminates the differences between the simulation environment and the real road in data distribution, decision logic and human-vehicle interaction, provides a real and controllable training and verification environment for high-level automatic driving system, and greatly improves the safety and adaptability of the model in the actual scene.

[0080] The software system workflow mainly includes "in-loop driver social value orientation identification" and "environmental vehicle control based on social game".

[0081] (1) In-loop driver social value orientation identification:

[0082] Firstly, a value is defined to describe the driving style of human drivers, i.e. social value orientation SVO (Social Value Orientation), which is represented by an angle characterizing the weight preference of the driver between "public interest" (safety) and "personal interest" (efficiency). The value range and meaning of the angle : : Extremely altruistic, completely prioritizing public safety; : Prosocial, giving priority to safety but also considering efficiency; : Selfish, only focusing on personal efficiency; : Competitive, mainly focusing on efficiency with a little antagonism.

[0083] Secondly, an angle quantization index is obtained by using the "efficiency / safety" feature weight ratio mapping learned by inverse reinforcement learning, which reflects the social preference of the in-loop driver in the automatic driving training process. The driving trajectory of the in-loop driver is collected, the reward function of the driver is solved by using inverse reinforcement learning method, and the SVO value is calculated based on the weight of each part of the reward function.

[0084] As shown in Figure 5 , the specific steps of calculating the social value orientation of the in-loop driver are as follows:

[0085] 1) Collection of human driver behavior data

[0086] ​Before training the autonomous driving algorithm, an initial driving environment is constructed to collect human driver's driving data, in which the environment vehicles are controlled by the conventional longitudinal and lateral driving model (IDM+MOBIL). The human drivers use the existing digital twin hardware platform to conduct 20 rounds of experiments, collect 20 driving trajectories and form a trajectory set . Each trajectory contains states (speed, position, etc.) and actions .

[0087] 2) Construct a human driving reward function model

[0088] Based on the existing human driving trajectory set , the features are extracted to define the reward function of inverse reinforcement learning, and two features are defined, which are representing driving efficiency and representing social safety .

[0089]

[0090]

[0091] The reward function is defined as:

[0092]

[0093] represent the weight parameters of and in the reward function .

[0094] 3) Construct a human driving behavior probability distribution

[0095] The maximum entropy inverse reinforcement learning method is used to estimate the parameters constructed in the last step. It is assumed that the human driver randomly (but prefers high returns) selects from a set of possible trajectories based on the returns of each trajectory. Since the real-world trajectory space is continuous and high-dimensional, it is not feasible to directly use human driving trajectories to solve the weight of the reward function. The present application uses the maximum entropy inverse reinforcement learning method to solve it. The process of human selecting driving behavior under the maximum entropy inverse reinforcement learning framework follows the Boltzmann distribution, and the higher the return of the trajectory, the higher the probability of being selected. The probability distribution model is modeled as:

[0096]

[0097] The formula represents that under the given , the trajectory The probability of being chosen by a human driver is approximately equal to the ratio of its exponential return to the sum of exponential returns of all candidate trajectories. Trajectory Refers to all possible driving trajectories of a human driver. Since only a limited number of real human driving data is collected, a random generation method is used to generate a series of trajectories according to the constraints of the vehicle kinematic model, according to the same trajectory length, and based on the initial state (Taken from the first moment of each example trajectory in the human example) to generate a series of trajectories . Molecule is directly proportional to the total return of the trajectory Under the current reward parameter . Denominator is the normalization of the weight of all candidate trajectories (or sampled trajectories), which approximates the partition function (used for probability distribution normalization).

[0098] 4) Solve the reward function parameter

[0099] is the set of observed "example trajectories" generated by real human driving. It is hoped that the reward parameter will be adjusted so that the inverse reinforcement learning model has the highest probability of generating trajectories on these trajectories. Log-likelihood is used to measure the "model's ability to explain the example trajectory". The larger the value, the better the model can reproduce human behavior.

[0100] The core of maximum entropy inverse reinforcement learning is to find a set of reward parameters, so that under this reward, human behavior is the most "attractive" among all possible behaviors. By embedding the probability distribution model into the log-likelihood as the optimization objective, and adjusting by gradient ascent until the trajectory distribution of the model is consistent with the statistical characteristics of the human example trajectory (acceleration, distance, speed distribution). The expression formula of the log-likelihood objective function is as follows:

[0101]

[0102] The gradient solving formula based on the log-likelihood objective function is:

[0103]

[0104] Solve by gradient ascent, and get the optimal by iterating the objective function and the regularization coefficient with learning rate : ​Finally, the parameters of the expert example under the parameters .

[0105] 5) SVO angle mapping

[0106] Map the weight ratio to the SVO angle. Through the third step, the converged efficiency feature weight and the safety feature weight , the social value orientation angle is solved . The calculation formula is as follows:

[0107]

[0108]

[0109] Where the threshold is used to distinguish "competitive" from other types.

[0110] Finally, the relative weights of efficiency and safety are converted into an intuitive SVO angle , which is used to evaluate the social value orientation of the in-loop driver.

[0111] (2) Environment vehicle control based on social game:

[0112] As shown in Figures 6-8 , first, the ego vehicle (driver-in-loop vehicle) is taken as the leader and the environment vehicle as the follower, and based on the distance between the vehicles, it is determined whether the environment vehicle is in game with the ego vehicle. In this invention, when the longitudinal distance between the environment vehicle and the ego vehicle is less than 3m, the game interaction is triggered, otherwise the general lateral and longitudinal driver model is used to control the environment vehicle.

[0113] Second, the lateral and longitudinal behavior of the leader and follower can be solved based on Stackelberg game. Two cost functions are constructed, which are safety cost function and efficiency cost function .

[0114]

[0115] is used to represent the lane changing behavior, where -1 represents left lane changing, 0 represents maintaining the current lane, and 1 represents right lane changing. When not changing lanes, only the safety cost of the current lane is calculated; when changing lanes, the safety of the target lane is evaluated.

[0116]

[0117] represents the longitudinal speed difference between the vehicle and the preceding vehicle, and the larger the speed difference, the higher the risk of rear-end collision. This represents the net distance between the two vehicles; the smaller the distance, the higher the risk. (Exponential term and coefficient) This is used to balance the safety impacts of "approaching speed" and "being too close".

[0118]

[0119] Used to measure the speed difference and distance to the nearest vehicle in the target lane, with the same structure as... The formula is consistent. Used to measure the additional risks associated with lateral offset and heading deviation.

[0120]

[0121] Efficiency Cost At the current speed Maximum speed over distance The efficiency cost is measured by the reciprocal of the squared difference; the closer it is to the maximum speed, the lower the efficiency cost. Represents the weighting coefficient, reflecting the first The degree to which a vehicle prioritizes speed and efficiency; The larger the value, the stronger the penalty for deviating from the maximum speed. This represents a small positive number added to the denominator to prevent it from approaching zero. Solving for the optimal decision can be viewed as a two-level optimization problem, typically solved using the approach of "backward induction".

[0122] The specific steps are as follows:

[0123] 1) The decision-making behavior of the vehicle As the optimal solution in the inner game. This indicates the longitudinal speed of the vehicle (leader).

[0124] 2) Decision-making behavior of environmental vehicles Discretize the vehicle's motion dynamics within the range of constraints, and for each pair of discretized actions, calculate the comprehensive game cost of the vehicle in that environment. , Choose to reduce costs Minimal behavior The output represents the final game outcome behavior of the environmental vehicle, and serves as the decision-making behavior of the environmental vehicle.

[0125] (3) Environmental vehicle control based on quintic polynomial programming and MPC:

[0126] like Figure 9 As shown, the calculation of environmental vehicle control quantities is mainly based on environmental vehicle behavior obtained through game theory. The system employs quintic polynomial programming to determine the environmental vehicle trajectory and utilizes the MPC controller to output control quantities.

[0127] First, construct the fifth-degree polynomial as follows:

[0128]

[0129] In the formula, It is the constant term of the polynomial, which represents the value of the ordinate of the curve when the x-coordinate is 0. Let the parameters of the polynomial be to be determined. Let be the trajectory curve function of the environmental vehicle. Let the starting and ending points of the environmental vehicle trajectory be . , The vehicle's attitude at the start and end times should be consistent with... The axes remain parallel, meaning that at these two points, both the heading angle and the steering angle of the front wheels are 0.

[0130] Secondly, based on these conditions, a solution equation based on a fifth-order polynomial is established to obtain the motion trajectory of the vehicle in the environment.

[0131]

[0132] Finally, the obtained trajectory and the desired velocity are compared. The lateral and longitudinal control quantities of the vehicle are obtained by inputting them into the MPC controller. Since trajectory planning and MPC controller construction are routine operations and not the focus of this invention, they will not be described in detail here.

[0133] As can be seen, the proposed social game-driven adaptive control method for environmental vehicles addresses the problems of monotonous environmental vehicle behavior and lack of response to human driving styles in existing simulations. This invention introduces online estimation of Social Value Orientation (SVO) and integrates safety and efficiency costs into the game framework to dynamically adjust the vehicle's strategy. This method can capture the social preferences and interaction details in driver behavior in real time, making the generated training data more closely resemble real road scenarios in terms of diversity and uncertainty, significantly enhancing the generalization ability and robustness of the autonomous driving model.

[0134] This application also provides a computer-readable storage medium, such as flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, server, app store, etc., which stores a computer program. When the program is executed by a processor, it implements the corresponding function. The computer-readable storage medium of this embodiment, when executed by a processor, implements the human-in-the-loop autonomous driving training and testing method based on digital twins, as described in this embodiment.

[0135] It should be noted that, according to the needs of implementation, each step / component described in the present application can be split into more steps / components, or two or more steps / components or part of the operation of the steps / components can be combined into a new step / component, to achieve the purpose of the present application.

[0136] The size of the serial number of each step in the above embodiment does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0137] It should be understood that, for those skilled in the art, improvements or changes can be made according to the above description, and all these improvements and changes shall fall within the protection scope of the appended claims of the present application.

Claims

1. A human-in-the-loop training and testing method for autonomous driving based on digital twins, characterized in that, Includes the following steps: Acquire driving data of drivers in the loop based on digital twins, input it into a pre-built social value orientation identification model of drivers in the loop, output the weight of drivers' preference between safety and efficiency, and then convert the weight into a perspective for evaluating the social value orientation of drivers in the loop. The driver's own vehicle in the loop is the leader, and virtual environmental vehicles are pre-built as followers. The distance between the vehicles determines whether the environmental vehicles are in a game with the driver's own vehicle. When a game occurs, the behavior strategy of the environmental vehicle is output according to the pre-constructed game model. The game model is specifically based on the social value orientation of the driver in the loop of the vehicle to construct the game comprehensive cost function of the environmental vehicles related to the vehicle within a certain range. Moreover, the game comprehensive cost function is constructed based on the horizontal and vertical behaviors of the leader and the follower. Solve the game comprehensive cost function and take the behavior strategy of the environmental vehicle with the lowest cost as the optimal solution. The optimal solution is used to plan the environmental vehicle trajectory, and the control quantity of the environmental vehicle is output through the vehicle controller.

2. The human-in-the-loop training and testing method for autonomous driving based on digital twins according to claim 1, characterized in that, The specific construction process of the in-loop driver social value orientation identification model is as follows: Multiple rounds of experiments were conducted using existing digital twin hardware platforms to collect driving trajectories and generate trajectory sets; The objective function of the trajectory set is constructed based on the human driving reward function model and the probability distribution of human driving behavior using inverse reinforcement learning; the solution of the objective function is the reward parameter of the human driving reward function model, which corresponds to the driver's preference weights for driving efficiency and social safety. Solving the objective function yields the two preference selection weights under the optimal reward parameters.

3. The human-in-the-loop training and testing method for autonomous driving based on digital twins according to claim 2, characterized in that, Specifically, the weighting ratio is mapped to the perspective value of the social value orientation of the driver in the loop.

4. The human-in-the-loop training and testing method for autonomous driving based on digital twins according to claim 1, characterized in that, The game-theoretic cost function includes a safety cost function and an efficiency cost function. The safety cost is measured by the speed difference and distance to the nearest vehicle in the target lane, as well as the additional risks caused by lateral deviation and heading deviation. The efficiency cost is measured by the reciprocal of the square difference between the current speed and the maximum speed.

5. The human-in-the-loop training and testing method for autonomous driving based on digital twins according to claim 1, characterized in that, When solving the game's overall cost function, the solution to the optimal behavioral strategy is treated as a two-level optimization problem. The decision-making behavior of the vehicle is taken as the optimal solution of the inner game, and the decision-making behavior of the environmental vehicle is discretized within the range of vehicle motion dynamics constraints. For each pair of discretized actions, the overall game cost of the environmental vehicle is calculated, and the behavioral strategy that minimizes the overall game cost is selected as the final game outcome behavior of the environmental vehicle. The decision-making behavior includes the corresponding vehicle's switching behavior and longitudinal speed.

6. The human-in-the-loop training and testing method for autonomous driving based on digital twins according to claim 1, characterized in that, The environmental vehicle trajectory is specifically planned using a fifth-order polynomial based on the optimal solution.

7. The human-in-the-loop training and testing method for autonomous driving based on digital twins according to claim 1, characterized in that, The vehicle controller is either an MPC or an LQR controller.

8. The human-in-the-loop training and testing method for autonomous driving based on digital twins according to any one of claims 1-7, characterized in that, If it is determined that there will be no game between the environmental vehicles and the driver vehicle based on the distance between the vehicles, then the environmental vehicles will be controlled based on the pre-built horizontal and vertical driver models.

9. A human-in-the-loop training and testing system for autonomous driving based on digital twins, characterized in that, The system includes a digital twin hardware platform and a social game-based environment vehicle controller. The digital twin hardware platform is used to realize closed-loop interaction between "human-vehicle-environment" and autonomous driving training. Specifically, it continuously collects data on the physical vehicle status and external environment through sensors to build an interactive environment for the driver and the virtual scene. The social game-based environment vehicle controller is used to generate and control the environment vehicle in the virtual scene according to the method of any one of claims 1-8, so that the interaction process between the environment vehicle and the driver is close to the interaction process between autonomous vehicles and other vehicles in the environment in the real world.

10. A computer storage medium, characterized in that, It contains a computer program that can be executed by a processor, which performs the human-in-the-loop autonomous driving training and testing method based on digital twins as described in any one of claims 1-7.

Images