Human-in-the-loop automatic driving training and testing method and system based on digital twinning

Abstract

The invention discloses a human-in-the-loop automatic driving training and testing method and system based on digital twinning, and the method comprises the steps: obtaining the driving data of in-the-loop drivers based on digital twinning, inputting the driving data into a pre-constructed in-the-loop driver social value orientation recognition model, outputting a preference selection weight representing the safety and efficiency of the in-the-loop driver, and converting the weight into an angle for evaluating the social value orientation of the in-the-loop driver; the method comprises the following steps: taking a vehicle with a driver in the surrounding as a leader, taking a pre-constructed virtual environment vehicle as a follower, and judging whether the environment vehicle games with the vehicle or not according to the distance between the vehicles; when the game occurs, outputting a behavior strategy of the environment vehicle according to a pre-constructed game model; planning an environment vehicle track according to the optimal solution, and outputting the control quantity of the environment vehicle. According to the invention, real-time interaction between an automatic driving vehicle and a simulation driving environment can be realized, and human-in-the-loop automatic driving training and testing based on digital twinning are efficiently realized.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a human-in-the-loop autonomous driving training and testing method and system based on digital twins. Background Art

[0002] 1. About Inverse Reinforcement Learning: In reinforcement learning, the Markov Decision Process (MDP) is often used as a theoretical framework for stochastic conditions. To find the optimal strategy, an agent makes decisions based on its current state to maximize its reward.

[0003] Define a Markov decision process as: ,in represents a finite set of states, represents the action space, is the state transition probability of the current state to the next state, is the reward function based on the state-action space, Represents the decay factor. In a complete Markov decision process, the strategy For a certain state Possible action The probability of , is only related to the current state and has nothing to do with historical information:

[0004] At the same time, the state value function is introduced and action-value function , it is expected that for any MDP, there is always an optimal strategy , we can obtain the corresponding optimal state value function and optimal action value function, and the corresponding Bellman optimal equation is:

[0005] In inverse reinforcement learning (IRL), we hope to use the idea of ​​IRL to learn a reward function that can guide the autonomous vehicle to make reasonable plans from the driver's decision-making behavior. , where each expert demonstration trajectory is Therefore, we can assume that the expert is trying to optimize an unknown reward function, and the goal of inverse reinforcement learning (IRL) is to find the optimal policy that performs as well as the expert demonstration. Assume that the optimal policy is , then

[0006]

[0007] At this point, according to the Bellman equation:

[0008] To obtain the reward function , rewrite the above equation as:

[0009] Therefore, inverse reinforcement learning has the ability to learn and extract implicit behavioral rules from expert example behaviors and is suitable for complex traffic scenarios.

[0010] 2. About the foundation of game theory: Game theory, also known as strategy theory, is the study of strategic interactions and is applicable to solving decision-making problems in multi-agent systems. Existing classic game models can be categorized in different ways from different perspectives. Based on the model's constituent elements, they can be divided into three main categories: game participants, cooperative relationships, and game strategy space.

[0011] Existing game models can be categorized into self-games, two-player games, and multi-player games based on the number of players (N). Self-games refer to games between an agent and its own replica or historical state when N = 1; two-player and multi-player games refer to games between multiple agents when N ≥ 2. In their co-authored book, "The Theory of Games and Economic Behavior," von Neumann and Oskar Morgenstern categorized games into cooperative and non-cooperative frameworks based on the relationship between competition and cooperation among game players. Cooperative games are those in which players can form alliances and maximize collective interests through cooperation. Non-cooperative games are those in which players (or players) do not cooperate with each other. This includes prohibiting the exchange of information, offensive and defensive alliances, post-game exchange of structures, mutual favoritism, and mandatory agreements. Non-cooperative games require players to independently choose strategies and strive for the best outcome for themselves. Furthermore, non-cooperative games can be divided into perfect information games and incomplete information games based on the degree to which players know relevant game information. Perfect information games require the simultaneous fulfillment of the following two conditions: 1) every player in the game knows the strategy set and payoff function of every other player, and 2) every player knows that every other player knows that they know the strategy set and payoff function of every other player. Incomplete information games are non-cooperative games in which some players are unaware of the payoffs of others. Furthermore, based on the order in which players choose strategies during the game, non-cooperative games can be divided into static games and dynamic games. Static games are games in which all players choose strategies simultaneously, or can be considered to choose strategies simultaneously. Dynamic games are games in which players' strategy choices and actions follow a sequential order, and players who choose and act later can see the decisions of other players before making their decisions. Dynamic games can be divided into perfect information games and incomplete information games based on the degree to which players know the game progress. Perfect information dynamic games refer to dynamic games in which all players have complete knowledge of the game's progress; imperfect information dynamic games refer to dynamic games in which some players have incomplete knowledge of the game's progress. Based on the number of game strategies, games can be divided into finite and infinite games. Finite games are those in which all players' strategy sets contain a finite number of strategies; infinite games are those in which at least one player has an infinite number of strategies.

[0012] Static non-cooperative games are typically represented using a strategic form, also known as a canonical form. Dynamic non-cooperative games introduce a virtual player called "Nature" to simulate the randomness and uncertainty inherent in the dynamic game process, typically represented using an extended form. The extended form can be converted to and from the strategic form; the specific conversion method is not detailed here. The following describes the components of the strategic and extended forms. The strategic form is often expressed in matrix form and has three components: the set of players N, player i's strategy set Si, and player i's payoff function Pi. The extended form has six components: the set of players N, the player's action sequence, also known as the game sequence, player i's information set Ii, player i's strategy set Si, player i's payoff function Pi, and the probability distribution of the virtual player "Nature." Cooperative games are typically represented using a coalition form, which has two components: the set of players N and the characteristic function v(S). Among them, for the subset S of the alliance formed by the players, its characteristic function v(S) represents the maximum value that all members of the alliance S can create.

[0013] 3. About Human-in-the-Loop Reinforcement Learning: With the increasing application of machine learning algorithms in real-world scenarios, humans have placed higher demands on accuracy, security, and cost. Furthermore, users in various application fields are often non-machine learning professionals. Therefore, how to enable non-machine learning professionals to participate in the training and deployment of algorithms has become a core issue for human-in-the-loop machine learning. Reinforcement learning, a type of machine learning, primarily involves learning through the interaction between an intelligent agent and its environment. Numerous studies have applied the principles of human-in-the-loop machine learning to the field of reinforcement learning.

[0014] Human-in-the-loop machine learning is a set of strategies that combine human and machine intelligence in artificial intelligence applications. The goal is usually to implement one or more of the following strategies: improve the accuracy of machine learning models, reach the target accuracy of machine learning models faster, combine human and machine intelligence to maximize accuracy, and assist human tasks with machine learning to improve efficiency. The broad human-in-the-loop machine learning architecture is an iterative process that combines human and machine components.

[0015] Annotation and active learning are core technologies in human-in-the-loop machine learning. Annotation is the process of labeling raw data to become machine learning training data. Most data scientists spend more time organizing and annotating datasets than building machine learning models. Quality control for manual annotation relies on more complex statistics than most machine learning models, so it is crucial to invest the necessary time to learn how to create high-quality training data. Active learning is the process of deciding which data to sample for manual annotation. No single algorithm, architecture, or set of parameters makes a machine learning model consistently accurate, nor is a single active learning strategy optimal for all use cases and datasets. Transfer learning involves retraining a portion of a neural model to adapt to a new task (pretrained model) or using the parameters of one neural model as input for another, adapting an existing model to a new task rather than training from scratch and avoiding a cold start.

[0016] Reinforcement learning is based on the idea of ​​learning through continuous interaction between an agent and its environment. The agent obtains information such as rewards and status from the environment, and the environment changes in response to the agent's actions. Furthermore, the agent is assumed to have a goal to achieve, and it learns how to achieve this goal by observing the impact of its interactions with the environment.

[0017] When using reinforcement learning algorithms to solve a decision problem, we usually first model the problem as a Markov decision process (MDP). MDP is an optimization model for an agent that performs actions in a random environment, consisting of a tuple . Define, where Represents a set of states, Represents a set of actions. Represents the state transfer function, which means that in state Execute actions Post-arrival state The probability of is the reward function, where $R(s, a)$ represents the reward when the agent is in state Execute actions Instant rewards when is a discount factor that defines the agent’s preference for immediate or distant rewards. The reward function defines the agent’s goal in the environment, while the transfer function reflects the effect of the agent’s actions in each specific state.

[0018] Reinforcement learning algorithms have proven effective in scenarios with carefully designed reward functions and easy interaction with the environment. However, in some real-world applications, explicit reward functions are not available. Furthermore, in some scenarios, the environment is tied to specific hardware devices. Frequent interaction between the agent and the environment is costly and can even lead to catastrophic failures in safety-critical scenarios.

[0019] To address these issues, human-in-the-loop reinforcement learning (HIL) introduces a mechanism for humans to interact with the reinforcement learning agent, providing reward functions and task knowledge. This reduces the frequency of interaction between the agent and the environment and safely shapes the agent's behavior. HIL tailors specific elements of the reinforcement learning algorithm to improve its performance or generate appropriate policies for specific tasks.

[0020] The advantage of human-in-the-loop reinforcement learning algorithms is that they integrate prior knowledge of the task with the existing performance of reinforcement learning algorithms. This makes them more effective than fully autonomous reinforcement learning algorithms in certain specific application scenarios. Humans obtain the interaction data generated during the interaction between the agent and the environment and guide the agent through feedback, primarily through modifications to rewards and actions. Therefore, the core of human-in-the-loop reinforcement learning algorithms is human feedback, which includes prior knowledge about the real world and evaluations of the agent's behavior. After obtaining feedback, rewards and actions are modified through methods such as reward shaping and strategy shaping, improving the learning efficiency of the reinforcement learning agent and the security of the deployment process.

[0021] Currently, human-in-the-loop autonomous driving training based on deep reinforcement learning faces two core bottlenecks: (1) The behavior of the environment vehicles is unrealistic and lacks adaptability to different driving styles. In existing virtual simulation platforms (Carla, SUMO, SMARTS, etc.), the environment vehicles are mostly controlled by fixed mathematical models (such as IDM+MOBIL) or simple ML models. The behavior patterns are single and conservative, making it difficult to reproduce the diversity and predictability of "human-human" driving interactions. As a result, the training data is seriously disconnected from the distribution of human driving data in real road scenarios, resulting in a significant decrease in the safety and adaptability of intelligent agents that perform well in simulation when migrated to reality.

[0022] (2) There is a lack of a hardware and software digital twin platform that works in conjunction with the environmental vehicle control method. Even with improved environmental vehicle control algorithms, it is difficult to truly reflect the closed-loop interaction of "human-vehicle-environment" in pure software simulation. The lack of a digital twin system that integrates hardware and software and can access the human-machine cockpit, motion platform, and real-world vehicle perception in real time makes it impossible for human-in-the-loop training to obtain high-fidelity, low-risk extreme scenario data. Summary of the Invention

[0023] The present invention addresses the problems of high cost of collecting data on real roads, difficulty in obtaining dangerous or extreme scenarios, and safety risks, as well as the problems in existing virtual simulations where the decision-making logic of environmental vehicles is too simple, the behavioral intentions are significantly different from those of real human drivers, and the simulation feedback data is inconsistent with real road data, making it difficult to directly implement the model. The present invention proposes a human-in-the-loop autonomous driving training and testing system based on digital twins to achieve real-time interaction between autonomous driving vehicles and simulated driving environments.

[0024] The technical solution adopted in the present invention is: A human-in-the-loop autonomous driving training and testing method based on digital twins is provided, comprising the following steps: Obtain driving data from the in-loop driver based on the digital twin and input it into a pre-built in-loop driver social value orientation identification model. The output represents the weight of the in-loop driver's preference between safety and efficiency. This weight is then converted into a perspective for evaluating the in-loop driver's social value orientation. The driver's ego vehicle in the ring is regarded as the leader, and the pre-built virtual environment vehicles are regarded as followers. The distance between the vehicles is used to determine whether the environment vehicles are in a game with the ego vehicle. When a game occurs, the behavior strategy of the surrounding vehicles is output based on a pre-built game model. This game model specifically constructs a comprehensive game cost function for the surrounding vehicles within a certain range that is related to the ego vehicle, based on the social value orientation of the in-loop driver of the ego vehicle. This comprehensive game cost function is constructed based on the horizontal and vertical behaviors of the leader and follower. The comprehensive game cost function is solved, and the behavior strategy of the surrounding vehicle with the lowest cost is regarded as the optimal solution. The trajectory of the environmental vehicle is planned according to the optimal solution, and the control quantity of the environmental vehicle is output through the vehicle controller.

[0025] Following the above technical solution, the specific construction process of the in-loop driver social value orientation identification model is as follows: Conduct multiple rounds of experiments using the existing digital twin hardware platform to collect driving trajectories and generate trajectory sets; The objective function of this trajectory set is constructed based on the human driving reward function model and the probability distribution of human driving behavior based on inverse reinforcement learning. The solution of the objective function is the reward parameter of the human driving reward function model, which corresponds to the driver's preference weight for driving efficiency and social safety. Solve the objective function and obtain the two preference selection weights under the optimal reward parameters.

[0026] Following the above technical solution, the weight ratio is specifically mapped to the angle value of the social value orientation of the driver in the loop.

[0027] Following the above technical solution, the comprehensive cost function of the game includes a safety cost function and an efficiency cost function. The safety cost is measured by the speed difference and distance with the nearest vehicle in the target lane, as well as the additional risk brought by the lateral offset and heading deviation; the efficiency cost is measured by the inverse of the square difference between the current speed and the maximum speed.

[0028] Following the above technical solution, when solving the comprehensive cost function of the game, solving the optimal behavior strategy is regarded as a two-level optimization problem, the decision-making behavior of the ego vehicle is regarded as the optimal solution of the inner-level game, and the decision-making behavior of the environmental vehicle is discretized within the range of the vehicle motion dynamics constraints. For each pair of discretized actions, the comprehensive cost of the game of the environmental vehicle is calculated, and the behavior strategy that minimizes the comprehensive cost of the game is selected as the final game result behavior of the environmental vehicle, where the decision-making behavior includes the switching behavior and longitudinal speed of the corresponding vehicle.

[0029] Following the above technical solution, the environmental vehicle trajectory is specifically planned based on the optimal solution using a quintic polynomial.

[0030] Following the above technical solution, the vehicle controller is an MPC or LQR controller.

[0031] Following the above technical solution, if it is determined based on the distance between the vehicles that the surrounding vehicle and the self-vehicle will not engage in a game, the surrounding vehicle is controlled based on the pre-built lateral and longitudinal driver models.

[0032] The present invention also provides a human-in-the-loop autonomous driving training and testing system based on digital twins, including a digital twin hardware platform and an environmental vehicle controller based on social games. The digital twin hardware platform is used to realize "human-vehicle-environment" closed-loop interaction and autonomous driving training, specifically by continuously collecting physical vehicle status and external environment data through sensors to build an interactive environment for the driver and the virtual scene; the environmental vehicle controller based on social games is used to generate and control the environmental vehicle in the virtual scene according to the method described in the above technical solution, so that the interaction process between the environmental vehicle and the driver is close to the interaction process between the autonomous driving vehicle and other vehicles in the environment in the real world.

[0033] The present invention also provides a computer storage medium, which stores a computer program that can be executed by a processor, and the computer program executes the human-in-the-loop autonomous driving training and testing method based on digital twins described in the above technical solution.

[0034] The beneficial effects of the present invention are as follows: based on the driving data of the in-loop driver based on the digital twin obtained online, the present invention obtains the real-time preference selection weight of the in-loop driver between safety and efficiency for evaluating the current in-loop driver's social value orientation, and then constructs a game model of the surrounding virtual environment vehicles based on the driver's social value orientation. When a game occurs between the self-vehicle and the environment vehicle, the game model is used to drive the environment vehicle to make an interactive behavior strategy close to that of real humans, and then the trajectory of the environment vehicle is planned according to the behavior strategy, and the control amount of the environment vehicle is output through the vehicle controller, so that the interaction process between these environment vehicles and the driver is closer to the interaction process between the autonomous driving vehicle and other vehicles in the environment in the real world, significantly improving the representativeness of the training samples and the generalization ability of the model in real scenarios, thereby efficiently completing the human-in-the-loop autonomous driving training and testing.

[0035] The digital twin-based human-in-the-loop autonomous driving training and testing system of the present invention includes a digital twin hardware platform and an environmental vehicle controller based on social game, realizing the "human-vehicle-environment" closed-loop interaction and autonomous driving training. Through the deep collaboration of hardware and software platforms and control algorithms, it effectively eliminates the differences between the simulation environment and the real road in data distribution, decision-making logic and human-vehicle interaction, providing a realistic and controllable training and verification environment for high-level autonomous driving systems, greatly improving the safety and adaptability of the model in actual scenarios.

[0036] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0038] Figure 1 This is a flowchart of a human-in-the-loop autonomous driving training and testing method based on digital twins according to an embodiment of the present invention; Figure 2 This is an architecture diagram of a human-in-the-loop autonomous driving training and testing system based on digital twins according to an embodiment of the present invention; Figure 3 This is a basic digital twin system architecture diagram of an embodiment of the present invention; Figure 4 This is a diagram of a human-in-the-loop driving simulation system for digital twins according to an embodiment of the present invention; Figure 5 This is a flowchart of calculating social value orientation according to an embodiment of the present invention; Figure 6 This is a social game flow chart according to an embodiment of the present invention; Figure 7 This is a flow chart of environmental vehicle control based on social game according to an embodiment of the present invention; Figure 8 This is a technical architecture diagram of an environmental vehicle controller based on social gaming according to an embodiment of the present invention; Figure 9 This is a flow chart of calculating the environmental vehicle control amount according to an embodiment of the present invention. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0040] It should be noted that the illustrations provided in the embodiments of the present invention are only schematic illustrations of the basic concept of the present invention. Therefore, the drawings only show components related to the present invention and are not drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component can be changed at will, and the component layout type may also be more complicated.

[0041] In the present invention, it should also be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer" and the like are used to indicate positions or locations based on those shown in the accompanying drawings. These terms are intended solely to facilitate the description of the present application and to simplify the description. They are not intended to indicate or imply that the devices or components referred to must have a specific orientation, be constructed, or operate in a specific orientation. Therefore, they should not be construed as limiting the present application. Furthermore, the terms "first" and "second" are used solely for descriptive and distinguishing purposes and should not be construed as indicating or implying relative importance.

[0042] In addition, it should be noted that the features of the various embodiments of the present invention may be combined or coupled in part or in whole, and, as will be appreciated by those skilled in the art, may interact and operate in different ways. Each embodiment may be implemented independently of one another or in an associated relationship.

[0043] The present invention mainly constructs a closed loop of "human-vehicle-environment", utilizes the online estimation of the driver's social value orientation (SVO) and game, drives the environmental vehicle to make interactive decisions close to those of real humans, and through the collaboration of hardware and software platforms and control algorithms, greatly narrows the distribution gap between the simulation environment and the real road, significantly improving the representativeness of the training samples and the generalization ability of the model in real scenarios.

[0044] like Figure 1 As shown, the human-in-the-loop autonomous driving training and testing method based on digital twins in an embodiment of the present invention includes the following steps: S1. Obtain driving data of the in-loop driver based on the digital twin and input it into a pre-built in-loop driver social value orientation identification model. Output represents the weight of the in-loop driver's preference between safety and efficiency. This weight is then converted into a perspective for evaluating the in-loop driver's social value orientation. S2: The driver's ego vehicle in the ring is regarded as the leader, and the virtual environment vehicles are pre-built as followers. The distance between the vehicles is used to determine whether the environment vehicles are in a game with the ego vehicle; S3. When a game occurs, output the behavior strategy of the surrounding vehicles based on a pre-constructed game model. The game model specifically constructs a comprehensive game cost function for the surrounding vehicles within a certain range that is related to the ego vehicle, based on the social value orientation of the in-loop driver of the ego vehicle. The comprehensive game cost function is constructed based on the horizontal and vertical behaviors of the leader and follower. Solve the comprehensive game cost function and take the behavior strategy of the surrounding vehicle with the lowest cost as the optimal solution. S4. Plan the trajectory of the environmental vehicle according to the optimal solution, and output the control amount of the environmental vehicle through the vehicle controller.

[0045] In order to implement the above method, the present invention proposes a human-in-the-loop autonomous driving training and testing system based on digital twins, which is used to realize real-time interaction between autonomous driving vehicles and simulated driving environments, collect training data, and further evaluate model performance. In order to solve the problems of high cost of collecting data on real roads, difficulty in obtaining dangerous or extreme scenes and safety risks, as well as the problems that the decision-making logic of environmental vehicles in existing virtual simulations is too simple, the behavioral intentions are far from those of real human drivers, and the simulation feedback data is inconsistent with the real road data, making it difficult to directly implement the model. Figure 2 As shown in the figure, an autonomous driving training and testing system is designed, which includes a digital twin hardware and software platform and an environmental vehicle controller based on social game.

[0046] The digital twin hardware and software platform primarily serves to provide a foundational hardware and software environment for the autonomous driving algorithm training process proposed in this invention, enabling closed-loop interaction between human, vehicle, and environment, as well as autonomous driving training. Sensors continuously collect data on the physical vehicle's status and external environment, creating an interactive environment between the driver and the virtual scene. Specifically, the digital twin hardware and software platform comprises two components: a hardware system and a software system. The hardware system includes a development platform for intelligent connected, wire-controlled passenger vehicles, a wireless bridge, a server, a driving simulator (including a three-degree-of-freedom motion platform and a real-car cockpit), and a 3D surround screen. The software system primarily comprises a virtual scene model, a virtual vehicle, and an environmental vehicle controller based on social game theory.

[0047] The present invention builds a basic digital twin system based on existing hardware. Figure 3 As shown in the figure, the system composition is presented from the abstract architecture level (while Figure 2 From the hardware implementation level, the two jointly explain the digital twin training and testing platform constructed by the invention from different levels). The digital twin system abstractly defines the three core collaborative subsystems that support the entire training process from a higher system functional module level, including the physical entity data acquisition platform, the human-in-the-loop driving simulation system and the data interaction and communication platform to support the "human-vehicle-environment" closed-loop interaction and autonomous driving training. The physical entity data acquisition platform is mainly composed of Figure 2 The intelligent connected wire-controlled passenger car and its sensors (cameras, lidar, etc.) are responsible for collecting real-world point cloud, video, posture and vehicle status data. The human-in-the-loop driving simulation system mainly consists of Figure 2 The driving simulator, 3D ring screen, server and software system (virtual scene model, virtual vehicle, environmental vehicle controller) in the system together provide a simulation environment for the driver to interact with the high-fidelity virtual environment. The data interaction communication platform is mainly composed of Figure 2 The wireless bridges (including Bridge 1 and Bridge 2) in the system are responsible for establishing a reliable, real-time, two-way data communication channel between the real vehicle, the driving simulator (people) and the virtual environment, and transmitting perception data and control instructions.

[0048] Figure 2 The specific hardware devices and their associated software modules (such as the environmental vehicle controller) listed in Figure 3 The three abstract functional subsystems defined (physical entity data acquisition platform, human-in-the-loop simulation, and data interactive communication) are the specific implementation carriers in the physical world. Figure 3 The architecture clearly outlines the logical relationship and data flow of the core functional modules of the system. Figure 2 The two demonstrate the specific technical facilities required to support these abstract functions. Together, they form a complete closed-loop interaction system for "human-vehicle-environment," providing a solid hardware and software foundation for the subsequent training, testing, and verification of autonomous driving algorithms.

[0049] Specifically, firstly, the intelligent connected wire-controlled passenger car platform is used as a physical entity data collection platform to provide real vehicle entity and environmental perception capabilities, with a built-in wire-controlled domain controller and chassis power system, and continuously collect vehicle status and external environment data through sensors such as GPS and IMU. Secondly, the human-in-the-loop driving simulation system consists of a driving simulator, a 3D ring screen and a local server, which builds a high-fidelity interactive environment for the driver and the virtual scene. Its detailed architecture is as follows: Figure 4As shown, the driving simulator consists of two integrated components: a three-degree-of-freedom motion platform and a real-car cockpit. The real-car cockpit, equipped with a force-feedback steering wheel, accelerator pedal, and gear controller, connects to a hardware-in-the-loop platform (e.g., dSPACE) via a serial port (e.g., DB9) to output real-world CAN control signals. The three-degree-of-freedom motion platform receives CAN messages and uses servo motors to reproduce the vehicle's posture. A 180-degree surround display system (3D surround) coupled with a high-performance server renders and projects the virtual driving scene onto the instrument panel and rearview mirror displays in real time. Finally, a wireless bridge serves as the data exchange communication platform, enabling bidirectional data transmission between the virtual world, cockpit, and remote real-car over an IP / TCP channel. Ultimately, control commands are issued using the CAN protocol, ensuring that the virtual vehicle's state is highly consistent with the real-world vehicle's motion. The seamless integration of these components provides a reliable physical foundation for subsequent autonomous driving algorithm training and validation.

[0050] As can be seen, this invention, based on the concept of digital twins, organically integrates a real-world, controlled-by-wire passenger vehicle, a three-degree-of-freedom cockpit, a surround-screen display, a motion platform, and a high-performance simulation environment to form an integrated hardware-software system for closed-loop interaction between "human-vehicle-environment." This platform supports both human-in-the-loop participatory training and seamlessly integrates non-human-in-the-loop (pure simulation) agent training. This significantly improves the flexibility and efficiency of training deployment, reduces the safety risks and costs of field testing, and provides reliable guarantees for the rapid iteration and performance evaluation of autonomous driving algorithms.

[0051] The virtual scene model and virtual vehicle in the software system are used to provide the basic space and controlled objects. The environmental vehicle controller based on social game is used to control the environmental vehicles that interact with the driver, so that the interaction process between these environmental vehicles and the driver is closer to the interaction process between the autonomous driving vehicle and other vehicles in the environment in the real world. The present invention improves the implementation of the algorithm by narrowing the distribution gap between simulation and reality. Through the deep collaboration of software and hardware platforms and control algorithms, the present invention effectively eliminates the differences between the simulation environment and the real road in data distribution, decision logic and human-vehicle interaction, providing a realistic and controllable training and verification environment for high-level autonomous driving systems, greatly improving the safety and adaptability of the model in actual scenarios.

[0052] The software system workflow mainly includes "in-loop driver social value orientation identification" and "environmental vehicle control based on social game".

[0053] (1) Identification of social value orientation of in-circuit drivers: First, a value is defined to describe the driving style of human drivers, namely social value orientation SVO (Social Value Orientation), which is used from an angle Represents the driver's weight preference between "public interest" (safety) and "personal interest" (efficiency). The value range and meaning of: : Extremely altruistic, giving full priority to public safety; : Pro-social, focusing on safety while taking efficiency into consideration; : Self-interest, focusing only on one's own efficiency; : Competition, mainly based on efficiency, with a slight confrontational nature.

[0054] Secondly, we use the "efficiency / safety" feature weight ratio mapping learned through inverse reinforcement learning to obtain an angle quantitative index that reflects the social preferences of the in-loop driver during the autonomous driving training process. We collect the driving trajectory of the in-loop driver, use the inverse reinforcement learning method to solve the driver's reward function, and calculate the SVO based on the weights of each part of the reward function. value.

[0055] like Figure 5 As shown in Figure 2, the specific steps for calculating the social value orientation of the in-circuit driver are as follows: 1) Human driver behavior data collection Before training the autonomous driving algorithm, an initial driving environment was constructed to collect driving data from human drivers. In this driving environment, the vehicle was controlled by a conventional lateral and longitudinal driver model (IDM+MOBIL). Human drivers conducted 20 rounds of experiments using the existing digital twin hardware platform, collecting 20 driving trajectories and forming a trajectory set. Each trajectory Contains status (speed, position, etc.) and action .

[0056] 2) Build a human driving reward function model Based on the existing human driving trajectory collection , extract the features to define the reward function of inverse reinforcement learning, and define two features, which represent driving efficiency. and social security .

[0057]

[0058]

[0059] The reward function is defined as:

[0060] Respectively represent and In the reward function The weight parameter in .

[0061] 3) Constructing a probability distribution of human driving behavior Use the maximum entropy inverse reinforcement learning method to estimate the parameters constructed in the previous step . Assume that human drivers make random choices (but prefer high rewards) from a series of possible trajectories based on the rewards of each trajectory. Since the real-world trajectory space is continuous and high-dimensional, it is not feasible to directly use human driving trajectories to solve the weight of the reward function. The present invention uses the maximum entropy inverse reinforcement learning method to solve it. Under the maximum entropy inverse reinforcement learning framework, the process of human selection of driving behavior follows the Boltzmann distribution. The higher the reward, the greater the probability of being selected. The probability distribution model is modeled as:

[0062] This formula represents the given Next, the trajectory The probability of being selected by a human driver is approximately equal to the ratio of its exponential return to the sum of the exponential returns of all candidate trajectories. Refers to all possible driving trajectories of human drivers. Since only a limited number of real human driving data are collected, a random generation method is used. According to the constraints of the vehicle kinematic model and the same trajectory length, based on the initial state Generate a sequence of trajectories (taken from the first moment of each example trajectory in the human example) .molecular Proportional to the trajectory In the current reward parameters Total return under . Denominator It is the normalization of the weights of all candidate trajectories (or sampled trajectories), approximating the originally difficult-to-calculate partition function (used for probability distribution normalization).

[0063] 4) Solving the reward function parameters

[0064] is the set of observed “example trajectories” generated by real human driving. We want to adjust the reward parameter , so that the probability of the inverse reinforcement learning model generating these trajectories is as high as possible. Log-likelihood This is used to measure the model's ability to explain example trajectories. A larger value indicates that the model is more able to reproduce human behavior.

[0065] The core of maximum entropy inverse reinforcement learning is to find a set of reward parameters that makes human behavior the most "attractive" among all possible behaviors under this reward. By embedding the probability distribution model into the log-likelihood as the optimization target, the gradient ascent is used to adjust the , until the model's trajectory distribution is consistent with the statistical characteristics of human example trajectories (acceleration, distance, and speed distribution). The log-likelihood objective function is expressed as follows:

[0066] The gradient solution formula based on the log-likelihood objective function is:

[0067] Solved by gradient ascent , through the objective function and the regularization coefficient , according to the learning rate Iterate to get the optimal : The final solution is the expert example Parameters under .

[0068] 5) SVO angle mapping The weight ratio is mapped to the SVO angle. The efficiency characteristic weight after convergence is obtained by the third step. and security feature weights , solving the social value orientation perspective The calculation formula is as follows:

[0069]

[0070] The threshold Used to distinguish "competitive type" from other types.

[0071] Finally, the relative weight of efficiency and safety is converted into an intuitive SVO perspective , used to evaluate the social value orientation of in-circuit drivers.

[0072] (2) Environmental vehicle control based on social game: like Figure 6-8 As shown, the ego vehicle (the one with the driver in the loop) is first considered the leader, and the surrounding vehicles are considered followers. Based on the distance between the vehicles, it is determined whether the surrounding vehicles are engaging in a game with the ego vehicle. In this invention, the game interaction is triggered when the longitudinal distance between the surrounding vehicles and the ego vehicle is less than 3 meters. Otherwise, the conventional lateral and longitudinal driver model is used to control the surrounding vehicles.

[0073] Secondly, the horizontal and vertical behaviors of leaders and followers can be solved based on Stackelberg game. Specifically, two cost functions are constructed, namely, security cost function and efficiency cost function .

[0074]

[0075] This value is used to represent lane-changing behavior, where -1 indicates a left lane change, 0 indicates staying in the current lane, and 1 indicates a right lane change. When not changing lanes, only the safety cost of the current lane is calculated; when changing lanes, the safety cost of the target lane is evaluated instead.

[0076]

[0077] Indicates the longitudinal speed difference between your vehicle and the vehicle in front. The greater the speed difference, the higher the risk of rear-end collision. Indicates the net distance between two vehicles. The smaller the distance, the higher the risk. Used to balance the safety impacts of "speed approach" and "too close distance".

[0078]

[0079] It is used to measure the speed difference and distance with the nearest vehicle in the target lane. The formula is consistent. A measure of the additional risk caused by lateral deviation and heading deviation.

[0080]

[0081] Efficiency costs At current speed Distance maximum speed Measured by the inverse of the square difference, the closer to the maximum speed, the lower the efficiency cost. Represents the weight coefficient, reflecting the The importance a vehicle places on speed and efficiency; The larger it is, the stronger the penalty for deviations from the maximum speed. represents a small positive number added to the denominator to prevent it from approaching zero. Solving the optimal decision can be viewed as a two-level optimization problem, typically solved using the "backward induction" approach.

[0082] The specific steps are as follows: 1) The decision-making behavior of the vehicle As the optimal solution of the inner game. Indicates the longitudinal velocity of the ego vehicle (leader).

[0083] 2) The decision-making behavior of the environment vehicle Discretize within the range of vehicle motion dynamics constraints, and for each pair of discretized actions, calculate the comprehensive game cost of the vehicle in the environment , , choose to make the cost Minimal behavior The game result behavior of the final environment vehicle is output as the decision-making behavior of the environment vehicle.

[0084] (3) Environmental vehicle control based on quintic polynomial programming and MPC: like Figure 9 As shown in the figure, the calculation of the environmental vehicle control quantity is mainly based on the environmental vehicle behavior obtained by the game , a quintic polynomial is used to plan the vehicle trajectory in the environment, and an MPC controller is used to output the control variable.

[0085] First, construct the quintic polynomial as:

[0086] Where, is the constant term of the polynomial, which represents the vertical coordinate value of the curve when the horizontal coordinate is 0. are the polynomial parameters to be determined, is the trajectory curve function of the environment vehicle. Let the starting point and end point of the environment vehicle trajectory be 、 , the vehicle's posture at the start and end time should be consistent with The axes remain parallel, i.e. the heading angle and the steering angle of the front wheels are both 0 at these two points.

[0087] Secondly, based on these conditions, a solution equation based on a quintic polynomial is established to obtain the motion trajectory of the environmental vehicle.

[0088]

[0089] Finally, the obtained motion trajectory is compared with the desired velocity The input to the MPC controller is used to obtain the lateral and longitudinal control variables of the surrounding vehicle. Since trajectory planning and MPC controller construction are conventional operations and are not the focus of this invention, they will not be described in detail in this invention.

[0090] As can be seen, the proposed social game-driven adaptive control method for environmental vehicles addresses the problem of existing simulations, where environmental vehicle behavior is monotonous and lacks responsiveness to human driving styles. This method introduces online estimation of social value orientation (SVO) and incorporates safety and efficiency costs into the game framework to dynamically adjust the environmental vehicle's strategy. This method captures the social preferences and interaction details of driver behavior in real time, making the generated training data more realistic in terms of diversity and uncertainty, significantly enhancing the generalization and robustness of autonomous driving models.

[0091] This application also provides a computer-readable storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a disk, an optical disk, a server, an App store, etc., on which a computer program is stored. When the program is executed by a processor, the computer-readable storage medium of this embodiment implements the human-in-the-loop autonomous driving training and testing method based on digital twins of the method embodiment.

[0092] It should be pointed out that, according to the needs of implementation, the various steps / components described in this application can be split into more steps / components, or two or more steps / components or partial operations of steps / components can be combined into new steps / components to achieve the purpose of the present invention.

[0093] The size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0094] It should be understood that those skilled in the art can make improvements or changes based on the above description, and all such improvements and changes should fall within the scope of protection of the appended claims of the present invention.

Claims

1. A human-in-the-loop autonomous driving training and testing method based on digital twins, characterized in that: The following steps are involved: Obtain driving data from the in-loop driver based on the digital twin and input it into a pre-built in-loop driver social value orientation identification model. The output represents the weight of the in-loop driver's preference between safety and efficiency. This weight is then converted into a perspective for evaluating the in-loop driver's social value orientation. The driver's ego vehicle in the ring is regarded as the leader, and the pre-built virtual environment vehicles are regarded as followers. The distance between the vehicles is used to determine whether the environment vehicles are in a game with the ego vehicle. When a game occurs, the behavior strategy of the surrounding vehicles is output based on a pre-built game model. This game model specifically constructs a comprehensive game cost function for the surrounding vehicles within a certain range that is related to the ego vehicle, based on the social value orientation of the in-loop driver of the ego vehicle. This comprehensive game cost function is constructed based on the horizontal and vertical behaviors of the leader and follower. The comprehensive game cost function is solved, and the behavior strategy of the surrounding vehicle with the lowest cost is regarded as the optimal solution. The trajectory of the environmental vehicle is planned according to the optimal solution, and the control quantity of the environmental vehicle is output through the vehicle controller.

2. The human-in-the-loop autonomous driving training and testing method based on digital twins according to claim 1 is characterized in that: The specific construction process of the in-loop driver social value orientation identification model is as follows: Conduct multiple rounds of experiments using the existing digital twin hardware platform to collect driving trajectories and generate trajectory sets; The objective function of this trajectory set is constructed based on the human driving reward function model and the probability distribution of human driving behavior based on inverse reinforcement learning. The solution of the objective function is the reward parameter of the human driving reward function model, which corresponds to the driver's preference weight for driving efficiency and social safety. Solve the objective function and obtain the two preference selection weights under the optimal reward parameters.

3. The human-in-the-loop autonomous driving training and testing method based on digital twins according to claim 2 is characterized in that: Specifically, the weight ratio is mapped to the angle value of the social value orientation of the driver in the loop.

4. The human-in-the-loop autonomous driving training and testing method based on digital twins according to claim 1 is characterized in that: The comprehensive cost function of the game includes a safety cost function and an efficiency cost function. The safety cost is measured by the speed difference and distance with the nearest vehicle in the target lane, as well as the additional risk brought by the lateral offset and heading deviation; the efficiency cost is measured by the inverse of the square difference between the current speed and the maximum speed.

5. The human-in-the-loop autonomous driving training and testing method based on digital twins according to claim 1 is characterized in that: When solving the comprehensive cost function of the game, solving the optimal behavior strategy is regarded as a two-level optimization problem. The decision-making behavior of the ego vehicle is regarded as the optimal solution of the inner-level game. The decision-making behavior of the environmental vehicle is discretized within the range of the vehicle motion dynamics constraints. For each pair of discretized actions, the comprehensive cost of the game of the environmental vehicle is calculated, and the behavior strategy that minimizes the comprehensive cost of the game is selected as the final game result behavior of the environmental vehicle. The decision-making behavior includes the switching behavior and longitudinal speed of the corresponding vehicle.

6. The human-in-the-loop autonomous driving training and testing method based on digital twins according to claim 1 is characterized in that: The environmental vehicle trajectory is specifically planned based on the optimal solution using a quintic polynomial.

7. The human-in-the-loop autonomous driving training and testing method based on digital twins according to claim 1 is characterized in that: The vehicle controller is an MPC or LQR controller.

8. The human-in-the-loop autonomous driving training and testing method based on digital twins according to any one of claims 1 to 7, characterized in that: If it is determined based on the distance between the vehicles that the surrounding vehicle and the ego vehicle will not engage in a game, the surrounding vehicle is controlled based on the pre-built lateral and longitudinal driver models.

9. A human-in-the-loop autonomous driving training and testing system based on digital twins, characterized by: It includes a digital twin hardware platform and an environmental vehicle controller based on social game. The digital twin hardware platform is used to realize the "human-vehicle-environment" closed-loop interaction and autonomous driving training. Specifically, the sensor continuously collects the physical vehicle status and external environment data to build an interactive environment for the driver and the virtual scene; the environmental vehicle controller based on social game is used to generate and control the environmental vehicle in the virtual scene according to the method described in any one of claims 1 to 8, so that the interaction process between the environmental vehicle and the driver is close to the interaction process between the autonomous driving vehicle and other vehicles in the environment in the real world.

10. A computer storage medium, characterized in that A computer program executable by a processor is stored therein, which executes the human-in-the-loop autonomous driving training and testing method based on digital twins as described in any one of claims 1 to 7.

Images