Systems and methods for flexible control of vehicles by different agents in autonomous driving simulation

Abstract

A method and system for controlling the behavior of an object. During a first time period, the object's behavior is controlled using a first agent, which applies a first behavior strategy to map observations about the object and its environment during the first time period to corresponding control actions. During a transition period following the first time period, control is transferred from the first agent to a second agent. During a second time period following the transition period, the object's behavior is controlled using a second agent, which applies a second behavior strategy to map observations about the object and its environment during the second time period to corresponding control actions applied to the object. During the transition period, the first agent applies the first behavior strategy to control the object, and the second agent applies the second behavior strategy to map observations about the object and its environment to corresponding control actions not applied to the object.

Description

[0001] This application claims priority to U.S. Patent Application No. 16 / 941,505, filed July 28, 2020, entitled "System and Method for Managing Flexible Control of Vehicles by Diverse Agents in Autonomous Driving Simulation," and U.S. Patent Application No. 16 / 989,776, filed August 10, 2020, entitled "System and Method for Managing Flexible Control of Vehicles by Diverse Agents in Autonomous Driving Simulation," which are incorporated herein by reference as if reproduced in their entirety. Technical Field

[0002] This invention relates to a control agent for robots in a simulated environment. Background Technology

[0003] Research and development (R&D) in the field of autonomous robot control heavily relies on simulation to train, test, evaluate, and validate autonomous driving software agents controlling vehicles. Generally, the more realistic and diverse the simulation, the more useful it is. Realism ensures the simulated environment mirrors the real world. Diversity in the simulation, particularly the diversity of simulated interactions between autonomous robots (e.g., self-driving vehicles under the control of a primary autonomous driving software agent under development) and other traffic participants (especially social vehicles) not under the control of the primary agent, ensures coverage of behavioral variations in the real world (i.e., variations in the behavior of self-driving and social vehicles). The key to realism and diversity is the behavior of traffic participants, especially social vehicles, in their interactions with self-driving vehicles and with each other. Simultaneously, the behavior of social vehicles needs to be composable, controllable, configurable, and automated to achieve expressive and repeatable simulations, which are crucial for the effective training, testing, evaluation, validation, and development of autonomous driving software agents. Therefore, autonomous driving simulations need to provide realistic and diverse social vehicle interaction behaviors, along with appropriate mechanisms to compose, control, configure, and automate the use of these behaviors. This, in turn, means that even though the primary goal of autonomous driving research and development is to develop a single agent capable of becoming a competent autonomous driver, autonomous driving simulations need to flexibly combine multiple different agents to help develop such a single agent.

[0004] Existing simulation systems offer limited options for independently controlling social vehicles in a simulated environment.

[0005] Therefore, there is a need for systems and methods that can be flexibly controlled by different autonomous driving software agents, which are used to control simulated vehicles in autonomous driving simulations, including social vehicles and autonomous vehicles. Summary of the Invention

[0006] This invention describes a method and system for enabling the transfer of control of an object from a first agent applying a first behavioral strategy to a second agent applying a second behavioral strategy. A behavioral strategy is a functional component that maps observations to control actions. Control of the object is transferred during a transition period that allows the second agent to be initialized to facilitate a smooth transition when transferring control of the object from the first agent to the second agent. Example embodiments can use autonomous driving software agents specifically designed for these scenarios to control objects in different ways across various scenarios. Using specialized autonomous driving software agents can reduce the computational resources (e.g., processor operations and / or memory access and capacity) required to control objects in certain applications, including simulated environments that may require simultaneous control of several autonomous driving software agents.

[0007] In at least one example aspect, the present invention describes a computer-implemented method for controlling the behavior of an object, comprising: controlling the behavior of the object during a first time period using a first agent, the first agent applying a first behavior strategy to map observations of the object and the environment during the first time period to corresponding control actions applied to the object; during a transition period following the first time period, transferring control of the object's behavior from the first agent to a second agent; and controlling the object's behavior during a second time period following the transition period using a second agent, the second agent applying a second behavior strategy to map observations of the object and the environment during the second time period to corresponding control actions applied to the object. During the transition period, the first agent applies the first behavior strategy to map observations of the object and the environment during the transition period to corresponding control actions applied to the object, and the second agent applies the second behavior strategy to map observations of the object and the environment during the transition period to corresponding control actions not applied to the object.

[0008] At least in the examples above, the observations mapped by the first behavioral policy and the observations mapped by the second behavioral policy each come from their respective different observation spaces.

[0009] In at least some of the above examples, during the transition period, a set of observations about the object and the environment is modified to include the observations about the object and the environment required by the second behavioral strategy.

[0010] In at least some of the above examples, the first time period corresponds to the time when the object appears in a first region defined by a first spatiotemporal boundary, the second time period corresponds to the time when the object appears in a second region defined by a second spatiotemporal boundary, the transition period corresponds to the time when the object appears in a transition region between the first region and the second region, and the method includes performing a transition when the object is detected to appear in the transition region after it appears in the first region.

[0011] In at least some of the foregoing aspects, the method further includes, during a further transition period following the second time period, transferring behavioral control of the object from the second agent to the first agent. During this further transition period, the second agent applies the second behavioral strategy to map observations of the object and the environment during the second transition period to corresponding control actions applied to the object, and the first agent applies the first behavioral strategy to map observations of the object and the environment during the further transition period to corresponding control actions not applied to the object.

[0012] In at least some of the above examples, the method is applied during simulation execution, the object is a simulated object, and the observations about the object and the environment are simulation observations about the object and the environment.

[0013] In at least some of the above examples, the object is a simulated social vehicle operating in a simulated environment, which also includes a simulated autonomous vehicle controlled by a corresponding autonomous vehicle agent throughout the first time period, the transition period, and the second time period, the corresponding autonomous vehicle agent applying autonomous vehicle behavior policies to map ongoing observations about the autonomous vehicle and the environment to corresponding autonomous vehicle control actions applied to the autonomous vehicle.

[0014] In at least some of the above examples, the second region and the transition region are fixed at a virtual position that moves together with the virtual position of the simulated vehicle within the simulated environment.

[0015] In at least some of the examples above, the second region and the transition region are fixed at virtual locations that are stationary relative to virtual physical locations within the simulated environment.

[0016] In at least some of the examples above, the computational intensity of the first behavioral strategy is lower than that of the second behavioral strategy.

[0017] In at least some of the above examples, the second behavioral strategy is used to map observations about the object and the environment from an observation space that is richer than the observation space from which the first behavioral strategy maps observations about the object and the environment.

[0018] In at least some of the above examples, the second behavioral strategy is used to map observations about the object and the environment to control actions from an action space that is richer than the action space from which the first behavioral strategy maps observations about the object and the environment to control actions.

[0019] According to another example aspect, there is a computer system including a processor and a memory coupled to the processor, the memory storing instructions that, when executed by the processor, configure the computer system to perform the method described in any one of the preceding aspects.

[0020] According to another example aspect, it is a computer program product, including a computer medium storing instructions for configuring a computer system to perform the methods described in any one of the preceding aspects. Attached Figure Description

[0021] To gain a more complete understanding of the exemplary embodiments and their advantages, reference is now made to the following detailed description taken in conjunction with the accompanying drawings.

[0022] Figure 1 This is a schematic diagram illustrating a simulator system and an example simulation, based on an example embodiment.

[0023] Figure 2 Shown by Figure 1 The operations performed by the bubble manager of the simulator system.

[0024] Figure 3 A graphical example of a region-based transition performed by the bubble manager is shown.

[0025] Figure 4 Draw a state diagram of a finite state machine for using logic in transition regions, based on the example.

[0026] Figure 5 Depict a simulated scenario using a region based on a static map.

[0027] Figure 6 Describe a simulated scenario in which bubbles are associated with and move together with the vehicle.

[0028] Figure 7 A simulated scenario depicting a conditional bubble with time boundaries is described.

[0029] Figure 8 Showing what can be used to implement Figure 1 A block diagram of a computer system with the characteristics of a simulator system.

[0030] Similar reference numerals can be used to denote similar components in different accompanying drawings. Detailed Implementation

[0031] Figure 1 This is a schematic diagram of simulator system 100 and a representative simulation 120 generated by simulator system 100 during simulation operation. In an example embodiment, simulator system 100 is used to train an artificial intelligence (AI) controller to control a vehicle. As used in this invention, a vehicle refers to a controllable moving object, and may include automobiles, trucks, buses, marine vessels, airborne vehicles, farm equipment, military equipment, warehouse equipment, construction equipment, and other robots.

[0032] The AI ​​controller of a vehicle may include one or more trained autonomous driving software agents (hereinafter generally referred to as agents, collectively as agents). An agent is a computer-implemented program or module that applies a learned behavioral model (i.e., a learned behavioral policy) to map observations about the vehicle and its environment to corresponding control actions (i.e., actions used to control the vehicle). In real-world applications, the target vehicle includes a set of sensors for sensing data, which collectively provide observations about the vehicle and its environment, and a set of controllers for controlling the vehicle actuators in response to corresponding actions generated by the agents. Observations about the vehicle and its environment include sensed information about the vehicle's operational characteristics (e.g., the state of the vehicle actuators, vehicle attitude, vehicle linear and angular velocities and accelerations), and sensed information about the environment in which the vehicle is operating (e.g., images derived from LIDAR, imagery, and / or radar units).

[0033] The simulator system 100 is used to generate a real-world simulation 120 (i.e., a simulation of a real-world environment), which is used to train the agent in a series of simulated scenarios before the trained agent is transferred to the AI ​​controller of a real-world vehicle that will operate in the real-world environment.

[0034] In this regard, simulator system 100 is used to run simulation 120, which includes a simulated autonomous vehicle 122 and one or more simulated social vehicles 124 (i.e., other vehicles in the operating environment of autonomous vehicle 122). In an example embodiment, simulator system 100 includes a vehicle agent 102 for controlling the simulated autonomous vehicle 122. The simulated autonomous vehicle 122 is the primary focus in the simulation, and the vehicle agent 102 is trained to learn vehicle agent behavior policies. The trained vehicle agent 122 can be transferred to the AI ​​controller of a real-world autonomous vehicle for controlling the real-world vehicle in a real-world environment. The vehicle agent 102 receives simulation observations about the state of the simulated autonomous vehicle 122 and maps those observations to the autonomous vehicle control actions that the simulated autonomous vehicle 122 will perform in the simulation environment. This process is repeated in multiple simulation time steps during simulation execution. Simulated social vehicles 124 are provided for interacting with the simulated autonomous vehicle 122 in simulation 120. Simulated social vehicles 124 are provided for simulating the behavior of real-world social vehicles that will interact with the real-world autonomous vehicle.

[0035] In the example embodiment, simulator system 100 is used to implement an instance of social agent 104, which applies corresponding behavioral policies to control the behavior of one or more social vehicles 124, enabling them to operate independently of the vehicle 122. Simulator system 100 is used to activate the instance of social agent 104 that controls the simulated behavior of the social vehicles 124.

[0036] In some simulation environments, one possible approach is to use a single social agent to manage all behaviors of all social vehicles in all scenarios. However, such an approach may face challenges in terms of computational requirements and the inability to provide diverse experiences. Such a universal agent would require significant computational resources, such as processor time and memory. For the computational scalability of the simulation, simulating social vehicle behavior and interactions with the highest fidelity in all scenarios may be undesirable and impractical.

[0037] Therefore, in the example embodiment, social agent 104 includes different types of social agents (e.g., social agents 104-A to 104-n), which can each apply different behavioral strategies (BP 106) to control the behavior of social vehicles (i.e., simulate the behavior of social vehicle 124). For example, social agent 104-B can apply a different behavioral strategy 106 than social agent 104-A, and so on. In at least some examples, this allows computationally efficient social agents 104 to be specialized for certain aspects of social vehicle control but unsuitable for others. For example, social agent 104-B can be a powerful, computationally intensive agent used to control social vehicle 124 where fine-grained interactions are important, such as handling unprotected left turns, busy intersections, or ramp merging. If intersection fidelity is not important, such as in constant-speed lane following situations, a much simpler, less computationally intensive social agent 104-A can be used.

[0038] In example embodiments, different social agents 104 may have different observation spaces and different action spaces. For example, one social agent 104 may be used to receive analog image data, while another social agent 104 may be used to receive analog radar data. In some examples, the action space is a set of control actions that can be obtained based on observations, and the observation space is a set of observations about the vehicle and the environment.

[0039] The use of multiple social agents 104 can also realize different ranges of social vehicle behavior, thereby enabling the presentation of a wide range of training, testing, verification, and evaluation scenarios for the autonomous vehicle agent 102. The diversity of social vehicle behavior can provide a realistic simulation of the different driving styles and abilities of human drivers that complicate real-world interactions on roads.

[0040] In the example embodiment, when social vehicle 124 moves from a simulation experience requiring one level of control to one requiring another, control of social vehicle 122 can transition from one social agent 104-A to another social agent 104-B during the simulation run of simulation 120. Given a specific operational scenario, a particular social agent 104 can be associated with a particular vehicle 124 to achieve a specific behavior. Agent-vehicle-scenario-behavior matching can be modified according to the requirements of the simulation scenario. Therefore, as will be described below, the example embodiment addresses dynamically changing and managing agent-vehicle associations during simulation run.

[0041] In an example embodiment, the simulator system 100 can use a heterogeneous computing configuration to implement social agents 104 that apply corresponding behavioral strategies. Various social agents 104 can be based on scripted behavioral strategies 106, model predictive control or similar classical methods, or be data-driven and trained through imitation learning or reinforcement learning.

[0042] At any given time, a single instance of social agent 104 can control a single social vehicle 122, or multiple social vehicles 122 can be controlled together as a spatial or logical group in batch mode. Different types of social agents 104 may be specifically designed for a particular scenario or task, such as highway merging, lane following, or handling stop signs, but may not be suitable for other scenarios. As previously suggested, different types of social agents 104 may employ different observation and action spaces. Some social agents 104 may be used to operate in observation and action spaces that are richer than those suitable for other social agents 104 (e.g., with a greater number of possible observations or actions, or higher-dimensional observations or actions).

[0043] In some example embodiments, at specific points in space and time, or when specific conditions are met during simulation operation, control of a subset of social vehicles 124 can be transferred from one social agent 104-A to another social agent 104-B, so as to use the most appropriate type of social agent 104 to provide the most appropriate interaction without wasting unnecessary computational resources to simulate every detail of the interaction, which is not important. In example embodiments, the simulator system 100 is used to flexibly select from a set of different social agents 104 for controlling social vehicles 124 in the simulated environment.

[0044] In an example embodiment, simulator system 100 is used to identify constraints regarding agent-vehicle-scene-behavior matching during agent-vehicle assignment. For example, social agent 104 may be used to expect to deliver specific types of simulated observations from assigned social vehicle 124 and to expect social vehicle 124 to perform specific types of actions (in some examples, via an intermediate controller). Therefore, a particular social agent 104 may only be applicable to certain scenes and behaviors, and not to others. Thus, simulator system 100 is used to perform agent-vehicle assignment to satisfy compatibility between matching observation and control spaces. In some examples, changes to social agent 104 may also require modifications to simulated social vehicle 124. For example, simulated social vehicle 124 may need to be prepared with appropriate simulated sensors and actuators, which in turn may require time to be properly initialized. Initializing simulated social vehicle 124 may include reconfiguring simulated vehicle 124 by installing new simulated sensors (e.g., simulated LiDAR) and enhancing at least one of the simulated physical entities, such as replacing a simple rectangular frame with an articulated body with wheels having individual representations of the corresponding coefficients of friction.

[0045] Furthermore, there are limitations regarding the handover of vehicle control between social agents 104. Vehicles have mass and inertia, so a sudden change of control may be physically inappropriate. Entering social agents 104 may require multiple simulation steps to properly initialize their internal states. Entering social agents 104 may need to acquire sufficient historical observations of the simulated social vehicle 124 and its surrounding environment during the transition period to correctly predict future observations and corresponding actions. In the example embodiment, the simulator system 100 is used to ensure a smooth agent-to-agent handover of control of vehicle 124, given these constraints. In the example embodiment, a smooth handover is characterized by no unreasonable changes in the simulated physical behavior of the simulated social vehicle 124 and no inconsistent internal control states of the entering social agents 104.

[0046] Therefore, in the example embodiment, the simulator system 100 is used to implement a bubble manager 108 for managing dynamic agent-vehicle allocation. The bubble manager 108 is used to apply a "region-based transition" approach to manage the dynamic switching of control of the simulated social vehicle 124 between different social agents 104.

[0047] As used in this invention, a "bubble" defines a region in which a specific agent-vehicle assignment is maintained if the simulated social vehicle 124 appears. The boundaries of the bubble are typically spatiotemporal. In some examples, the bubble may be defined statically relative to the simulation map. In some examples, the bubble may be bound to a specific object, such as the simulated vehicle 122, and move within the simulation map together with said object. In some examples, the bubble may also be defined by other expressible logical or functional conditions. The bubble in which the simulated social vehicle 124 is located at a given time determines the type of social agent 104 that will be used to control the simulated social vehicle 124.

[0048] In the example embodiment, the type of social agent 104 is primarily specified by the types of observations provided to social agent 104 from simulated social vehicle 124 and the types of actions expected to be performed by social agent 104 from simulated social vehicle 124. In the example embodiment, bubble manager 108 is a system for managing the definition, creation, activation, updating, application (i.e., coordination of control switching), deactivation, and destruction of bubbles during simulation operation.

[0049] Figure 2 This is a block diagram illustrating the operations performed by a bubble manager 108 according to an example embodiment during simulation design time 202, simulation loading time 204, and simulation runtime 206. In the example shown, bubbles are managed according to their specification, preparation, instantiation, and use. During simulation design time 202, when the simulation is designed by a scenario design process 208, bubbles are specified according to their spatiotemporal and conditional boundaries. Bubbles specified by the bubble design process 208 are stored as bubble specifications 210. Each bubble specification 210 also includes information about which social agents 104 are expected to control which simulated social vehicles 124 fall into a particular bubble. Bubble specifications 210 are saved to an allocated storage device 212, from which bubble specifications 210 are loaded by a simulation loader 214 as part of the simulation loading during simulation loading time 214. As a result, a bubble data structure 216 specifying bubble attributes is stored in the simulator system 100 memory. During simulation runtime 206, dynamically managed bubbles are activated according to the instantiation conditions specified in the bubble data structure 216. The agent-vehicle association data structure 220 is also stored in the memory of the simulator system 100 and dynamically updated by the bubble manager 108 to track agent-vehicle associations. It determines which social agents 104 receive which observations about the simulated social vehicles and the simulated social vehicle 124 in the simulation environment, and which simulated social vehicles 124 will receive and perform which actions from which social agents 124.

[0050] In the example embodiment, the bubble manager 108 uses a region-based transition method to manage the dynamic changes in agent-vehicle associations and the corresponding observations and action transfers and executions. In some applications, the region-based transition method described below helps simulate the smooth transfer of control of social vehicle 124 from one social agent 104 to another. The transition region of the bubble is distinguished from the agent's region. The agent's region is the part of the bubble where the expected agent-vehicle association is fully effective. The concept of the transition region and its use are as follows: Figure 3 As shown.

[0051] Figure 3 The simulation shows vehicle V (e.g., social vehicle 124) at 12 different time steps (e.g., time t1 to t2). 12 During this process, control of vehicle V is transferred between agent A (e.g., social agent 104A) and agent B (e.g., social agent 104B). Vehicle V is initially under the control of agent A and travels from the first bubble (agent A bubble) where agent A controls vehicle V to another bubble (agent B bubble) where agent B is expected to control vehicle V, then continues to travel to leave agent B bubble and return to agent A bubble where agent A is expected to control V. Agent A bubble includes "A area", and agent B bubble includes "B area". Between A area and B area are transition areas, referred to as A=>B transition area and B=>A transition area, respectively, to help manage the transfer.

[0052] When vehicle V enters the A=>B transition zone, agent A continues to control V as before. Therefore, the A=>B transition zone is also part of agent A's bubble. However, since vehicle V is in the A=>B transition zone, agent B will begin preparing to take over control. Specifically, agent B will begin receiving observations about the vehicle and its environment from vehicle V, executing its internal logic (e.g., applying its behavioral strategy (i.e., behavioral model) to the received observations), and generating actions based on the received observations.

[0053] Agent B can apply a different behavioral model (i.e., behavioral policy) than Agent A, and therefore the expected observations will differ from those received by Agent A from Vehicle V. For example, this might mean the need to instantiate and properly initialize a new set of virtual sensors, which could require up to m simulation time steps. In addition to initializing the new virtual sensors, all the internal states that Agent B relies on for properly generating actions may also require multiple time steps to be properly initialized. For instance, Agent B might need to rely on the history of n time steps to accurately estimate the environmental state or predict the future, and then generate actions based on the estimated state or predicted future. In this case, Agent B's actions are only available after n time steps. Therefore, the A=>B transition region needs to be large enough to accommodate the required max(m,n) simulation steps, which are necessary to properly prepare Agent B to control Vehicle V and to make any necessary modifications to Vehicle V.

[0054] In the example embodiment, when vehicle V is in the A=>B transition region, agent B begins to run or operate while overlapping with agent A. Actions generated by agent B are not used to control vehicle V. This allows agent B time to initialize correctly and time to bring any new sensors online. In the A=>B transition region, agent B may not yet be ready to control vehicle V. Prematurely switching control to B could cause undesirable and unnatural behavior in vehicle V (e.g., sudden changes in direction or sudden acceleration and deceleration not due to environmental realities), thus reducing the realism of the simulation. In short, when vehicle V is in the A=>B transition region, agent A continues to control vehicle V, and the action link between agent B and vehicle V is suspended or otherwise disabled.

[0055] Once vehicle V is in area B, agent B takes over control of vehicle V. At the other end of area B, when vehicle V leaves area B and returns to area A, during the B=>A transition area, agent B continues to control the vehicle, and bubble manager 108 similarly regulates: (1) the observation link from vehicle V to agents A and B, (2) the action link from agents A and B to vehicle V (the action link from agent A to vehicle V is suspended, and only agent B controls vehicle V), and (3) the corresponding agent A-specific sensor and calculation state initialization. Therefore, the B=>A transition area is also part of the agent B bubble.

[0056] In the example shown, the corresponding transition region lies between the two regions to facilitate the transfer of control in both directions. The overall logic for this transition management using transition regions can be implemented in a finite state machine, and... Figure 4 The state diagram of the finite state machine described in [the text] is summarized in [the text]. Figure 4In the state diagram shown, permissible transitions are allowed by default. In some examples, the "Turn off observations" transition may be optional. Transitions marked with * are allowed if the required observations remain on. Transitions marked with ** are prohibited by default unless the entering agent does not require initialization, is purely a passive reaction, or the resulting abrupt transition change is tolerable.

[0057] exist Figure 3 In the example, the relationships between regions A and B, and between regions A => B and B => A, are described as perfectly symmetrical. However, the duration or length of the transition region does not need to be symmetrical, provided that the required number of simulation time steps for a smooth handover are provided. In the example embodiment, (1) skipping transition regions is not allowed by default, (2) transitions from region B to region A => B or from region A to region B => A are not allowed, and (3) transitions compatible with the provisions explained above are allowed.

[0058] In some example embodiments, transitions that are prohibited in the default configuration based on skipping transition areas may be permitted if the bubble manager 108 determines that the incoming agent and its required sensors and controllers do not need to be initialized over time, are purely reactive (i.e., react only to current observations without considering recent history or possible future), or if the resulting abrupt changes are tolerable.

[0059] Although Figure 3 and Figure 4 Both descriptions state that when vehicle V is in region A, observation of agent B is disabled, and when vehicle V is in region B, observation of agent A is disabled. However, in some examples, this requirement can be optional, especially if there are sufficient computational resources to run the virtual sensors providing the observations. The reason is that multiple agents simultaneously receiving observations from the same vehicle—i.e., in the transition regions A=>B and B=>A—do not cause conflicts, unlike when multiple agents simultaneously control the same vehicle.

[0060] refer to Figure 5 In some example simulations, it is permissible for the transition regions A => B and B => A to coincide spatiotemporally. Figure 5 In the example, region A (located in) Figure 5 The outer part of square 505) is surrounded by the middle transition area, which is composed of... Figure 5(Represented by square 502). In this case, there is only one physical transition region with appropriate spatiotemporal boundaries. However, two logical transition regions can be defined by considering how a vehicle enters the physical transition region. If a vehicle enters the physical transition region by leaving region B, it is considered to have entered the B=>A logical transition region. If a vehicle enters the physical transition region by leaving region A, it is considered to have entered the A=>B logical transition region. The actual management of the handover is based on the logical regions. Therefore, when vehicle V is traveling in the A=>B logical transition region, the agent A bubble includes region A and the transition region; when vehicle V is traveling in the B=>A logical transition region, the agent B bubble includes region B and the transition region.

[0061] Unless otherwise stated, the example embodiments described below refer to logical transition regions.

[0062] Figure 5 The example shown represents a simulated scenario where bubble manager 108 performs transition management within the context of a region based on a static map. Specifically, static bubbles are introduced around specific intersections in the simulated map. Region B (rectangle 502), with its well-defined boundaries, is completely surrounded by a larger physical transition region (the region between rectangles 502 and 504), which supports two logical transition regions. Region A is defined as anywhere outside the outer boundary 504 of the physical transition region. This Region A illustrates the general default configuration using a default agent, to which control is always transferred when a vehicle leaves Region B. This default agent corresponds to an all-encompassing "background bubble."

[0063] Area B and its associated transition areas correspond to specific regions on the map. These regions can be specified by referencing map elements, such as the area around a specific intersection, or a specific lane or road segment. They can also be specified by referencing locations that can be expressed in a map coordinate system.

[0064] In the example embodiment, the simulator system 100 employing the bubble manager 108 can provide one or more of the following features:

[0065] Realism: The disclosed systems and methods enable coherent integration and smooth handover, allowing autonomous driving software agents (e.g., ML-based autonomous driving software agents trained by learning from real data through simulation or by reinforcement learning from complex interactions) to be used alternately to control social vehicles, thereby achieving more realistic interactions in the simulation.

[0066] Diversity: By allowing different autonomous driving software agents to take turns controlling social vehicles where and when they excel, simulations can be designed with greater variability and informational content (e.g., by describing length measurements) despite their differences in observation, actions, internal states, historical dependencies, and computational dependencies. Support for this diversity also opens up possibilities for crowdsourcing agents for social vehicles.

[0067] Computation: Realistic simulations (even realism in interaction, rather than sensor data) can require significant computational resources. The disclosed systems and methods allow computational resources to be flexibly used for behavioral simulations to provide the most relevant experience for training, testing, evaluating, or validating autonomous vehicle agents.

[0068] Scalability: By adaptively allocating computational resources only to the relevant parts of the simulation while keeping the rest of the simulation at a low fidelity, it is possible to enable the simulation to be scaled up to larger maps and more social vehicles.

[0069] Figure 6 Another example simulation using a vehicle-centric driving bubble is disclosed. In this embodiment, a driving bubble (e.g., a bubble including "region B") and associated transition regions are specified, which are connected to the driving simulated vehicle 122 (having a relatively stable position). As the simulated vehicle 122 moves, the bubble moves with it (maintaining its relative position).

[0070] In this example, Agent B guides simulated social vehicle 124-2 to make a U-turn. Transition area ( Figure 6 The definition and usage of the "T area" in this context are similar to those discussed above, except that there may be some restrictions on which edges a vehicle can enter the transition area from: if a vehicle crosses from above or below, it may not be counted as entering the transition area, and no handover will occur. Furthermore, probabilistic handovers are illustrated: simulated social vehicle 124-3, which enters the B-to-A transition area, does not turn around, even though it is in a position where it should. Simulated social vehicle 124-1 is shown in the A-to-B transition area.

[0071] exist Figure 6In this embodiment, the vehicle-centric driving bubble allows control of the simulated social vehicles 124 surrounding the simulated vehicle 122 to be transferred to a specific agent (e.g., a U-turn agent) to trigger desired interactive behaviors with the vehicle agent. By doing so, traffic in other locations unrelated to the simulated vehicle 122 can be simulated with less computation and a simpler behavioral model (i.e., behavioral policy) with lower interaction fidelity as the simulated vehicle 122 travels along a specific route. However, wherever the vehicle agent goes, the particularly relevant agent begins to control the simulated social vehicles 124 around it, providing the most realistic and meaningful interactions with an appropriate amount of computation and behavioral model (i.e., behavioral policy) complexity.

[0072] Figure 7 An example simulation demonstrating a conditional bubble with time boundaries is shown. In this embodiment, the bubble is anchored to an intersection but can be conditionally activated by a simulated vehicle 122 approaching the intersection. Furthermore, the associated regions of the bubble also have time boundaries (between t1 and t2, t2 and t3, t6 and t7, and t7 and t8), which follow a desired order: the transition region begins before region B and ends after region B. The handover of the simulated vehicle 122 is also shown. The transition region needs to appear first and persist for a sufficient number of time steps before region B appears; this is a spatiotemporal version using only spatial transition regions to ensure a smooth transition. This adds technical complexity, but the underlying logic is essentially similar to the spatial and driving embodiments.

[0073] Time bubbles can also have a global spatial extent, as they can cover the entire area of ​​the simulation. For example, at 7:30 AM simulation time, all vehicles, including the driver, can turn to use the "peak hour" version of their respective agents. As another example, when rain conditions are set, all vehicles can turn to use the "rainy day" version of their respective agents.

[0074] Using general conditional trigger regions with time boundaries (time on and off settings) allows bubbles and transition regions to be introduced where none exist. This enables the dynamic creation or activation of bubbles based on arbitrarily complex conditions, unlike the previous two embodiments which were limited to the mapping area or the vehicle's location. Therefore, it provides the flexibility to adaptively change the vehicle-agent association according to different needs. Furthermore, it can be used for globally adjusting the vehicle-agent association.

[0075] As described above, the bubble manager 108 enables the spatiotemporal and conditional regions (“bubbles”) to specify the desired agent-vehicle assignments and to record observations, actions, calculations, and initialization requirements for managing the dynamic changes in the assignments.

[0076] In various example embodiments, at least one of the bubbles and associated regions can be spatially specified relative to the map. In various applications, the bubbles and their associated regions can be spatiotemporal, can be purely time-based, can be conditionally activated according to the simulation state, can travel with traffic participants, can be used as global defaults, and can be prioritized.

[0077] In the example implementation, the bubble can be updated and applied in each simulation step according to the bubble specification and simulation state. In some examples, the bubble is constructed based on transition regions and surrogate regions, where the transition region is sandwiched between two surrogate regions (in terms of spatiotemporal and conditional aspects). In the example implementation, the time series of the regions in the temporal implementation follows a specified order: the transition region begins before entering the surrogate region (region B) and ends after leaving the surrogate region (region B).

[0078] The above description focuses on transitioning social vehicle-social agent associations. The systems and methods described herein can also be used to transition autonomous vehicle agent-autonomous vehicle associations in certain applications. For example, different autonomous vehicle agents 102 can be provided to control different versions of simulated autonomous vehicles 122. Each version of the simulated autonomous vehicle 112 has a different AI controller, which is trained to learn a behavioral model (i.e., behavioral policies) to ensure the realism and diversity of the experiences of these particular autonomous vehicle agents in the simulation while using a reasonable amount of computational resources.

[0079] While the above embodiments have been described in relation to vehicle control, they can be extended to non-vehicle traffic participants, particularly pedestrians, and non-vehicle traffic participants such as traffic lights. Both pedestrians and traffic lights in a simulated environment can utilize sophisticated agents to make the relevant interactions realistic. For example, pedestrian behavior can differ significantly between rural and urban areas, in large groups, or when acting alone. Similarly, traffic light policies might change at 4 PM to accommodate afternoon peak hours. Therefore, different bubble managers 108 can be applied to facilitate the transfer of control between agents of any controllable object.

[0080] Furthermore, in some examples, transition areas can be used to manage multiple agents or controllers as the physical vehicle travels on actual roads. For instance, if two different sets of agent strategies are used to control a vehicle driving on a highway and driving on city streets, bubbles and transition areas can be used to manage the handover to ensure a physically smooth and safe transition.

[0081] The idea of ​​transition regions can also be used in other fields, either in simulations or in the real world, where transitions occur between different agents with different observation types and action types.

[0082] In the example embodiments, the components, modules, systems, and agents described above may be implemented using one or more computer devices, servers, or systems, each including a combination of hardware processing circuitry and machine-readable instructions (software and / or firmware) executable on the hardware processing circuitry. The hardware processing circuitry may include any or some combination of a microprocessor, the core of a multi-core microprocessor, a microcontroller, a programmable integrated circuit, a digital signal processor, or another hardware processing circuitry.

[0083] See Figure 8 This document describes example embodiments of a computer system 2010 for implementing one or more modules, systems, and agents included in simulator system 100. System 2010 includes at least one processor 2004 controlling the overall operation of system 2010. Processor 2004 may include one or more central processing units, graphics processing units, AI processing units (e.g., neural processing units, tensor processing units, field-programmable gate arrays, or application-specific integrated circuits for performing AI calculations), and associated hardware accelerators. At least one processor 2004 is coupled to multiple components via a communication bus (not shown), which provides a communication path between the components and processor 2004. The system includes a memory 2012, which may include random access memory (RAM), read-only memory (ROM), permanent (non-volatile) memory, and permanent memory may be one or more of magnetic hard disk drives, flash erasable programmable read-only memory (EPROM) (“flash memory”), or other suitable forms of memory.

[0084] Operating system software 2040, executed by processor 2004, can be stored in permanent memory of memory 2012. Multiple application programs 202, executed by processor 2004, are also stored in permanent memory. Application programs 2042 may include software instructions for implementing systems, methods, agents, and modules, including the aforementioned simulator system 100. The software instructions of simulator system 100 can be executed by at least one processor 2004 to perform the aforementioned methods. In consideration of this invention, the coding of software for performing the aforementioned methods is within the scope of those skilled in the art.

[0085] Although the present invention describes methods and processes by steps performed in a certain order, one or more steps in the methods and processes may be omitted or modified as appropriate. Where appropriate, one or more steps may be performed in an order other than that described.

[0086] Although the invention has been described at least partially in terms of method, those skilled in the art will understand that the invention is also directed to various components for performing at least some aspects and features of the method, whether by hardware components, software, or any combination thereof. Accordingly, the technical solutions of the invention can be embodied in the form of a software product. Suitable software products can be stored in pre-recorded storage devices or other similar non-volatile or computer-readable media, including, for example, DVDs, CD-ROMs, USB flash drives, removable hard drives, or other storage media. The software product includes instructions tangibly stored thereon that enable a processing device (e.g., a personal computer, server, or network device) to perform examples of the methods disclosed herein.

[0087] The invention may be embodied in other specific forms without departing from the subject matter of the claims. The exemplary embodiments described are merely illustrative in all respects and not restrictive. Features selected from one or more of the foregoing embodiments may be combined to create alternative embodiments not explicitly described, and features suitable for such combinations should be understood to be within the scope of the invention.

[0088] All values ​​and sub-ranges within the scope of the disclosure are also disclosed. Furthermore, although the systems, devices, and processes disclosed and shown herein may include a specific number of elements / components, modifications may be made to say the systems, devices, and assemblies to include more or fewer of such elements / components. For example, although any element / component disclosed may be mentioned in the singular, embodiments disclosed herein may be modified to include multiple such elements / components. The subject matter described herein is intended to cover and encompass all appropriate technical changes.

Claims

1. A computer-implemented method for controlling the behavior of an object, characterized in that, include: By using a first agent to control the behavior of the object during a first time period, the first agent applies a first behavior strategy to map observations about the object and the environment during the first time period to corresponding control actions applied to the object. During the transition period following the first time period, the behavioral control of the object will be transferred from the first agent to the second agent; By using a second agent to control the behavior of the object during a second time period following the transition period, the second agent applies a second behavior strategy to map observations about the object and the environment during the second time period to corresponding control actions applied to the object; During the transition period, the first agent applies the first behavior strategy to map observations about the object and the environment during the transition period to corresponding control actions applied to the object, and the second agent applies the second behavior strategy to map observations about the object and the environment during the transition period to corresponding control actions not applied to the object.

2. The method according to claim 1, characterized in that, The observations mapped by the first behavioral strategy and the observations mapped by the second behavioral strategy each come from their respective different observation spaces.

3. The method according to claim 1 or 2, characterized in that, During the transition period, a set of observations about the object and the environment is modified to include the observations about the object and the environment required by the second behavioral strategy.

4. The method according to claim 1 or 2, characterized in that, The first time period corresponds to the time when the object appears in a first region defined by a first spatiotemporal boundary, the second time period corresponds to the time when the object appears in a second region defined by a second spatiotemporal boundary, the transition period corresponds to the time when the object appears in a transition region between the first region and the second region, and the method includes performing a transition when the object appears in the transition region after it appears in the first region.

5. The method according to claim 1 or 2, characterized in that, Also includes: During another transition period following the second time period, the behavioral control of the object will be transferred from the second agent to the first agent; During the other transition period, the second agent applies the second behavior strategy to map observations about the object and the environment during the second transition period to corresponding control actions applied to the object, and the first agent applies the first behavior strategy to map observations about the object and the environment during the other transition period to corresponding control actions not applied to the object.

6. The method according to claim 1 or 2, characterized in that, The method is applied during simulation, the object is a simulated object, and the observations about the object and the environment are simulation observations about the object and the environment.

7. The method according to claim 6, characterized in that, The object is a simulated social vehicle operating in a simulated environment, which also includes a simulated autonomous vehicle controlled by a corresponding autonomous vehicle agent throughout the first time period, the transition period, and the second time period. The corresponding autonomous vehicle agent applies autonomous vehicle behavior policies to map ongoing observations about the autonomous vehicle and the environment to corresponding autonomous vehicle control actions applied to the autonomous vehicle.

8. The method according to claim 7, characterized in that, The first time period corresponds to the time when the object appears in a first region defined by a first spatiotemporal boundary, the second time period corresponds to the time when the object appears in a second region defined by a second spatiotemporal boundary, the transition period corresponds to the time when the object appears in a transition region between a first bubble and a second bubble, the method includes performing a transition when the object appears in the transition region after it appears in the first region, and wherein the second region and the transition region are fixed at virtual positions, the virtual positions moving together with the virtual positions of the simulated vehicle within the simulation environment.

9. The method according to claim 7, characterized in that, The first time period corresponds to the time when the object appears in a first region defined by a first spatiotemporal boundary, the second time period corresponds to the time when the object appears in a second region defined by a second spatiotemporal boundary, the transition period corresponds to the time when the object appears in a transition region between the first region and the second region, the method includes performing a transition when the object appears in the transition region after it appears in the first region, and wherein the second region and the transition region are fixed at virtual locations, the virtual locations being stationary relative to virtual physical locations within the simulation environment.

10. The method according to claim 1, 2, 7, 8, or 9, characterized in that, The computational intensity of the first behavior strategy is lower than that of the second behavior strategy.

11. The method according to claim 10, characterized in that, The second behavioral strategy is used to map observations about the object and the environment from the observation space, which is richer than the observation space from which the first behavioral strategy maps observations.

12. The method according to claim 11, characterized in that, The second behavioral strategy is used to map observations about the object and the environment to control actions from an action space that is richer than the action space from which the first behavioral strategy maps observations about the object and the environment to control actions.

13. A computer system, characterized in that, include: processor; A memory coupled to the processor, the memory storing instructions that, when executed by the processor, configure the computer system to: By using a first agent to control the behavior of an object during a first time period, the first agent applies a first behavior strategy to map observations about the object and the environment during the first time period to corresponding control actions applied to the object. During the transition period following the first time period, the behavioral control of the object will be transferred from the first agent to the second agent; By using a second agent to control the behavior of the object during a second time period following the transition period, the second agent applies a second behavior strategy to map observations about the object and the environment during the second time period to corresponding control actions applied to the object; During the transition period, the first agent applies the first behavior strategy to map observations about the object and the environment during the transition period to corresponding control actions applied to the object, and the second agent applies the second behavior strategy to map observations about the object and the environment during the transition period to corresponding control actions not applied to the object.

14. The computer system according to claim 13, characterized in that, The observations mapped by the first behavioral strategy and the observations mapped by the second behavioral strategy each come from their respective different observation spaces.

15. The computer system according to claim 13, characterized in that, During the transition period, a set of observations about the object and the environment is modified to include the observations about the object and the environment required by the second behavioral strategy.

16. The computer system according to claim 13 or 14, characterized in that, The first time period corresponds to the time when the object appears in a first region defined by a first spatiotemporal boundary, the second time period corresponds to the time when the object appears in a second region defined by a second spatiotemporal boundary, the transition period corresponds to the time when the object appears in a transition region between the first region and the second region, and further includes performing a transition when the object is detected to appear in the transition region after it appears in the first region.

17. The computer system according to claim 13 or 14, characterized in that, The memory stores additional instructions that, when executed by the processor, configure the computer system to transfer behavioral control of the object from the second agent to the first agent during a further transition period following the second time period; and wherein, during the further transition period, the second agent applies the second behavioral strategy to map observations about the object and the environment during the second transition period to corresponding control actions applied to the object, and the first agent applies the first behavioral strategy to map observations about the object and the environment during the further transition period to corresponding control actions not applied to the object.

18. The computer system according to claim 13 or 14, characterized in that, When executed by the processor, the instructions configure the computer system to perform a simulation, wherein the object is a simulated object, and the observations about the object and the environment are simulation observations about the object and the environment.

19. The computer system according to claim 18, characterized in that, The object is a simulated social vehicle operating in a simulated environment, which also includes a simulated autonomous vehicle controlled by a corresponding autonomous vehicle agent throughout the first time period, the transition period, and the second time period. The corresponding autonomous vehicle agent applies autonomous vehicle behavior policies to map ongoing observations about the autonomous vehicle and the environment to corresponding autonomous vehicle control actions applied to the autonomous vehicle.

20. The computer system according to claim 19, characterized in that, The first time period corresponds to the time when the object appears in a first region defined by a first spatiotemporal boundary, the second time period corresponds to the time when the object appears in a second region defined by a second spatiotemporal boundary, and the transition period corresponds to the time when the object appears in a transition region between a first bubble and a second bubble. The transition is performed when the object appears in the transition region after it appears in the first region, and the second region and the transition region are fixed at virtual positions that move together with the virtual position of the simulated vehicle within the simulation environment.

21. The computer system according to claim 19, characterized in that, The first time period corresponds to the time when the object appears in a first region defined by a first spatiotemporal boundary, the second time period corresponds to the time when the object appears in a second region defined by a second spatiotemporal boundary, and the transition period corresponds to the time when the object appears in a transition region between the first region and the second region. The transition is performed when the object appears in the transition region after it appears in the first region, and the second region and the transition region are fixed at virtual locations, which are stationary relative to virtual physical locations within the simulation environment.

22. The computer system according to claim 13, 14, 19, 20, or 21, characterized in that, The computational intensity of the first behavior strategy is lower than that of the second behavior strategy.

23. The computer system according to claim 22, characterized in that, The second behavioral strategy is used to map observations about the object and the environment from the observation space, which is richer than the observation space from which the first behavioral strategy maps observations.

24. The computer system according to claim 23, characterized in that, The second behavioral strategy is used to map observations about the object and the environment to control actions from an action space that is richer than the action space from which the first behavioral strategy maps observations about the object and the environment to control actions.

25. A computer program product, characterized in that, Includes instructions that, when executed by a processor of a computing system, cause the computing system to perform the method of any one of claims 1 to 12.

26. A computer-readable medium, characterized in that, Includes instructions that, when executed by a processor of a computing system, cause the computing system to perform the method of any one of claims 1 to 12.

Images