Method, system and computer program product for generating a control obstacle function

Abstract

A method, system, and computer program product for generating a control obstacle function are disclosed. The present disclosure relates to an automated system for generating an approximate safety condition for monitoring and verification. A system and method for generating a safety condition for a cyber-physical system having a state space S, an action space A, and trajectory data labeled as safe or unsafe are described. In operation, the system receives an input and then minimizes a loss function to cause a neural network to become an obstacle function. Based on the obstacle function, the system can then determine whether the cyber-physical system is entering an unsafe state, such that if the cyber-physical system is entering an unsafe state, the cyber-physical system is caused to initiate a manipulation to place the cyber-physical system in a safe state.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of and serves as a non-provisional patent application to U.S. Provisional Application No. 62 / 953,088, filed on December 23, 2019, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present invention relates to machine learning systems and, more particularly, to automated systems for generating security conditions for monitoring and verification. Background Art

[0004] The present disclosure is devoted to a process for generating so-called barrier functions. The concept of barrier functions in their most recent form began with the work of Prajna et al. (see the list of incorporated references, references No. 6 and No. 7), although this concept was implicit in earlier work. Barrier functions are incorporated to provide safety for hybrid and nonlinear dynamic systems. This concept was introduced into control systems by Wieland et al. (see reference No. 8) and was later reformulated in many ways, particularly in recent work by Ames et al. (see references No. 2 and No. 3). For an overview of barrier functions and related formulations, see the work of Ames et al. (2019) (see reference No. 1). For a formulation of barrier functions, see also the work of Ohnishi et al. (see reference No. 5).

[0005] Control barrier functions have been used to ensure safety in the control of many different systems. It is often the case that barrier functions are identified analytically (and particularly manually) according to a specific application, and there do not appear to be many general methods for constructing barrier functions for general classes of control systems. An exception is given in the work of Wu et al. (see Reference No. 9), where a general method for constructing barrier functions is given for fully actuated simple mechanical systems.

[0006] A related work is that of Fisac ​​et al. (see Reference No. 4). In this work, the authors develop a method to approximate the global safe region of the state space of a given control system. Their method does this by learning a fixed function of a shrinkage operator such that the support of the shrinkage operator approximates the global safe region. While this method has the advantage of improving the region approximation, the weakness of the method is that it cannot provide an absolute guarantee that the support of the function indicates the true safe region.

[0007] In contrast, as the process specified in the present disclosure is directly aimed at learning a barrier function, it is guaranteed to be supported in the global safety region of the control system. Therefore, the approximate barrier function generated by the present method is more likely to be supported in the global safety region, and the method can be easily used in succession with a satisfiability modulo theory solver to obtain a true (rather than just an approximate) barrier function, from which a provably safe region of the state space of the system can be identified. At a more general level, another advantage of the method provided by the present disclosure is that it naturally encompasses the method of Fisac ​​et al. (and is more general than the method of Fisac ​​et al.). In other words, the method of Fisac ​​et al. can be regarded as a specific step of the present invention. Therefore, for example, the function generated by the method of Fisac ​​et al. can be found in the present invention and can then be further improved in subsequent learning steps, as described in further detail below.

[0008] The method specified in the present disclosure is an unexpected improvement over the prior art because when researchers in control theory have considered barrier functions, they have focused on barrier functions that are explicitly identified and described, rather than learning barrier functions from data. For example, the closest work (i.e., Fisac ​​et al.) is inspired by the theory of contraction operators related to reinforcement learning. In contrast, because the method of the present disclosure explicitly focuses on, rather than attempts to learn, the barrier function, the method is more likely to produce high-quality approximations. In particular, the function learned by the present method is more likely to be the actual barrier function (because it can potentially be proven using a satisfiability modulo theory solver), and therefore any safety guarantees about the output of the process will likely be of higher quality. Likewise, compared to the manual and analytical approaches taken in the literature on control theory, the present method can potentially learn barrier functions for significantly more complex dynamical systems that can be handled manually feasibly.

[0009] Other work on using machine learning to learn barrier functions has been done by Zhao et al. (2020) (see reference No. 10) and another by Chang et al. (2020) (see reference No. 11). However, there are some important differences between these works and the present invention. First, the work of Zhao et al. is applicable to uncontrolled dynamic systems, that is, systems that do not have control variables, or alternatively, systems in which such control variables are governed by a set control strategy. In addition, the work of Zhao et al. requires a specific set of initial conditions, and the learned barrier function is intended to ensure the safety of this initial condition. Moreover, the work of Chang et al. learns barrier functions together with control strategies as a way to help make strategy learning safe. In particular, the purpose of this method is only to learn barrier functions that verify the safety of a specific learned control strategy.

[0010] Unlike these works, the disclosed system learns barrier functions that are not limited to verifying the safety of a particular set of initial conditions or control strategies. Rather, the disclosed system generally seeks to ensure the safety of the state space of the control system, and can potentially be used to find complete safe regions of as much state space as possible, subject to limitations imposed by the data, dimensionality, and other aspects of a given application. Thus, given the problems of the prior art, the disclosed system addresses the ongoing need for improved and automated systems that generate approximate safety conditions for monitoring and verification. Summary of the invention

[0011] A system and method for generating a safety condition of a cyber-physical system having a state space S, an action space A, and trajectory data labeled as safe or unsafe is described. In one aspect, the system includes one or more processors and a memory. The memory is a non-transitory computer-readable medium encoded with executable instructions, such that when the instructions are executed, the one or more processors perform multiple operations, such as: (1) receiving an input, the input being at least one of a discrete dynamics model, a continuous dynamics model, a set of state space data points, a neural network, an initial safety function, and a method for determining an action that is optimal according to the neural network in a given dynamics system state; (2) if the discrete dynamics model is part of the input, minimizing a first loss function and a second loss function and then continuing, and if the discrete dynamics model is not part of the input, continuing; (3) optimizing the neural network h by minimizing a discriminative loss function θ becomes a barrier function; and (4) outputs the barrier function.

[0012] In another aspect, the system enables the cyber-physical system to initiate physical operations based on a barrier function.

[0013] In another aspect, the system performs the following operations: determining whether the cyber-physical system is entering an unsafe state based on the barrier function, so that if the cyber-physical system is entering an unsafe state, the cyber-physical system is caused to initiate manipulation to place the cyber-physical system in a safe state.

[0014] In yet another aspect, minimizing the first loss function and the second loss function comprises the following operations:

[0015] Minimize the first loss function, which makes the neural network h θ Learn an approximation of the initial security function;

[0016] Minimize the second loss function, which makes the neural network h θSatisfies the Bellman-like equation, which in turn modifies the neural network h θ , so that the neural network h θ Closer to being a barrier function than its predecessor;

[0017] Iterate the minimization of the second loss function to make the neural network h θ Satisfies the Bellman-like equation, which further modifies the neural network h θ , so that the neural network h θ is closer to being a barrier function than its predecessor; and

[0018] The minimization of the second loss function is iterated a predetermined number of times.

[0019] In addition, when minimizing the discriminative loss function, the loss function is minimized for the discrete dynamics model And minimize the loss function for the continuous dynamics model

[0020] On the other hand, the discrete dynamics model is τ:S×A→S, where S represents the state space of the dynamics system, A represents the action space, and τ represents the transfer function that produces the state of the information-physical system derived from the input state-action pairs.

[0021] In addition, the continuous kinetic model is where s represents the state of the system, and a represents the control action being taken, and denotes the time derivative of s, and f denotes a vector field function that produces the time derivative of the state resulting from the input state-action pair.

[0022] Finally, the present invention also includes a computer program product and a computer-implemented method. The computer program product includes computer-readable instructions stored on a non-transitory computer-readable medium, and the computer-readable instructions can be executed by a computer having one or more processors so that when the instructions are executed, the one or more processors perform the operations listed herein. Alternatively, the computer-implemented method includes an act of causing a computer to execute such instructions and perform the resulting operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The objects, features and advantages of the present invention will become apparent from the following detailed description of various aspects of the present invention, taken in conjunction with the following drawings, in which:

[0024] Figure 1 is a block diagram depicting components of a system according to various embodiments of the present invention;

[0025] Figure 2 is an illustrative diagram of a computer program product embodying one aspect of the present invention;

[0026] Figure 3 is a function h that depicts the result of the learning process according to various aspects of the present invention. θ Graphics;

[0027] Figure 4 is a table providing statistics from example evaluations of the system according to various aspects of the invention;

[0028] Figure 5 is a function h describing the learning process according to various aspects of the present invention. θ Graphics;

[0029] Figure 6 is depicted in Under the assumption of h θ (x, θ , s) relative to (x, θ ) values;

[0030] Figure 7 is a diagram outlining the barrier function learning process; and

[0031] Figure 8 is a block diagram depicting control of an apparatus according to various embodiments. DETAILED DESCRIPTION

[0032] The present invention relates to machine learning systems, and more specifically, to automated systems for generating security conditions for monitoring and verification. The following description is presented to enable one of ordinary skill in the art to make and use the invention and incorporate it into the context of a particular application. Various modifications and multiple uses for different applications will be apparent to those skilled in the art, and the general principles defined herein can be applied in a wide range of aspects. Therefore, the present invention is not intended to be limited to the aspects presented, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0033] In the following detailed description, many specific details are set forth to provide a more detailed understanding of the present invention. However, it should be understood by those skilled in the art that the present invention can be practiced without being limited to these specific details. In other cases, known structures and devices are shown in block diagram form rather than in detail to avoid obscuring the present invention.

[0034] The reader's attention is directed to all papers and documents filed concurrently with this specification and which are open to public inspection together with this specification, the contents of all of which are incorporated herein by reference. All features disclosed in this specification (including any accompanying claims, abstract, and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise. Therefore, unless expressly stated otherwise, each feature disclosed is merely an example of a general series of equivalent or similar features.

[0035] Before describing the present invention in detail, a list of cited references is first provided. Next, a description of each major aspect of the present invention is provided. Subsequently, an introduction is provided to the reader so that a general understanding of the present invention can be obtained. Finally, specific details of various embodiments of the present invention are provided to obtain an understanding of specific aspects.

[0036] (1) List of references

[0037] The following references are cited throughout this application. For clarity and convenience, these references are listed here as a central resource for the reader. The following references are incorporated herein by reference as if fully set forth herein. These references are cited in this application by reference to the following corresponding reference numbers:

[0038] 1.Ames,AD,Coogan,S.,Egerstedt,M.,Notomista,G.,Sreenath,K.,&Tabuada,P.(2019).Control Barrier Functions: Theory and Applications.IEEE EuropeanControl Conference(pp.3420-3431).Naples,Italy:IEEE.

[0039] 2.Ames, AD, Grizzle, JW, & Tabuada, P. (2014). Control Barrier Functionbased Quadratic Programs with Application to Adaptive Cruize Control. IEEE Conference on Decision and Control (pp.6271-6278). Los Angeles, CA: IEEE.

[0040] 3.Ames,A.D.,Xu,X.,Grizzle,J.W.,&Tabuada,P.(2017).Control BarrierFunction Based Quadratic Programs for Safety Critical Systems.IEEETransactions on Automatic Control,62(8),3861-3876.

[0041] 4.Fisac,J.F.,Lugovoy,N.F.,Rubies-Royo,V.,Ghosh,S.,&Tomlin,C.J.(2019).Bridging Hamilton-Jacobi Safety Analysis and ReinforcementLearning.Retrieved from UC Berkeley:Jaime Fisac Papers:found atpeople.eecs.berkeley.edu / ~jfisac / papers / Bridging_Safety_and_RL.pdf.

[0042] 5.Ohnishi,M.,Wang,L.,Notomista,G.,&Egerstedt,M.(2019).Barrier-Certified Adaptive Reinforcement Learning With Applications to BrushbotNavigation.IEEE Transactions on Robotics,35(5),1186-1205.

[0043] 6.Prajna,S.(2006).Barrier certificates for nonlinear modelvalidation.Automatica,42(1),117-126.

[0044] 7.Prajna,S.,&Jadbabaie,A.(2004).Safety Verification of Hybrid SystemsUsing Barrier Certificates.HybridSystems:Computation and Control(pp.477-492).Philadelphia,PA:Springer.

[0045] 8. Wieland, P., & F. (2007). Constructive safety using control barrier functions. IFAC Symposium on Nonlinear Control Systems (pp. 462 - 467). Pretoria, South Africa: Elsevier.

[0046] 9. Wu, G., & Sreenath, K. (2016). Safety - Critical Geometric Control for Systems on Manifolds Subject to Time - Varying Constraints. IEEE Transactions on Automatic Control, in review.

[0047] 10. Zhao, H., Zeng, X., Chen, T., & Liu, Z. (2020). Synthesizing barrier certificates using neural networks. ACM International Conference on Hybrid Systems: Computation and Control. Sydney, Australia: ACM.

[0048] 11. Chang, Y.-C., & Gao, S. (2020). Stabilizing Neural Control Policies with Invariance Guarantee Using Self - Learned Lyapunov Critics. Unpublished.

[0049] (2) Main aspects

[0050] Various embodiments of the present invention include three "main" aspects. The first aspect is a system for generating a security condition for monitoring and verification. The system is usually in the form of computer system operating software or in the form of a "hard-coded" instruction set. The system can be incorporated into a variety of devices that provide different functions. The second main aspect is a method that is usually in the form of software that operates using a data processing system (computer). The third main aspect is a computer program product. A computer program product usually represents computer-readable instructions stored on a non-transitory computer-readable medium such as an optical storage device (e.g., a compact disk (CD) or a digital versatile disk (DVD)) or a magnetic storage device (e.g., a floppy disk or tape). Other non-limiting examples of computer-readable media include: a hard disk, a read-only memory (ROM), and a flash-type memory. These aspects will be described in more detail below.

[0051] Figure 1 A block diagram depicting an example of a system of the present invention (i.e., computer system 100) is provided in . The computer system 100 is configured to perform calculations, processes, operations, and / or functions associated with a program or algorithm. In one aspect, certain processes and steps discussed herein are implemented as a series of instructions (e.g., software programs) that reside in a computer-readable memory unit and are executed by one or more processors of the computer system 100. When executed, the instructions cause the computer system 100 to perform specific actions and exhibit specific behaviors, as described herein. In various aspects, the computer system 100 can be specifically implemented in any device that operates to perform the functions described herein that are suitable for a specific application, such as a desktop computer, a mobile phone or smart phone, a tablet computer, a computer specifically implemented in a mobile platform, or one or more other devices that can execute instructions individually and / or collectively to perform related operations / processing.

[0052] The computer system 100 may include an address / data bus 102 configured to transmit information. In addition, one or more data processing units (such as a processor 104 (or multiple processors)) are connected to the address / data bus 102 and / or associated memory. The processor 104 is configured to process information and instructions. In one aspect, the processor 104 is a microprocessor. Alternatively, the processor 104 can be a different type of processor, such as a parallel processor, an application specific integrated circuit (ASIC), a programmable logic array (PLA), a complex programmable logic device (CPLD) or a field programmable gate array (FPGA), or any other processing component that can be operated to perform related operations.

[0053] The computer system 100 is configured to utilize one or more data storage units (memories). The computer system 100 may include a volatile memory unit 106 (e.g., random access memory (“RAM”), static RAM, dynamic RAM, etc.) coupled to the address / data bus 102, wherein the volatile memory unit 106 is configured to store information and instructions for the processor 104. The computer system 100 may also include a non-volatile memory unit 108 (e.g., read-only memory (“ROM”), programmable ROM (“PROM”), erasable programmable ROM (“EPROM”), electrically erasable programmable ROM (“EEPROM”), flash memory, etc.) coupled to the address / data bus 102, wherein the non-volatile memory unit 108 is configured to store static information and instructions for the processor 104. Alternatively, the computer system 100 may execute instructions retrieved from an online data storage unit such as in “cloud” computing. In one aspect, the computer system 100 may also include one or more interfaces, such as interface 110, coupled to the address / data bus 102. The one or more interfaces are configured to enable the computer system 100 to connect with other electronic devices and computer systems. The communication interface implemented by the one or more interfaces may include wired (e.g., serial cable, modem, network adapter, etc.) and / or wireless (e.g., wireless modem, wireless network adapter, etc.) communication technology.

[0054] In one aspect, the computer system 100 may include an input device 112 coupled to the address / data bus 102, wherein the input device 112 is configured to transmit information and command selections to the processor 104. According to one aspect, the input device 112 is an alphanumeric input device (such as a keyboard), which may include alphanumeric keys and / or function keys. Alternatively, the input device 112 may be an input device other than an alphanumeric input device. In one aspect, the computer system 100 may include a cursor control device 114 coupled to the address / data bus 102, wherein the cursor control device 114 is configured to transmit user input information and / or command selections to the processor 104. In one aspect, the cursor control device 114 is implemented using a device such as a mouse, a trackball, a trackpad, an optical tracking device, or a touch screen. Notwithstanding the foregoing, in one aspect, the cursor control device 114 is directed and / or enabled via input from the input device 112, such as in response to using special keys and key sequence commands associated with the input device 112. In the alternative, cursor control device 114 is configured to be managed or directed via voice commands.

[0055] In one aspect, computer system 100 may also include one or more optional computer usable data storage devices, such as storage device 116 coupled to address / data bus 102. Storage device 116 is configured to store information and / or computer executable instructions. In one aspect, storage device 116 is a storage device such as a magnetic or optical drive (e.g., a hard disk drive ("HDD"), a floppy disk, a compact disk read only memory ("CD-ROM"), a digital versatile disk ("DVD")). According to one aspect, display device 118 is coupled to address / data bus 102, wherein display device 118 is configured to display video and / or graphics. In one aspect, display device 118 may include a cathode ray tube ("CRT"), a liquid crystal display ("LCD"), a field emission display ("FED"), a plasma display, or any other display device suitable for displaying video and / or graphic images and alphanumeric characters recognizable by a user.

[0056] The computer system 100 presented herein is an example computing environment according to one aspect. However, the non-limiting example of the computer system 100 is not strictly limited to being a computer system. For example, one aspect provides a computer system 100 representing a class of data processing analysis that can be used according to various aspects described herein. In addition, other computing systems can also be implemented. In fact, the spirit and scope of the present technology are not limited to any single data processing environment. Therefore, on the one hand, one or more operations of various aspects of the present technology are controlled or implemented using computer executable instructions (such as program modules) executed by a computer. In one implementation, such a program module includes routines, programs, objects, components and / or data structures configured to perform specific tasks or implement specific abstract data types. In addition, one aspect provides one or more aspects of the present technology implemented by utilizing one or more distributed computing environments, such as, in this computing environment, tasks are performed by remote processing devices linked by a communication network, or such as, in this computing environment, various program modules are located in local and remote computer storage media including memory-storage devices.

[0057] Figure 2, a diagram of a computer program product (i.e., storage device) embodying the present invention is depicted in . The computer program product is depicted as a floppy disk 200 or an optical disk 202 such as a CD or DVD. However, as previously mentioned, the computer program product generally represents computer-readable instructions stored on any compatible non-transitory computer-readable medium. The term "instructions" as used in connection with the present invention generally indicates a set of operations to be performed on a computer, and may represent a fragment of an entire program or a single separate software module. Non-limiting examples of "instructions" include computer program code (source or object code) and "hard-coded" electronic devices (i.e., computer operations encoded into a computer chip). "Instructions" are stored on any non-transitory computer-readable medium, such as in a computer's memory or on a floppy disk, CD-ROM, and flash drive. In any case, the instructions are encoded on a non-transitory computer-readable medium.

[0058] (3) Introduction

[0059] The present disclosure provides a novel machine learning technique for learning a function that effectively and timely monitors the safety of a system. The function can be used to significantly improve a variety of systems. For example, the function can be used to: (1) provide input to a runtime monitoring system to evaluate the safety of the system and determine whether to intervene; (2) be used in a formal verification process to attempt to prove (or destroy) the safety of the algorithm being used to control the system; or (3) be used for control or decision-making systems. The type of system we consider is a cyber-physical system (CPS), which includes autonomous systems such as self-driving cars, autopilot systems, etc. In the case where the state space of the system is a set S, the machine learning method produces a mathematical function as output. Intuitively, for each state s∈S, the function measures the degree h(s) of being a globally safe state. Here, being a globally safe state s means that there exists a safe trajectory starting with s. This is in contrast to an instantaneous safe state where no such safe trajectory may exist (e.g., if the car is driving too fast in the direction of an obstacle, then it may be safe now, but this will not be the case for a long time). In the expected semantics, those states s such that h(s)>0 are estimated to be globally safe. The process requires a sequence of labeled states, where the label indicates whether the state is considered to be instantaneously safe. These constitute the data required to obtain h using machine learning.

[0060] The central premise of the present invention is specifically a method for learning a class of functions called barrier functions for a range of control systems. Barrier functions, once found, characterize the global safe region of their corresponding state space and can therefore be used practically for safety monitoring, informing control / decision making, and for formal verification of system safety. There are not many general methods for finding barrier functions, the traditional methods are manual, and the present process is the first to do so using machine learning.

[0061] The systems and methods of the disclosed invention have a number of unique uses and advantages. In terms of usage, there are three main modes in which the invention can be employed: (1) for (runtime) safety monitoring, (2) for notifying control / planning modules, and (3) for formal verification.

[0062] First, the purpose of safety monitoring is to provide an indication of system safety at a given time. In particular, safety monitoring is often used at runtime, and when a state that is considered too close to being unsafe is encountered, a monitor flag appears and a redundant / backup / fail-safe system is allowed to take over control of the system. This operation can be implemented in a variety of applications, non-limiting examples of which include autonomous vehicles. As a specific example scenario, for an autonomous car, if the safety monitor determines that the vehicle is too close to driving out of the specified driving lane, the safety monitor may take over the driving to move the vehicle away from the lane marker. In the above notation, given a threshold value δ>0, the present invention can be used: whenever a state s is encountered, such that h(s)<δ, a monitoring flag appears for runtime monitoring. This can be used on autonomous systems as well as human-operated systems. Therefore, in some aspects, the present invention can be used as a safety monitoring tool for information-physical systems (such as automobiles, air or spacecraft, etc.), so that the present invention is used on the platform or at a remote location to actively monitor the safety of the platform.

[0063] Second, the present invention can be used in conjunction with automatic control or planning functions. In these cases, at each time t, the system is in state s t , and select action a from action set A t. For example, in an autonomous car, the action may be adjusting the steering angle or throttle, while in an autonomous aircraft, the action may be adjusting the ailerons. In this setting, the above function h can be used to filter acceptable actions as follows. Assume that the system is provided with an estimate τ(s, a) of the state that will be reached by taking action a from s. Then, given a mechanism for sampling candidate actions a, the system can operate by first evaluating whether h(τ(s, a))>δ against a fixed threshold δ. If the inequality holds, action a is selected, otherwise a new candidate action a′ is sampled and the process is repeated. Therefore, the system can be implemented as an automatic control or decision-making module on a cyber-physical system, wherein the automatic control or decision-making module consumes state information obtained from sensors or other modules and generates an output that is routed to actuators or to other modules.

[0064] Finally, for a threshold δ, the condition h(s) > δ can be used as an approximate definition that it indicates that the state s is (globally) safe. Therefore, this condition can be used as a formally verified safety specification to obtain mathematical guarantees about the system behavior for this condition, or examples of violations of this condition that may indicate safety failures. Therefore, in some aspects, the present invention operates to generate functions (neural networks for approximating barrier functions) that can be used by external tools (such as satisfiability modulo theory solvers or computer proof assistants) as specifications against which formal verification of control or decision-making modules of cyber-physical systems is performed.

[0065] Those skilled in the art will appreciate that the processing described herein may be incorporated into a variety of systems, including applications for advanced driver assistance, flight management, flight control, autonomous driving, etc. For each of these domains, the present invention may be used to provide safety monitoring capabilities, improved control capabilities, or for formal verification purposes (e.g., in conjunction with a more general verification and validation regime). Specific details are provided below.

[0066] (4) Specific details of each implementation method

[0067] As mentioned above, the present disclosure provides an automated system for identifying functions that monitor system safety in a timely and effective manner. As a preliminary issue, the concept of barrier functions for both discrete-time control systems and continuous-time control systems must be defined. Assume first that there is a discrete-time control system defined by a transfer function τ: S×A→S, where S and A are the state space and action space of the system, respectively. Therefore, taking an action a∈A in a state s∈S leads to a subsequent state τ(s, a). The transfer function can be known or learned from the system trajectory. It is also assumed that there is a subset containing all states that are considered instantaneously safe It is expected that the state of the system never leaves K. The barrier function is the function Such that there exists a constant η∈[0,1) where, for all s∈S satisfying h(s)>0, And if Then h(s) ≤ 0. For such a function, a state s∈S such that h(s) > 0 is known to be globally safe; that is, starting from s, the system state can be maintained indefinitely in the future in K.

[0068] To define the barrier function for a continuous-time control system, as described above, assume that S and A are the state space and action space, and is a safe state set. Assume and And assume that the dynamics of the system is given by the differential equations Definition, where is a Lipschitz continuous function. Then, the barrier function of this system is the function Such that there exists a constant κ>0, where, for all s∈K satisfying h(s)>0, And if Then h(s)≤0. Then, as mentioned before, the state s∈S such that h(s)>0 is known to be globally safe.

[0069] Next, we describe a method for learning a barrier function from data in both discrete and continuous settings. We begin with the discrete setting. Assume that the control system is defined by the transfer function τ: S×A→S and the safety set K as described above. Next, assume that there is system state data obtained in some way For example, the system state data obtained from previous system simulations, random state space sampling, or real-world data collection. Then, D can be partitioned into sets K′=D∩K and U=D∩(S\K) so that K′ includes safe data points and U includes unsafe data points. Next, let is a neural network, and it is expected to use the neural network to learn the obstacle function of the system.

[0070] like Figure 7 As shown, the learning process in the discrete setting consists of two fundamentally different learning phases; learning phase one 700 and learning phase two 702. Learning phase one 700 involves learning using a shrinking Bellman operator, which is similar to Q-learning and the work of Fisac ​​et al. (see Reference No. 4). In this work, the goal is to learn a function that satisfies the Bellman equation

[0071]

[0072] where γ∈(0,1) is the discount factor, and is a differentiable function that satisfies L(s) ≥ 0 if and only if s∈K. For γ≈1, the set {s∈S:V(s)>0} gives an approximation of the global safe set, which improves to γ→1. The first stage of the learning process is a generalization of this basic idea. Briefly, the above Bellman equation is generalized to make it related to the barrier function condition and to allow substitution of the function L in the equation. More details are in the following paragraphs.

[0073] It should be noted that Figure 7 A number of example inputs 704 are provided. Although each of these inputs must be provided, the nature of the inputs 704 will vary depending on the particular application in which the system is employed. For example, in some cases, the maximum value h for action a may be directly calculated. θ (τ(x, a)) (i.e., when the action set is small enough), but in other cases, machine learning algorithms can be used to make the estimate.

[0074] As in Fisac ​​et al., this treatment makes use of the differentiable function This function satisfies the weaker condition that L(s) > 0 only when s ∈ K. This function can be learned or given, and is considered to be an upper bound and a first approximation to the barrier function. Also fixed are the barrier function constant η ∈ [0, 1), the discount factor γ ∈ (0, 1), and other constants δ1, δ2 > 0. With these constants in place, the learning process can be described. The first step is to let h θ Learn an approximation of the function L; to do this, minimize the loss function.

[0075]

[0076] Note that the purpose of the first sum is to ensure that the value h at the unsafe point θ In the range The manner in which this minimization is performed will be clear to a person skilled in the art.

[0077] The goal of the subsequent learning step is to make h θ Learn to satisfy

[0078]

[0079] This expression corresponds to the Bellman operator if Then the Bellman operator is contractive, where h θ Seek to learn a fixed function. Note that in most cases the equality in this expression does not imply that h θ is the barrier function. Intuitively, h θ Provides a function that is closer to the barrier function than L. To do this, use hθ The current state of is periodically substituted for L in the above equation. Assuming that the equality in the above expression can always be obtained, this process is equivalent to finding a function sequence L≥H1≥H2≥…, where H1 satisfies

[0080]

[0081] And the subsequent function H k , k≥2, satisfying

[0082]

[0083] Alternatively, as in the work of Fisac ​​et al., one can let γ approach 1 while learning the function H k Any one of .

[0084] Theoretically, when all functions H k When a single constant is used as the lower bound, the limit Converges to the barrier function. In practice, the exact equality of the above expressions cannot be guaranteed; therefore, the process only seeks to minimize the difference between the two sides of the equality. The goal is thus to learn a sequence of functions L = H0, H1, H2, ... that satisfies the following formula

[0085]

[0086] More precisely, at step k (where H k-1 is what we just learned), minimize the loss function

[0087]

[0088] After a certain number of learning epochs determined by the user and based on performance or other criteria, stop and set h θ The current state is defined as H k . After a number of iterations k=n of replacing L in the Bellman equation, this stage in the learning process is terminated. One thing to note is that one of the inputs to this process is an algorithm for accurately computing otherwise approximating the maximum value for A in the above loss function (as mentioned above, this can be computed directly if the set A is sufficiently small, but in other cases it can also be approximated using a separate machine learning process). In the simplified practice described below, A in the maximum value is replaced by a fixed finite subset of A. Note, however, that the computation of this maximum value itself can be formulated as a reinforcement learning problem, and techniques from this field can be applied to this purpose. Other approximations are also possible. The manner in which the above learning stage is performed in practice will be apparent to those skilled in the art.

[0089] The second and final stage in the learning process (i.e., learning stage two 702) is used to refine hθ , to satisfy the barrier function property by changing the loss function more precisely to

[0090]

[0091] This function pushes h more directly toward satisfying the barrier function properties θ And an approximate barrier function 706 is generated as output. Note that the constant α is positive and close to 0, and the term -αh θ (x j ) encourages greater support {s∈S:h θ (s)>0}, the idea is to find a larger global safe set. Typically, it may be necessary to change α during this phase. In particular, α typically starts with a larger value and decreases over time. This completes the description of the learning process for discrete control systems. The way to perform the above learning phase in practice will be obvious to those skilled in the art.

[0092] The following describes the barrier function learning process for a continuous control system. Assume that the control system consists of a differential function The definition and safety set K are as described above. Note that by defining a discrete approximation of the system, any of the learning stages in the discrete case can also be used in the continuous case. In fact, the transition function τ: S×A→S can be defined so that if an action a∈A is chosen in state s∈S, then τ(s,a) is an approximation of the state of the system after some fixed time interval (which can be obtained, for example, by Euler or Runge-Kutta methods).

[0093] After exploiting these phases, the main learning phase is as follows. As before, let is a data set, which is divided into instantaneous safe data points K′=D∩K and unsafe data points U=D∩(S\K); let is a neural network used to learn the obstacle function; and α, δ1, δ2>0 are set as constants. In addition, the obstacle function is fixed to a constant κ>0. Then, the neural network Learning is performed according to the following loss function:

[0094]

[0095] Similar to the second stage in the discrete case, this loss function directly drives h θ To satisfy the barrier function properties, the term -αh θ (x j) encourages greater support. Moreover, the constant α can start with a larger value and decrease over time. This completes the description of the learning process in the continuous case. The way to perform the above learning phase in practice will be obvious to those skilled in the art.

[0096] Next, several example applications are described in which the system has been simplified for practice. In the first example, consider a control system (i.e., a controlled dynamic system) with a state space Action Space The transfer function T defined by T((x, v), a) = (x + v, x + v + a) is S × A → S, and the instantaneous safety set Intuitively, the triple (x, v, a) can be thought of as the (position, velocity, acceleration) of the target, where the acceleration is controlled; and the point (x, v) = (0, 0) is an unstable fixed point. The ground truth global safe set can be shown as And therefore the effectiveness of the method in this example can be accurately evaluated.

[0097] The dataset D used to learn the approximate barrier function consists of states from 10,000 trajectories of the system, where each trajectory ends in the first state outside of K and starts from and The initial point and all actions are uniformly randomly selected. The transfer function τ: S×A→S used in the loss function in the learning process is an approximation of the learned T. The neural network used to learn the approximate barrier function is fully connected, has 2 hidden layers, 20 nodes per layer, and has a hyperbolic tangent nonlinearity. In this case, only the second stage of the learning process is used using a discrete-time barrier function. In particular, the experiment minimizes the loss function where η = 0.5, δ1 = 0.01, δ2 = 0.1, and various values ​​of α. The initial value of α is chosen to be large enough to make h θ is learned to be generally positive in the instantaneous safe region of the state space. Then, α is periodically reduced so that h θ Better conformity to the barrier function properties until l 0.5，0，0.01，0.1 ( θ ) is 0. In order to calculate the maximum value in the loss function, A is restricted to the subset {-1, 0, 1}. Figure 3 Plotting the function h resulting from the learning process θ . Specifically, Figure 3The values ​​of the barrier function 300 learned on the state space of the system are depicted, along with the (analytically learned) ground truth safety boundary 300 (i.e., the boundary between a globally safe state and a non-globally safe state). In this case, the learned barrier function correctly under-approximates the actual safe set.

[0098] Since the global safe region is known, the process evaluates h θ Indicates how good the accuracy of the global safety region is. Specifically, for the instantaneous safety state set h is calculated on a 200×200 grid of points θ The value of h θ Whether the point is correctly identified as globally safe or unsafe, corresponding to h θ (s)>0 and h θ (s)≤0. Figure 4 Statistics for this evaluation are given in the depicted table 400 .

[0099] It is worth noting that h θ The false safety rate 402 is 0%, which is h θ The globally unsafe points are incorrectly indicated as a percentage of the globally safe points. This means that h θ Be especially careful and do not mislead the control system into moving to an unsafe state. On the other hand, h θ The false unsafe rate 404 of h is 8.7%, that is, the false unsafe rate is the percentage of global safe points that the barrier function incorrectly indicates as globally unsafe. This shows that h θ Slightly conservative. However, the table 400 will h θ The results of h are compared with some baseline methods for determining the safety of a state. These methods take a given state and generate a certain number of random simulations of a certain length starting from that state, and mark the state as globally safe if and only if at least one of the simulations does not terminate in an unsafe state. As indicated in Table 400, h θ It is more accurate in determining the global safe state, while at the same time correctly identifying all unsafe states. The process is implemented in the Python programming language, and the implementation is familiar to those skilled in the art.

[0100] For the second example, consider a double integrator with the state space A continuous-time dynamical system with action space A = [1-,1] has a function defined by f((x,v),a) = (v,a) So that the dynamics is represented by the differential equations and instantaneous safety set As in the previous example, (x, v, a) can be thought of as (position, velocity, acceleration). The ground truth global safe set is also known in this example; it can be shown as The dataset D used to learn the approximate barrier function consists of a uniformly spanned set The neural network used is composed of 240×800 grid points. is fully connected, has 2 hidden layers, 20 nodes per layer, and has hyperbolic tangent nonlinearity. For this case, the learning process consists of minimizing the loss function for κ = 4, δ1 = 0.01, δ2 = 0.1, and various values ​​of α. As in the previous example, α is initially set to a sufficiently large positive number and is decreased throughout the process until is equal to 0. In order to calculate the maximum value in the loss function, A is again restricted to the subset {-1, 0, 1}. Figure 5 Plot the resulting function h θ . Specifically, Figure 5 The values ​​of the barrier function learned for the state space of the system and the corresponding ground truth safety boundary 500 are depicted. Note that in this case, the ground truth global safety region is also known and is qualitatively shown as h θ Identify the area accurately.

[0101] For the third and final example, consider the setting where the vehicle is driving on a straight road and the goal is to learn an obstacle function that indicates whether the vehicle can still stay in the lane. This situation is modeled as a discrete-time control system as follows. Let the state space be Where W = 4.2m, and The variable x represents the position of the center of the vehicle on the lane; the span of the lane relative to x is [-w, w], where w = 3.7 m. Also, the variable θ is the angle between the direction the vehicle is facing and the direction parallel to the road; and s is the speed of the vehicle, which is assumed to be non-negative and has a maximum value s max In addition, let the length of the vehicle be 4.12379394m and the width be 1.94305298m; and for a given vehicle state (x, θ, s), the x coordinates of the four corners of the vehicle can be easily calculated, which are denoted as x ur , x ul , x lr , x ll Next, let the action space be in, as well as The variable φ represents the steering angle of the vehicle, and a represents the acceleration. Note that the steering angle must be limited in such a way that the centripetal acceleration (r is the radius of the vehicle path resulting from the steering angle) with the friction coefficient between the vehicle and the road μ = 0.7 and the acceleration due to gravity The product μg of is bounded. Let the instantaneous safety region be the state set K = {(x,θ,s)∈S:-w≤x ur ,x ul , x lr ,x ll ≤w}.

[0102] Next, we define a transition function τ:S×A→S so that, given a state (x, θs)∈S with a selected action (φ, a)∈A, τ((x, θ, s), (φ, a)) is the transition function chosen as The state of the vehicle after the time gap of . Note that τ can be calculated analytically. The dataset D used to learn the approximate barrier function is obtained by uniformly randomly sampling 100,000 states for S. As in other examples, the neural network used is fully connected, has 2 hidden layers, 20 nodes per layer, and has hyperbolic tangent nonlinearity. For this example, two stages of discrete-time learning processing are used. Let γ = 0.75, η = 0.5, δ1 = 0.01, and δ2 = 0.4, and define the initial function by

[0103]

[0104] The learning process is done by minimizing the loss function To make h θ ≈L. The process is performed by minimizing Several iterations to promote h θ The closer we get to satisfying the barrier function properties, the more we continue. Finally, the process performs the minimization of the loss function l 0.5，α，0.01，0.4 (θ) steps (where α≈0) until l 0.5，0，0.01，0.4 (θ)=0, to push h θ To more accurately satisfy the barrier function properties. To calculate the maximum value in the loss function, restrict φ to the set {0, ±3 -k φ max : k = 0, 1, 2, 3, 4}, and a will be restricted to Based on the results of vehicle simulation, the final function h θ Empirically it seems to successfully indicate all global unsafe states, albeit with extreme caution. Figure 6 Depicted in Under the assumption of h θ The value of (x,θs) relative to (x,θ). Specifically, Figure 6Plotted are the values ​​of the barrier function learned for a portion of the system's state space.

[0105] The neural network architecture used in this case is the same as in the previous example. As a general rule of thumb, the task of learning a barrier function for a particular dynamical system should be viewed as being lower bounded by the reinforcement learning task for the same system. Thus, architectures that are sufficient for a fixed dynamical system are good starting points for barrier function learning architectures for reinforcement learning. Similar remarks apply to the problem of having enough data to learn a barrier function.

[0106] There are a number of additional options for the loss function that can be used in this process. In particular, the loss function can be made and An alternative to this is that the loss function uses h θ The value of is used to weight the term that penalizes violation of the barrier function condition. This makes the learning more stable in some cases. In addition, a loss function can be created for a continuous-time control system that is similar to The Bellman-like operator is formed by the Bellman-like operator, which works directly with continuous dynamics and does not require a discrete dynamic model of the system. These loss functions are used in The CNN is used before the CNN stage and can better guide the neural network to learn the approximate barrier function at the beginning of processing.

[0107] (4.1) Device control

[0108] like Figure 8 As shown, the system 100 can be used to control the device 800 (e.g., a mobile device display, a virtual reality display, an augmented reality display, a computer monitor, a motor, a machine, a drone, a camera, an autonomous vehicle, etc.) based on the obstacle function. For example, the device 800 can be controlled to move the device or otherwise initiate a physical action based on the obstacle function. As an example, if the obstacle function determines that the autonomous vehicle is planned to move out of the driving lane (the state is unsafe) by evaluating the current (estimated) system state based on sensor readings and the output of the sensor fusion / state estimation algorithm used on the platform, the system 100 will cause the vehicle to change the route (turn, brake, accelerate, etc.) to move the vehicle and maintain a safe state within the driving lane or otherwise. Similar examples can be envisioned in the case where the obstacle function corresponds to different safety attributes. In addition, the output of the obstacle function can be used in a loop with the control system as part of a learning process to better adjust or optimize the control of the system for safety performance.

[0109] Finally, although the present invention has been described according to several embodiments, it should be readily recognized by those skilled in the art that the present invention may have other applications in other environments. It should be noted that many embodiments and implementations are possible. Moreover, the appended claims are in no way intended to limit the scope of the present invention to the specific embodiments described above. In addition, any statement of "means for..." is intended to evoke a device-plus-function reading of the components and claims, and any component that does not specifically use the statement "means for..." is not intended to be interpreted as a device-plus-function component, even if the claim otherwise includes the word "means". In addition, although specific method steps have been stated in a particular order, these method steps may occur in any desired order and fall within the scope of the present invention.

Claims

1. A method for generating a control obstacle function for ensuring a safe condition of an autonomous vehicle, wherein the autonomous vehicle has a state space S, an action space A, and trajectory data marked as safe or unsafe, the method comprising the following actions: causing one or more processors to execute instructions encoded on the memory such that when executing the instructions, the one or more processors perform the following operations: Receive input, the input is a discrete dynamics model or a continuous dynamics model, a set of state space data points, a neural network h θ , an initial safety function L and a function for determining the neural network h under a given dynamic system state s θ is at least one of the methods of action a that is optimal; If the discrete dynamics model is part of the input, minimizing the first loss function and the second loss function and then continuing, and if the discrete dynamics model is not part of the input, continuing; in, The first loss function minimizes So that h θ ≈L, and wherein the second loss function minimizes Make where L is replaced by the functions H0, H1, H2, …, H k ,in, Where l represents the loss function, δ represents the fixed threshold, η represents the barrier function constant, k represents the number of iterations, A represents the action set, γ represents the discount factor, and τ represents the transfer function; The neural network h is transformed into θ The refinement becomes the control barrier function, that is, minimizing So that for all instantaneously safe x j ∈D, So that h θ (x j )>0, and for all unsafe x j ∈D,h θ (x j )≤0, where D represents the data set used to learn the approximate barrier function, outputting the control obstacle function; determining whether the autonomous vehicle is entering an unsafe state using the control obstacle function; and When the autonomous vehicle is in the unsafe state, the autonomous vehicle is enabled to initiate a physical operation.

2. The method according to claim 1, wherein: The physical operation is a manipulation that puts the autonomous vehicle in a safe state.

3. The method according to claim 1, wherein: Minimizing the first loss function and the second loss function includes the following operations: Minimize the first loss function, which makes the neural network h θ Learn an approximation of the initial security function; Minimize the second loss function, which makes the neural network h θ Satisfies the Bellman equation, which in turn modifies the neural network h θ , obtaining an updated approximation of the control obstacle function; Iterations of minimizing the second loss function are performed to make the neural network h θ Satisfies the Bellman equation, which further modifies the neural network h θ , obtaining an updated approximation of the control obstacle function; and The minimization of the second loss function is iterated a predetermined number of times.

4. The method according to claim 1, wherein: When minimizing the discriminative loss function, the loss function is minimized for the discrete dynamics model And minimize the loss function for the continuous dynamics model 5. The method according to claim 1, wherein: The discrete dynamics model is τ: S×A→S, where S represents the state space of the dynamics system, A represents the action space, and τ represents a transfer function that produces the state of the autonomous vehicle derived from an input state-action pair.

6. The method according to claim 1, wherein: The continuum kinetic model is where s represents the state of the system, and a represents the control action being taken, and denotes the time derivative of s, and f denotes a vector field function that produces the time derivative of the state produced by the input state-action pair.

7. A system for generating a control obstacle function, the control obstacle function being used to ensure a safe condition of an autonomous vehicle, the autonomous vehicle having a state space S, an action space A, and trajectory data marked as safe or unsafe, the system comprising: A memory and one or more processors, the memory being a non-transitory computer-readable medium having executable instructions encoded thereon, such that when the instructions are executed, the one or more processors perform the following operations: Receive input, the input is a discrete dynamics model or a continuous dynamics model, a set of state space data points, a neural network h θ , an initial safety function L and a function for determining the neural network h under a given dynamic system state s θ is at least one of the methods of action a that is optimal; If the discrete dynamics model is part of the input, minimizing the first loss function and the second loss function and then continuing, and if the discrete dynamics model is not part of the input, continuing; Wherein, the first loss function minimizes So that h θ ≈L, and wherein the second loss function minimizes Make where L is replaced by the functions H0, H1, H2, ..., H k ,in, Where e represents the loss function, δ represents the fixed threshold, η represents the barrier function constant, k represents the number of iterations, A represents the action set, γ represents the discount factor, and τ represents the transfer function; The neural network h is transformed into θ The refinement becomes the control barrier function, that is, minimizing So that for all instantaneously safe x j ∈D, So that h θ (x j )>0, and for all unsafe x j ∈D,h θ (x j )≤0, where D represents the data set used to learn the approximate barrier function, outputting the control obstacle function; determining whether the autonomous vehicle is entering an unsafe state using the control obstacle function; and When the autonomous vehicle is in the unsafe state, the autonomous vehicle is enabled to initiate a physical operation.

8. The system according to claim 7, wherein: The physical operation is a manipulation that puts the autonomous vehicle in a safe state.

9. The system according to claim 7, wherein: Minimizing the first loss function and the second loss function includes the following operations: Minimize the first loss function, which makes the neural network h θ Learn an approximation of the initial security function; Minimize the second loss function, which makes the neural network h θ Satisfies the Bellman equation, which in turn modifies the neural network h θ , obtaining an updated approximation of the control obstacle function; Iterations of minimizing the second loss function are performed to make the neural network h θ Satisfies the Bellman equation, which further modifies the neural network h θ , obtaining an updated approximation of the control obstacle function; and The minimization of the second loss function is iterated a predetermined number of times.

10. The system according to claim 7, wherein: When minimizing the discriminative loss function, the loss function is minimized for the discrete dynamics model And minimize the loss function for the continuous dynamics model 11. The system according to claim 7, wherein: The discrete dynamics model is τ: S×A→S, where S represents the state space of the dynamics system, A represents the action space, and τ represents a transfer function that produces the state of the autonomous vehicle derived from an input state-action pair.

12. The system according to claim 7, wherein: The continuum kinetic model is where s represents the state of the system, and a represents the control action being taken, and denotes the time derivative of s, and f denotes a vector field function that produces the time derivative of the state produced by the input state-action pair.

13. A computer program product for generating a control obstacle function for ensuring a safe condition of an autonomous vehicle, the autonomous vehicle having a state space S, an action space A, and trajectory data marked as safe or unsafe, the computer program product comprising: A non-transitory computer-readable medium having executable instructions encoded thereon such that when the instructions are executed by one or more processors, the one or more processors perform the following operations: Receive input, the input is a discrete dynamics model or a continuous dynamics model, a set of state space data points, a neural network h θ , an initial safety function L and a function for determining the neural network h under a given dynamic system state s θ is at least one of the methods of action a that is optimal; If the discrete dynamics model is part of the input, minimizing the first loss function and the second loss function and then continuing, and if the discrete dynamics model is not part of the input, continuing; Wherein, the first loss function minimizes So that h θ ≈L, and wherein the second loss function minimizes Make where L is replaced by the functions H0, H1, H2, ..., H k ,in, Where e represents the loss function, δ represents the fixed threshold, η represents the barrier function constant, k represents the number of iterations, A represents the action set, γ represents the discount factor, and τ represents the transfer function; The neural network h is transformed into θ The refinement becomes the control barrier function, that is, minimizing So that for all instantaneously safe x j ∈D, So that h θ (x j )>0, and for all unsafe x j ∈D,h θ (x j )≤0, where D represents the data set used to learn the approximate barrier function, outputting the control obstacle function; determining whether the autonomous vehicle is entering an unsafe state using the control obstacle function; and When the autonomous vehicle is in the unsafe state, the autonomous vehicle is enabled to initiate a physical operation.

14. The computer program product of claim 13, wherein: The physical operation is a manipulation that puts the autonomous vehicle in a safe state.

15. The computer program product of claim 13, wherein: Minimizing the first loss function and the second loss function includes the following operations: Minimize the first loss function, which makes the neural network h θ Learn an approximation of the initial security function; Minimize the second loss function, which makes the neural network h θ Satisfies the Bellman equation, which in turn modifies the neural network h θ , obtaining an updated approximation of the control obstacle function; Iterations of minimizing the second loss function are performed to make the neural network h θ Satisfies the Bellman equation, which further modifies the neural network h θ , obtaining an updated approximation of the control obstacle function; and The minimization of the second loss function is iterated a predetermined number of times.

16. The computer program product of claim 13, wherein: When minimizing the discriminative loss function, the loss function is minimized for the discrete dynamics model And minimize the loss function for the continuous dynamics model 17. The computer program product of claim 13, wherein: The discrete dynamics model is τ: S×A→S, where S represents the state space of the dynamics system, A represents the action space, and τ represents a transfer function that produces the state of the autonomous vehicle derived from an input state-action pair.

18. The computer program product of claim 13, wherein: The continuum kinetic model is where s represents the state of the system, and a represents the control action being taken, and denotes the time derivative of s, and f denotes a vector field function that produces the time derivative of the state produced by the input state-action pair.

Images