Systems, methods, and products for safe and effective override of autonomous systems

Abstract

The present disclosure relates to systems, methods, and products for safe and effective override of autonomous systems. A system for competence assessment of an autonomous system is described. The system extracts semantic concepts representing a situation. Actions taken by the autonomous system are associated with a set of semantic concepts activated when the actions are taken in the situation. The system measures outcomes of the actions taken in the situation and generates a reward metric. The semantic concepts representing the situation are stored as a memory along with the actions taken in the situation and the reward metric. A prospective simulation result is generated based on a recall of the memory. A competence metric and an experience metric are determined. When at least one of the competence metric and the experience metric is above a minimum value, competence operational control of the autonomous system is maintained. When at least one of the competence metric and the experience metric is below a minimum value, an alert is generated.

Description

[0001] Cross-reference to related applications

[0002] This application is a non-provisional application filed on September 26, 2019, in the United States, entitled “System and Method for Autonomous System Introspection”, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This invention relates to a system for competency assessment of autonomous systems, and more specifically, to a system for competency assessment of autonomous systems that can initiate a handover to a human or assistive system in the event of predicted low competency. Background Technology

[0004] Autonomous systems need to continuously estimate their ability to handle impending situations. Existing techniques for competence estimation and warning (such as Daftry et al. (see reference No. 9 in the incorporated list of references)) use deep learning, which only works well under well-defined and controlled conditions, is based solely on kinematic projections one frame in advance, and cannot assess competence for hypothetical future situations.

[0005] Kansky et al. (see Reference No. 4) demonstrated that even subtle changes in input can lead to catastrophic failures in deep networks. There are no well-established methods in this field to address the problem of deep networks being overly sensitive to minute changes in input (or low-level features). Therefore, existing techniques cannot evaluate competence for hypothetical and novel situations. Furthermore, existing techniques directly predict biased competence quantities, which can easily lead to response bias. Additionally, existing techniques do not evaluate empirical depth.

[0006] Therefore, a system is still needed that not only learns and adapts resiliently to situations it has not yet been trained or programmed to handle, but also knows which of those situations it is competent to handle and when to seek help. Humans can have more confidence in machines that know their limits. Summary of the Invention

[0007] The present invention relates to a system for competence assessment of an autonomous system, and more particularly, to a system for competence assessment of an autonomous system that is capable of initiating a handover to a human or an assistive system in the event of a predicted low competence. The system includes a non-transitory computer readable medium and one or more processors, the non-transitory computer readable medium having executable instructions encoded thereon such that, when executed, the one or more processors perform a plurality of operations. The system extracts a plurality of semantic concepts representing a situation. An action taken by the autonomous system is associated with a set of semantic concepts that were active at the time the action was taken in the situation. Using a task-specific reward function of performance of the autonomous system, the system measures an outcome of the action taken in the situation and generates a reward metric. The plurality of semantic concepts representing the situation are stored as a memory along with the action taken in the situation and the reward metric. A prospective simulation result is generated based on a recall of the memory. Based on the prospective simulation result, the system determines at least one of a competence metric representing a competence of the autonomous system in the situation and an experience metric representing an experience of the autonomous system in the situation. When at least one of the competence metric and the experience metric is above an established minimum allowable value, a competent operational control of the autonomous system is maintained. When at least one of the competence metric and the experience metric is below the established minimum allowable value, an alert is generated.

[0008] In another aspect, the system determines an experience depth estimate representing a density of visits at any point in a learned latent space conditioned on a task goal.

[0009] In another aspect, the competence metric is a Receiver-Operator Characteristic (ROC) curve based competence metric.

[0010] In another aspect, when at least one of the competence metric and the experience metric is below an established minimum allowable value, the system initiates a handover of control of the autonomous system to a user.

[0011] In another aspect, when at least one of the competence metric and the experience metric is below an established minimum allowable value, the system initiates a handover of control from a human user to an assistive system.

[0012] In another aspect, the autonomous system is an autonomous driving system.

[0013] Finally, the present invention also includes a computer program product and a computer-implemented method. The computer program product includes computer-readable instructions stored on a non-transitory computer-readable medium that can be executed by a computer having one or more processors, such that upon execution of the instructions, the one or more processors perform the operations listed herein. Alternatively, the computer-implemented method includes the act of causing a computer to execute these instructions and perform the resulting operations. BRIEF DESCRIPTION OF DRAWINGS

[0014] The objects, features and advantages of the present invention will be apparent from the following detailed descriptions of the various aspects of the application in conjunction with reference to the following drawings, where:

[0015] Figure 1 is a block diagram depicting components of a system for competency assessment of autonomous systems according to some embodiments of the present disclosure;

[0016] Figure 2 is an exemplification of a computer program product according to some embodiments of the present disclosure;

[0017] Figure 3 is an exemplification of operations of a method for autonomous system introspection according to some embodiments of the present disclosure;

[0018] Figure 4 is an exemplification of training of an action generator according to some embodiments of the present disclosure;

[0019] Figure 5 is an exemplification of how a scenario world model predicts the future based on efficiently encoded past experience according to some embodiments of the present disclosure;

[0020] Figure 6A illustrates real distribution of prospective simulation results and decoy reward distribution for an agent trained on two-hundred million frames according to some embodiments of the present disclosure;

[0021] Figure 6B illustrates real distribution of prospective simulation results and decoy reward distribution for an agent trained on one million frames according to some embodiments of the present disclosure;

[0022] Figure 6C illustrates average receiver operating characteristic (ROC) curves derived from simulation results according to individual ones of twenty real episodes according to some embodiments of the present disclosure; and

[0023] Figure 6D illustrates distribution of area under the curve (AUC) scores from ROC curves for twenty real episodes according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0024] The present invention relates to a system for competency assessment of autonomous systems, and more particularly, to a system for competency assessment of autonomous systems that can learn and adapt to new situations. The following description is presented to enable one of ordinary skill in the art to make and use the invention and to incorporate it in the context of particular applications. Various modifications, as well as a variety of uses in different applications will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to a wide range of aspects. Thus, the present invention is not intended to be limited to the aspects presented, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0025] In the following detailed description, numerous specific details are set forth in order to provide a more thorough understanding of the present invention. However, it will be apparent to one skilled in the art that the present invention can be practiced without necessarily being limited to these specific details. In other instances, well-known structures and devices are shown in block diagram form, rather than in detail, in order to avoid obscuring the present invention.

[0026] The reader's attention is directed to all papers and documents which are filed concurrently with this specification and which are open to public inspection with this specification, and the contents of all such papers and documents are incorporated herein by reference. All the features disclosed in this specification, (including any accompanying claims, abstract, and drawings) can be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise. Thus, unless expressly stated otherwise, each feature disclosed in this specification is one example only of a generic series of equivalent or similar features.

[0027] Before describing the present invention in detail, first a list of references is provided. Next, a description of various main aspects of the present invention is provided. Subsequently, the introductory part provides the reader with a general understanding of the present invention. Finally, specific details of various embodiments of the present invention are provided to enable an understanding of specific aspects.

[0028] (1) Incorporated List of References

[0029] The following references are cited and incorporated throughout this application. For clarity and convenience, these references are listed herein for the reader as a centralized resource. The following references are hereby incorporated by reference as if fully set forth herein. These references are cited in this application by reference to the corresponding literature reference number as follows:

[0030] 1. Dutordoir, V., Salimbeni, H., Deisenroth, M., & Hensman, J. (2018). Gaussian Process Conditional Density Estimation. ArXiv: 1810.12750.

[0031] 2. Fawcett, Tom (2006). "An Introduction to ROC Analysis". Pattern Recognition Letters. 27 (8): 861-874.

[0032] 3. Ketz, N., Kolouri, S., & Pilly, P. (2019). Using World Models for Pseudo-Rehearsal in Continual Learning. ArXiv: 1903.02647.

[0033] 4. Kansky K, Silver T, Mély DA, Eldawy M, Lázaro-Gredilla M, Lou X, Dorfman N, Sidor S, Phoenix S, George D. (2017). "Schema networks: Zero-Shot Transfer with a Generative Causal Model of Intuitive Physics." In Proceedings of the 34th International Conference on Machine Learning. Volume 70: 1809-1818.

[0034] 5. Kolouri, Soheil, Charles E. Martin, and Heiko Hoffmann. (2017). "Explaining Distributed Neural Activations via Unsupervised Learning." In CVPR Workshop on Explainable Computer Vision and Job Candidate Screening Competition, vol. 2.

[0035] 6. Liou, Cheng-Yuan; Huang, Jau-Chi; Yang, Wen-Chie. (2008). "Modeling Word Perception Using the Elman Network". Neurocomputing. 71 (16-18): 3150.

[0036] 7. Merrild, J., Rasmussen, M. A., & Risi, S. (2018). "HyperNTM: Evolving Scalable Neural Turing Machines through HyperNEAT." International Conference on the Applications of Evolutionary Computation, 750-766.

[0037] 8. Daftry, S., Zeng, S., Bagnell, J. A., and Hebert, M. (2016). "Introspective Perception: Learning to Predict Failures in Vision Systems." In 2016 IEEE / RSJ International Conference on Intelligent Robots and Systems (IROS), 1743-1750.

[0038] 9. Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G. and Petersen, S. (2015). "Human-Level Control Through Deep Reinforcement Learning." Nature, 518(7540), 529-533.

[0039] 10. Miikkulainen, R., Liang, J., Meyerson, E., Rawal, A., Fink, D., Francon, O., and Hodjat, B. (2019). “Evolving Deep Neural Networks.” In Artificial Intelligence in the Age of Neural Networks and Brain Computing, 293-312.

[0040] 11. Pilly, P. K., Howard, M. D., and Bhattacharyya, R. (2018). “Modeling Contextual Modulation of Memory Associations in the Hippocampus.” Frontiers in Human Neuroscience, 12.

[0041] (1) Main Aspects

[0042] Various embodiments of the present invention include three “main” aspects. The first main aspect is a system for competency assessment of autonomous systems. The system is typically in the form of computer system operating software or in the form of a “hard-coded” instruction set. The system can be incorporated into a wide variety of devices that provide different functionality. The second main aspect is a method, typically in the form of software, that is operated using a data processing system (computer). The third main aspect is a computer program product. The computer program product typically represents computer-readable instructions stored on a non-transitory computer-readable medium such as an optical storage device (e.g., a compact disc (CD) or a digital versatile disc (DVD)) or a magnetic storage device such as a floppy disk or magnetic tape. Other, non-limiting examples of computer- readable media include a hard disk, a read-only memory (ROM), and flash-type memories. These aspects will be described in more detail below.

[0043] Figure 1A block diagram depicting an example of a system (i.e., computer system 100) of the present application is provided. The computer system 100 is configured to perform calculations, processes, operations, and / or functions associated with programs or algorithms. In one aspect, certain processes and steps discussed herein are implemented as a series of instructions (e.g., software program) resident in a computer- readable memory unit and executed by one or more processors of the computer system 100. When executed, the instructions cause the computer system 100 to perform particular actions and exhibit particular behavior, as described herein. In various aspects, the computer system 100 can be embodied in any device or devices operable to perform functions suitable for a particular application as described herein, such as a desktop computer, a mobile or smart phone, a tablet computer, a computer embodied in a mobile platform (e.g., one or more autonomous vehicles), or any other device or devices that can individually and / or collectively execute instructions to perform relevant operations / processes.

[0044] The computer system 100 can include an address / data bus 102 that is configured to communicate information. Additionally, one or more data processing units, such as a processor 104 (or processors), are coupled with the address / data bus 102. The processor 104 is configured to process information and instructions. In an aspect, the processor 104 is a microprocessor. Alternatively, the processor 104 can be a different type of processor, such as a parallel processor, application-specific integrated circuit (ASIC), programmable logic array (PLA), complex programmable logic device (CPLD), or field programmable gate array (FPGA).

[0045] The computer system 100 is configured to utilize one or more data storage units. The computer system 100 can include a volatile memory unit 106 (e.g., random access memory ("RAM"), static RAM, dynamic RAM, etc.) coupled with the address / data bus 102, wherein a volatile memory unit 106 is configured to store information and instructions for the processor 104. The computer system 100 further can include a non-volatile memory unit 108 (e.g., read-only memory ("ROM"), programmable ROM ("PROM"), erasable programmable ROM ("EPROM"), electrically erasable programmable ROM "EEPROM"), flash memory, etc.) coupled with the address / data bus 102, wherein the non-volatile memory unit 108 is configured to store static information and instructions for the processor 104. Alternatively, the computer system 100 can execute instructions retrieved from an online data storage unit such as in "Cloud" computing. In an aspect, the computer system 100 also can include one or more interfaces, such as an interface 110, coupled with the address / data bus 102. The one or more interfaces are configured to enable the computer system 100 to interface to other electronic devices and systems. The communication interface or interfaces that can be implemented by the one or more interfaces can include wireline (e.g., serial cables, modems, network adaptors, etc.) and / or wireless (e.g., wireless modems, wireless network adaptors, etc.) communication technology.

[0046] In one aspect, the computer system 100 can include an input device 112 coupled with the address / data bus 102, wherein the input device 112 is configured to communicate information and command selections to the processor 104. According to one aspect, the input device 112 is an alphanumeric input device, such as a keyboard, that can include alphanumeric keys and / or function keys. Alternatively, the input device 112 can be an input device other than an alphanumeric input device. In an aspect, the computer system 100 can include a cursor control device 114 coupled with the address / data bus 102, wherein the cursor control device 114 is configured to communicate user input information and / or command selections to the processor 104. In an aspect, the cursor control device 114 is implemented using a device such as a mouse, a trackball, a trackpad, an optical tracking device, or a touch screen. Although the foregoing

[0047] In one aspect, computer system 100 can also include one or more optional computer usable data storage devices such as storage devices 116 coupled with address / data bus 102. Storage devices 116 are configured to store information and / or computer executable instructions. In one aspect, storage devices 116 are implemented as a storage subsystem that includes a magnetic hard disk drive, a floppy disk drive along with associated storage media, a Compact Disk (CD) drive along with associated storage media, a Digital Video Disk (DVD) drive along with associated storage media, and / or a tape drive along with associated storage media. In one aspect, storage devices 116 are configured to store operating system 126, application programs 128, other program modules, and / or data.

[0048] Computer system 100 presented herein is an example computing environment in accordance with one aspect. However, the non-limiting example of computer system 100 is not strictly limited to being a computer system. For example, one aspect provides that computer system 100 represents a class of data processing analyses that can be used in accordance with various aspects described herein. Additionally, other computing systems can be implemented. Indeed, the spirit and scope of the technology is not limited to any single data processing environment. Thus, in one aspect, one or more operations of various aspects of the technology are controlled or implemented using computer-executable instructions, such as program modules, being executed by a computer. In one implementation, such program modules include routines, programs, objects, components, and / or data structures that are configured to perform particular tasks or implement particular abstract data types. Additionally, one aspect provides that one or more aspects of the technology are implemented using one or more distributed computing environments, such as where tasks are performed by remote processing devices that are linked through a communications network, or such as where various program modules are located in both local and remote computer-storage media including memory-storage devices.

[0049] Figure 2An exemplary diagram of a computer program product (i.e., storage device) embodying the present application is depicted. The computer program product is depicted as a floppy disk 200 or an optical disk 202 such as a CD or DVD. However, as previously mentioned, the computer program product generally represents computer-readable instructions stored on any compatible non-transitory computer-readable medium. The term "instructions" as used with respect to this application generally indicates a set of operations to be performed on a computer, and can represent pieces of a whole program or a single, independent software module. Non-limiting examples of "instructions" include computer program code (source or object code) and "hard-coded" electronics (i.e., computer operations encoded into a computer chip). The "instructions" are stored on any non-transitory computer-readable medium, such as a hard disk, a floppy disk, a CD-ROM, a flash drive, or other storage device. In any case, the instructions must be present on a storage medium or device that is non-transitory.

[0050] (3) Specific details of various embodiments

[0051] Autonomous systems need to constantly estimate their ability to handle upcoming situations (i.e., instantaneous proficiency level). The present application described herein predicts performance in familiar and novel situations, and if necessary, has enough time (e.g., 10 seconds or more for an autonomous driving car) to hand over control to a human user. The state-of-the-art (SOA) of proficiency estimation and warning is based only on kinematic projections one frame ahead, and cannot assess proficiency for hypothetical future situations (see reference No. 8).

[0052] The invention described herein continuously stores all sequentially occurring experiences in a compact, scalable episodic memory. With enough experiences, the invention can initiate the process of learning concepts in order to facilitate generalization beyond training. Concepts are then continuously integrated into semantic memory. When encountering novel situations, the novel situations are decomposed into their constituent concepts, and then the invention uses both semantic and episodic memory to perform a series of stochastic prospective simulations for both the target task purpose and the decoy task purpose. As the simulations project into the future, two dynamic performance estimate distributions (target and decoy) are produced. These are analogous to the signal and noise distributions in signal detection theory. The farther apart these two distributions are, the higher the confidence is about the task competence. This is captured in the unbiased area under the receiver operating characteristic (ROC) curve metric, also known as the area under the curve or AUC, which is a single number that captures the system's competence in similar situations. Thus, a competence rating can be obtained by performing ROC curve analysis on the stochastic episodic recall. Using the invention, a competence rating can be continuously produced during vehicle operation, and when it falls below a minimum acceptable value, the control system should hand over control to the user. Thus, the invention is useful as part of a control system for protecting the safety of the user and the autonomous system when the competence rating is low.

[0053] In another non-limiting example, the invention according to embodiments of the present disclosure can predict the performance of a human user, and predict the human's ability to successfully and safely perform operations in familiar and novel situations. In this case, the competence rating is a control signal to an assistive system to tell the assistive system when to assist the user, or possibly take over control in unsafe situations. For example, if the human user's competence rating falls below a certain threshold, then the assistive system (or backup system) takes over. Thus, two instances of the invention are possible: one instance introspects an autonomous system (such as a vehicle), and the other instance introspects a user (such as a driver or pilot).

[0054] The invention described herein records the experience of a system (e.g., a land, sea, or air vehicle or a human) and then introspects by recalling relevant episodes to compute a measure of the system's competence to handle a particular current situation or hypothesized situation. Episodic and semantic memories can be stored implicitly or explicitly. The term "system" is used to describe anything that acts in an environment. The invention learns how the system (i.e., a controller or actor) acts in different situations and then is able to predict the system's competence in new situations; note that the system can be a land, sea, or air vehicle (autonomous or semi-autonomous) or a human. The invention described herein can be applied to anything that acts in the world, including an adult, a child (e.g., a child safety monitor), an autonomous vehicle, a non-autonomous vehicle (e.g., can the vehicle travel over rough mud without breaking down?), a robot, or more abstractly a team of individuals (such as an army infantry unit facing a situation involving a particular type of weapon, a number of hostile combatants, and a type of terrain).

[0055] A prior art method called deep sense learning (DSL) (element 322) is used to extract semantic concepts (element 320) that are used in the meta-knowledge representation of any situation. A brain-inspired episodic world model (element 318) is used to store in a compact, scalable form the episodic and semantic concepts (element 320) that are experienced in sequence by the system and the actions (element 316) taken in response (e.g., steering wheel angle, throttle, brake parameters) as well as any rewards (element 404) received for the actions. As a non-limiting example, in an autonomous car, the reward for a lane tracking task is how far from the centerline of the car's lane the car is.

[0056] The episodic sequence of all sensor observations, actions, and instantaneous performance metrics is an example of the recorded experience. In addition, known semantic concepts extracted from the raw data at different granularities (e.g., the region of construction, the entities involved) can be recorded. The invention described herein provides a way to keep the set of concepts consistent as new experiences are encoded. The actions (element 316) taken by the system are stored in the action generator (AG) module (element 314) in association with the semantic concepts (element 320) that were active when the action was taken. The learned semantic concepts (element 320) are selectively activated by the current input (i.e., sensor data (element 312)). For example, the invention has learned many concepts over its lifetime; however, only a subset of semantic concepts are relevant to a particular scene or frame.

[0057] Further, as the AG module (element 314) is exposed to more and more situations, it is able to generate generative actions (element 316) that better mimic and anticipate the controller actions (element 302) taken by the system (which includes the PACS (300) and the vehicle (304)). The invention, including the EWM (element 318), the DSL (element 322), and the AG module (element 314), learns to mimic the behavior of the autonomous control system (element 300) of the platform. Machine learning, including reinforcement learning, typically maps states / raw inputs (sensor measurements) to actions. The invention described herein additionally extracts semantic concepts (element 320) that are selectively activated by the states / raw inputs and maps them to actions. Because the system has semantic concepts (element 320) in memory, it can also perform ROC curve analysis to evaluate competency for hypothetical situations, e.g., it can be assigned to "follow a leader" on "muddy roads" and on "rainy days". Each of these aspects will be described in more detail below.

[0058] The prior art uses measures that suffer from response bias (i.e., probability of success). In contrast, the invention described herein includes methods that generate prospective simulation results from these memories and use these simulation results to construct unbiased subject-operator characteristic (ROC) curve-based measures of competency. The ROC curve is a prior art graphical method that illustrates the diagnostic ability of a binary classifier system as its discrimination threshold varies (see reference No. 2). Additionally, the invention is used to prospectively assess whether desired task goals will continue to be met and to quantify expected deviations. Finally, the invention provides a method to estimate the experience depth of the system by assessing the access density at any point in the learned latent space under the given task goal conditions.

[0059] The invention described herein generates prospective simulation results based on stored scenario sequences, resulting in comparative ROC curve competency measures (i.e., success of a task relative to other tasks) that provide more information than mere probability of success. It represents the situation in terms of semantic concepts, which makes it more generalizable and robust. The invention is a complement to the underlying machine learning system (hereinafter "system"), such as Figure 3The platform autonomous control system (PACS) (element 300) shown in the middle is trained to accomplish a set of different tasks (i.e., different operational objectives or commands). The system can be a deep reinforcement learning neural network or a rule-based expert system, or indeed a human; in any case, the system is a black box that maps environment states to controller actions (element 302) that implicitly learns multiple distinct policies that are appropriately deployed under different environmental conditions. The invention then learns how the system produces actions in response to different situations, and the effectiveness of those actions. Subsequently, when the system does not show itself competent in a given situation, the invention can produce a warning or alarm (e.g., a visual warning on the vehicle display, an audible warning such as a beeping sound or tone).

[0060] Figure 3 Multiple modules that make up the invention are depicted. The modules are used to learn from the environment and estimate the competence of the vehicle (element 304), and produce competence warnings and handover requests (element 306). Note that the roles of the vehicle (element 304) and the user (element 308) can switch for different use cases (not shown), where the system analyzed by the invention is the user (element 308), and the invention estimates the user's competence and notifies the vehicle when the user needs assistance.

[0061] (3.1) Perception module (element 310)

[0062] The perception module (element 310) encodes the sensor data (element 312) into a latent representation using an existing technology autoencoder (element 400) in Figure 4 An autoencoder is an existing technology artificial neural network used to learn an effective encoding. They are well known to those skilled in the art. The purpose of an autoencoder is to learn a representation (encoding) of a set of data, usually for dimensionality reduction. See reference No. 6 for a detailed description of autoencoding using Elman networks.

[0063] The external inputs to the perception module (element 310) include low-level signals (i.e., sensor data (element 312)) from various sensors such as cameras, LIDAR, RADAR, GPS, and weather. The internal sensing (proprioception) of the system with respect to the state of the external environment, such as the vehicle's speed and relative position in the world, is another input to the perception module (element 310).

[0064] (3.2) Action generator (element 314)

[0065] The output of the perception module (element 310), which is a latent representation of the sensor data, enters the action generator (element 314), which learns a mapping from the perception module (element 310) context to the actions (element 302) taken by the vehicle (element 304). The action generator (element 314) module is implemented by an actor-critic model (see reference No. 9), which is a well-known reinforcement learning method for systems to act, and based on the rewards produced, the critic system adapts the actor, implemented by an evolving recurrent actor-critic network, to learn reward-driven actions for various tasks based on situational context and memory recall. Neuroevolution is used to optimize the perception-action module of the action generator (element 314), CoDeepNEAT (see reference No. 10) for direct encoding and HyperNEAT (see reference No. 7) for indirect encoding. The action selection (element 316) is output to the episodic world model (element 318).

[0066] Figure 4 How the action generator (element 314) is trained is illustrated. During the first offline period, the online experience log is used for the evolutionary optimization of the modular structure of the action generator (element 314) to match the actions of the system (e.g., vehicle (element 304)) as much as possible on the online data. Concepts (element 320) are extracted by the DSL (element 322) from the activations / activities (element 324) in the action generator (element 314) on the same data.

[0067] (3.3) Deep Sensing Learning (DSL) (element 322)

[0068] Deep Sensing Learning (DSL) (element 322) is a prior art system by Kolouri et al. (see reference No. 5) that is used to extract and cluster activities in the action generator (element 314) network, as Figure 3 and Figure 4These activity clusters (i.e., activities (element 324)) are used as symbolic representations of the inputs that elicited them, and remain consistent across multiple experiences as novel elements are learned. Concepts need not be semantically meaningful; but since similar (but not identical) inputs can produce similar activity clusters, they serve to generalize the encoded sensor data from the perception module (element 310). When new features in the environment are sufficiently different that the activity clusters in the action generator network (element 314) change significantly, all networks that rely on DSL concepts (element 320) are retrained and reorganized to incorporate the new concepts (element 406) without forgetting previous concepts. This is done offline using interleaved training with generated data that combines the latest data and previous experiences. Random noise in the input layer is used to trigger simulated rollouts of episodes from previous experiences based on the merged associations between action generator (element 314) activities and concepts (see the prospective simulation section). Kolouri et al. (see Reference No. 5) demonstrated that, due to the generalization capability it provides, classification error was reduced by 42.5% when incorporating DSL-extracted concepts into deep networks.

[0069] The low-dimensional latent space representation of the environment provided by the perception module (element 310) is fed into the EWM (element 318), which stores sequentially occurring episodic experiences associated with actions (element 316) taken by the system during these experiences, as well as normalized reward values, as Figure 5 The EWM (element 318) consists of an autoregressive recurrent network and a discretized index code that restricts the simulation outcomes to be context-dependent (see Reference No. 12). By basing the internal model of the invention on the memory systems of mammals, past experiences can be generated to assess performance related to any condition of interest, and provide a constrained estimate of performance in new scenarios that are related but unseen. These generated experiences take the form of prospective simulation outcomes based on some initial conditions, and help to robustly assess competency for various strategies against these conditions. By generating highly accurate simulation outcomes that can be recursively propagated for many time steps, we can improve the fidelity of competency measures and increase the reliability of system behavior. The innovation of the described invention is to continuously incorporate discovered meta-knowledge into the EWM (element 318) to improve the accuracy of simulations of previous experiences and improve generalization to new scenarios.

[0070] (3.4) Episodic World Model (EWM) (element 318)

[0071] The EWM (element 318) models the effective forward model of the world dynamics conditioned on the system actions. When the input state (in the form of a series of possible concept distributions conditioned on the current action) is presented together with the associated set of actions, the EWM (element 318) predicts the next reward (t+1) (element 402), the next concept (t+1) (element 403), and the termination of the current episode. t+1 refers to the next frame in discrete time. It depends on the running speed of the autonomous system (e.g., 30 hertz for an autonomous driving car). Each rollout runs in the compressed low-dimensional latent space of the previous experiences. These experiences are encoded into a heterogeneous network that mimics the CA3 and dentate gyrus (DG) circuits in the hippocampus to achieve pattern completion and separation capabilities, respectively (see references No. 3 and No. 12).

[0072] The task-specific reward function generates the reward (element 404), which is a per-time evaluation of the system performance. As an example of a reward (element 404) function for an autonomous driving system, the driving rules learned in the driver training courses and manuals and embodied in the signs and road markings can be used to supply a score for the safe operation of the vehicle (element 304). A more advanced reward (element 404) function can add goal-oriented rewards (e.g., how efficiently the vehicle reached the target location). The EWM (element 318) is prior art by the authors of the present disclosure, implemented by an interconnected recurrent heterogeneous network (see reference No. 3). During the first online period after applying the present invention to the system, the training of the EWM (element 318) is done in a self-supervised manner using the input perception results (element 406) in the perception module (element 310), which are encoded by the autoencoder (element 400) into latent representations concepts to produce encoded perception results (element 401). The training of the EWM (element 318) is done in a self-supervised manner (i.e., by minimizing the reconstruction error (also known as prediction error) of each frame), as shown. Figure 4

[0073] The present invention facilitates incremental learning and avoids catastrophic forgetting by additionally using all previous experience data from the online period that can be stored in an explicit memory buffer and from retrospective simulation results (replays) based on the latest EWM (element 318) and action generator (element 314) for various random initial conditions and different task goals. Furthermore, the prediction error (“surprise”) is used as a multiplier for the encoding strength of the experience in the EWM (element 318), whereby frames with high prediction error are presented multiple times until a performance criterion is reached.

[0074] ​During the second online period, the system's raw experience continues to be recorded in a FIFO (first-in-first-out) memory buffer. Then offline, the autoencoder (element 400) is updated with the new data and previous data. Next, the action generator (element 314) is re-optimized, which uses the action generator (element 314) evolved during the first offline period to generate seeds for the network for a new cycle of neuroevolution. After this step, the concept, rule list, and policy explanations are re-extracted. Given that the autoencoder (element 400), action generator (element 314), and concepts (element 320) can all have been updated, especially as the content and number of meta-knowledge concepts change, the EWM (element 318) also needs to be reorganized by retraining with the experience in the memory buffer. This process continues during subsequent online and offline periods, with the expectation that the autoencoder (element 400), action generator (element 314), concepts (element 320), and EWM (element 318) eventually stabilize to a core set of experiences and explore the task policy of the training system. However, the invention is always able to incorporate any new strategies exhibited by the system, as well as any new meta-knowledge encountered during new experiences.

[0075] (3.5) Prospective Simulation Results Analysis (element 500) and Competency Estimation (elements 506 and 508)

[0076] The EWM (element 318) generates a forward projection of performance based on past experience from any condition of interest and provides a constrained estimate of performance in new scenarios that are related but unseen. These generated experiences help to make a robust competency assessment of the system's ability to achieve the desired performance metrics in the near future, as Figure 5each rollout operates within the compressed low-dimensional latent space memorized by the EWM (element 318), where the next state (element 326) is modeled stochastically in the form of a probability distribution aided by an index code. The initial state is encoded as a latent vector, and a particular action is sampled from the action generator (element 314). The sample generated according to the predictive distribution is saved and used as input for the next time step (i.e., t+1), which in turn will sample a new action and next state (element 326). This process can continue ad-hoc as needed until the end of the scenario is predicted or a certain prescribed length is reached. The various memory recalls from the EWM (element 318) during the simulation include reward metrics (described in detail below), and these metrics are combined to form a final cumulative reward value. There can be different ways to calculate the cumulative reward for a particular simulation, which can depend on the type of reward or task. For example, if the reward for an autonomous vehicle is to obey road rules and keep the driver safe, then the cumulative reward can be the minimum reward value reached during the simulation. However, if the reward is simply to reach the destination, then the cumulative reward will be the maximum reward value reached. The results of each simulation are tallied in a labeled histogram, as Figure 5 The decoy and target simulations are run, where the target simulation uses the actions (element 316) selected by the action generator (element 314) during the simulation, while the decoy simulation randomly selects from any action other than the one selected by the action generator (element 314). The ROC curve (element 502) is constructed in the standard way by plotting the true positives against the false positives for each reward, where the true positives are the values from the target simulation, and the false positives are the values from the decoy. The ROC curve (element 502) plots the true positive rate against the false positive rate. The conventional measure of the area under the curve (AUC) (element 504) is then applied, revealing how likely the system is to achieve good results in the future of the simulation.

[0077] The EWM (element 318) predicts the future based on the efficiently encoded past experience. The latent representation conditioned on the current action is used to learn a predictive distribution of the possible next state. Once encoded in the latent space, many forward-looking simulation results can be generated by feeding back the stochastic temporal predictions of the EWM (element 318) to itself. The cumulative reward distributions for the target and decoy tasks in these simulation results will determine the ROC curve for each policy, and the AUC measure indicates the competency of the system over the simulated time period.

[0078] Future competency assessments (i.e., competency awareness (element 506)) are maintained only for a period of time limited to a maximum length of accurate predictions from the system. Prediction accuracy is a function of the depth of experience stored in the invention (as described below), the frame rate or length of time of each iteration stored in the EWM (element 318), the randomness of potential action samples from each iteration by the action generator (element 314), and the number of forward looking simulation iterations into the future.

[0079] A pilot study was conducted to explore the effectiveness of these forward looking simulations in preserving existing knowledge in incremental learning services (see Reference No. 3). Results are shown in Figures 6A to 6D Here, using the Atari set of games as tasks, it was found that interleaving forward looking simulations of previously learned tasks with new tasks can preserve learned temporal predictions across several tasks. Figure 6A and Figure 6B Real reward distributions and lure reward distributions are shown for forward looking simulation results for agents trained on two hundred million frames Figure 6A ( Figure 6B ) and separate agents trained on less than one million frames Figure 6A In Figure 6B , the unfilled solid area (element 606) represents the real reward distribution for an agent trained on one million (1M) frames, the unfilled dashed area (element 608) represents the lure reward distribution, and the filled area (element 610) represents the overlap area.

[0080] Figure 6C Average ROC curves are shown for 20 simulations of 50 time steps for each of 20 real episodes for agents trained on 20M frames (element 612) and 1M frames (element 614). The dashed line (element 616) represents a test of no difference. As Figure 6C shown, forward looking simulations are able to distinguish between the 20M policy and the 1M policy based on the respective ROC curves. Figure 6DThe distribution of AUC scores from the 20 real episodes is depicted, showing that the well-trained agents have significantly higher AUC. The unfilled dashed area (element 618) represents the distribution of agents trained on 20M frames, the unfilled solid area (element 620) represents the distribution of agents trained on 1M frames, and the filled area (element 622) represents the overlap. These plots show that this architecture can model a variety of potential experiences; however, on average, they reflect the true behavior of the environment, making it possible to reliably distinguish between experienced and novice strategies.

[0081] (3.6) Experience Depth Estimator (DoEE) (element 508)

[0082] The DoEE (element 508) provides a reliable estimate of the frequency with which a strategy is used in a particular task objective service under given environmental conditions. This is learned online using an Evolutionary Neural Turing Machine (ENTM) (see Reference No. 7) by estimating the access density to any point in the learned latent space that is influenced by a given task objective and strategy, while also enabling robust generalization across very similar states. The ENTM is trained to mimic a Gaussian Process (GP) model of access density. Due to its evolutionary optimization and Turing-complete properties, the ENTM is trained to learn a universal algorithm to provide smooth access estimates influenced by a given task and strategy. The GP model can be iteratively trained on a data set comprising a series of observations in the EWM (element 318) latent space that are obtained using a particular strategy to complete a particular task. The ENTM stores the fact that these observations have been obtained and minimizes a loss based on the distance between its estimates and the GP estimates of access density. In the next iteration of this process, the data set is augmented with new observations, a new GP is trained on this new data set, the ENTM is provided with the new observations only, and its ability to approximate the latest iteration of the GP access density is again evaluated. This iterative training process can continue indefinitely, with the end result being an ENTM that uses its internal memory to provide conditional access density estimates in an online manner. Due to the Bayesian nature of the GP model, it can be used with very little available data. This also has the additional benefit of explicitly measuring the uncertainty of its density estimates. The GP model itself can also be replaced by the ENTM by employing an online batch learning approach that does not require the storage of a complete data history (Reference No. 1).

[0083] (3.7) Competency Awareness (CA) (element 506)

[0084] The Competency Awareness (element 506) module takes as input the normalized Competency Assessment Metric (CAM) (element 328) and the Experience Depth Estimate (DoEE) and compares the results to a user-defined minimum required competency level. One approach is to multiply them (e.g., Competency = CAM * norm(DoEE)). The DoEE can be normalized by establishing some desirable DoEE (e.g., norm(DoEE) = min(l, DoEE / (desirable_DoEE)). In one embodiment, the CAM and DoEE are treated separately. A minimum allowable level of the CAM is established, and a minimum allowable level of the DoEE is established. If either of these metrics falls below its established minimum, the CA (element 506) module issues a warning or alert, and / or actively initiates a handover of control of the autonomous system (element 306) to the user. For example, in an autonomous driving system (e.g., an autonomous vehicle), the handover of control (element 306) can mean that one or more vehicle components (e.g., braking mechanisms, steering mechanisms, acceleration mechanisms) are no longer controlled by the autonomous system, and the user needs to control the vehicle components to drive and maneuver the vehicle.

[0085] The semi-autonomous control system includes a driver safety and support system. Developers of autonomous vehicles will establish a minimum acceptable proficiency rating (MAPR) for their vehicles that will eliminate accidents caused by controllers that are not proficient in the scenario. The invention described herein will allow human users to confidently engage the autonomous control system because they know that their autonomous vehicle will not attempt situations that they are not equipped to handle. This is especially important in complex, time-critical, dynamic environments. Developers of semi-autonomous vehicles also need a way to determine when a human user is not as proficient in handling a current impending situation based on their behavior in similar previous situations and can benefit from the assistance technology that takes over partial or full control of the vehicle to ensure the safety of the occupants. Thus, the invention described herein results in an override of the safety and / or efficacy of the autonomous system or the human user when at least one of the competency metric and the experience metric falls below the established minimum allowable value.

[0086] Further, the level of autonomy can be an issue in certain systems where a variable autonomous system, such as an unmanned aerial vehicle, can sometimes fly to a location without supervision, but an operator must maintain control throughout the time in case of a problem. The inventive system according to embodiments of the present disclosure will allow operators to increase the level of autonomy of these systems for longer periods of time because they will be more able to judge their ability to handle situations. For a long time, there has been a need for an assistance system, such as a "pilot assistant," that can intelligently decide how much help a pilot or driver needs. The invention described herein can be used to better predict when a user does indeed need help, and thus better accept assistance technology.

[0087] Finally, while the present application has been described in terms of several embodiments, those of ordinary skill in the art will recognize that the application can be practiced with other embodiments that are within the scope of the claims. It should be noted that there are many alternative ways of implementing both the process and apparatus according to the present application. Furthermore, it should be noted that the application is defined by the appended claims and not by the specific embodiments described herein. Specifically, any recitations of "means for" are intended to evoke a claim limitation of a means-plus-function, and not a step-plus-function limitation as it is otherwise used, such as "means for" concluding dependent claims. Any elements that do not specifically use the words "means for" should not be read as a means-plus-function element, even if the claim otherwise includes the words "means for." Furthermore, while specific steps are recited in a particular order, these steps can be performed in any desired order and fall within the scope of the application.

Claims

1. A system for safe and effective overriding of an autonomous system, the system comprising: A non-transitory computer-readable medium and one or more processors, wherein executable instructions are encoded on the non-transitory computer-readable medium such that, when the executable instructions are executed, the one or more processors perform the following operations: Extract multiple semantic concepts representing the context; Associating the actions taken by the autonomous system with the set of semantic concepts activated when the actions are taken in the context; Using a task-specific reward function based on the performance of the autonomous system, the results of the actions taken in the situation are measured and a reward metric is generated. The multiple semantic concepts representing the situation, along with the actions taken in the situation and the reward metric, are stored together as sequentially occurring episode experiences; Decompose the new context into a set of semantic concepts; The following simulation operation is performed using the set of semantic concepts and the stored sequentially occurring episode experiences: The objective simulation is based on a realistic reward distribution generated by the actions taken by the autonomous system in the new situation. Decoy simulation, wherein the decoy simulation generates a decoy reward distribution based on any action other than that taken by the autonomous system in the new situation; Based on the true reward distribution and the decoy reward distribution, a subject-operator characteristic ROC curve is constructed, and the area under the curve (AUC) is calculated as a measure of the autonomous system's competence in handling the new situation. When the competence level exceeds the established minimum allowable value, maintain competence operation control over the autonomous system. as well as An alarm is generated when the competence level is lower than the established minimum allowable value.

2. The system according to claim 1, wherein, The one or more processors further perform the following operations: determine an empirical depth estimation result DoEE, wherein the DoEE is an estimate of the frequency with which the autonomous system has taken action in a given situation.

3. The system according to claim 1, wherein, The one or more processors further perform the following operations: when the competence level is lower than the established minimum allowable value, initiate the transfer of control of the autonomous system to a human user.

4. The system according to claim 1, wherein, The one or more processors further perform the following operations: when the competence level is below the established minimum allowable value, initiate the transfer of control from the human user to the assistance system.

5. The system according to claim 1, wherein, The autonomous system mentioned is an autonomous driving system.

6. A computer-based method for securely and effectively controlling an autonomous system, the method comprising the following actions: To cause one or more processors to execute instructions encoded on a non-transitory computer-readable medium, such that, when executed, the one or more processors perform the following operations: Extract multiple semantic concepts representing the context; Associating the actions taken by the autonomous system with the set of semantic concepts activated when the actions are taken in the context; Using a task-specific reward function based on the performance of the autonomous system, the results of the actions taken in the situation are measured and a reward metric is generated. The multiple semantic concepts representing the situation, along with the actions taken in the situation and the reward metric, are stored together as sequentially occurring episode experiences; Decompose the new context into a set of semantic concepts; The following simulation operation is performed using the set of semantic concepts and the stored sequentially occurring episode experiences: The objective simulation is based on a realistic reward distribution generated by the actions taken by the autonomous system in the new situation. Decoy simulation, wherein the decoy simulation generates a decoy reward distribution based on any action other than that taken by the autonomous system in the new situation; Based on the true reward distribution and the decoy reward distribution, a subject-operator characteristic ROC curve is constructed, and the area under the curve (AUC) is calculated as a measure of the autonomous system's competence in handling the new situation. When the competence level exceeds the established minimum allowable value, maintain competence operation control over the autonomous system. as well as An alarm is generated when the competence level is lower than the established minimum allowable value.

7. The method according to claim 6, wherein, The one or more processors further perform the following operations: determine an empirical depth estimation result DoEE, wherein the DoEE is an estimate of the frequency with which the autonomous system has taken action in a given situation.

8. The method according to claim 6, wherein, The one or more processors further perform the following operations: when the competence level is lower than the established minimum allowable value, initiate the transfer of control of the autonomous system to a human user.

9. The method according to claim 6, wherein, The one or more processors further perform the following operations: when the competence level is below the established minimum allowable value, initiate the transfer of control from the human user to the assistance system.

10. The method according to claim 6, wherein, The autonomous system mentioned is an autonomous driving system.

11. A computer program product for secure and effective over-control of an autonomous system, the computer program product comprising: Computer-readable instructions stored on a non-transitory computer-readable medium, which can be executed by a computer having one or more processors to cause the processors to perform the following operations: Extract multiple semantic concepts representing the context; Associating the actions taken by the autonomous system with the set of semantic concepts activated when the actions are taken in the context; Using a task-specific reward function based on the performance of the autonomous system, the results of the actions taken in the situation are measured and a reward metric is generated. The multiple semantic concepts representing the situation, along with the actions taken in the situation and the reward metric, are stored together as sequentially occurring episode experiences; Decompose the new context into a set of semantic concepts; The following simulation operation is performed using the set of semantic concepts and the stored sequentially occurring episode experiences: The objective simulation is based on a realistic reward distribution generated by the actions taken by the autonomous system in the new situation. Decoy simulation, wherein the decoy simulation generates a decoy reward distribution based on any action other than that taken by the autonomous system in the new situation; Based on the true reward distribution and the decoy reward distribution, a subject-operator characteristic ROC curve is constructed, and the area under the curve (AUC) is calculated as a measure of the autonomous system's competence in handling the new situation. When the competence level exceeds the established minimum allowable value, maintain competence operation control over the autonomous system. as well as An alarm is generated when the competence level is lower than the established minimum allowable value.

12. The computer program product of claim 11, further comprising instructions to cause the one or more processors to perform the following operation: determine an empirical depth estimation result DoEE, wherein, The DoEE is an estimate of the frequency with which the autonomous system has taken action in a given situation.

13. The computer program product of claim 11, further comprising instructions to cause the one or more processors to perform the following operation: when the competence magnitude is below the established minimum allowable value, initiate the transfer of control of the autonomous system to a human user.

14. The computer program product of claim 11, further comprising instructions to cause the one or more processors to perform the following operation: when the competence magnitude is below the established minimum allowable value, initiate the transfer of control from the human user to an assistance system.

15. The computer program product according to claim 11, wherein, The autonomous system mentioned is an autonomous driving system.

Images