Explicit ethical machines that use analogous scenarios to provide actionable guardrails

Abstract

The apparatus includes at least one memory (204, 210, 212) configured to store information associated with a current scenario (302a, 402a) to be evaluated by the ML / AI algorithm (112), the information including an initial reward function (302b, 402b) associated with the current scenario. The apparatus also includes at least one processor (202) configured to (i) identify one or more policies (304, 404) associated with one or more prior scenarios similar to the current scenario, (ii) determine one or more differences between parameters optimized by the initial reward function and one or more reward functions (306, 406) associated with the one or more prior scenarios, (iii) modify the initial reward function based on at least one of the determined one or more differences to generate a new reward function (308, 408), and (iv) generate a new policy (310, 410) for the current scenario based on the new reward function.

Description

[Technical field]

[0001] The present disclosure relates generally to machine learning (ML) and other artificial intelligence (AI) systems. More specifically, the present disclosure relates to explicitly ethical machines that use similar scenarios to provide actionable guardrails. [Background technology]

[0002] With the increasing adoption of autonomous and semi-autonomous decision-making algorithms in various domains, it is becoming more common for algorithms to encounter scenarios involving ethical dilemmas. One example of an ethical dilemma is frequently referred to as the "trolley problem," which generally considers a hypothetical situation in which a runaway trolley is heading towards multiple people in one track, but can also be deflected into another track occupied by a single person. Summary of the Invention

[0003] The present disclosure relates to explicit ethical machines that use similar scenarios to provide actionable guardrails.

[0004] In a first embodiment, a method includes obtaining, using at least one processor, information related to a current scenario to be evaluated by a machine learning / artificial intelligence (ML / AI) algorithm, the information including an initial reward function related to the current scenario. The method also includes identifying, using the at least one processor, one or more policies associated with one or more prior scenarios similar to the current scenario. The method further includes determining, using the at least one processor, one or more differences between parameters optimized by the initial reward function and the one or more reward functions associated with the one or more prior scenarios. The method also includes modifying, using the at least one processor, the initial reward function based on at least one of the one or more determined differences to generate a new reward function. Furthermore, the method includes generating, using the at least one processor, a new policy for the current scenario based on the new reward function.

[0005] In a second embodiment, the apparatus includes at least one memory configured to store information associated with a current scenario to be evaluated by the ML / AI algorithm, the information including an initial reward function associated with the current scenario. The apparatus also includes at least one processor configured to identify one or more policies associated with one or more prior scenarios similar to the current scenario, determine one or more differences between parameters optimized by the initial reward function and the one or more reward functions associated with the one or more prior scenarios, modify the initial reward function based on at least one of the determined one or more differences to generate a new reward function, and generate a new policy for the current scenario based on the new reward function.

[0006] In a third embodiment, a non-transitory computer-readable medium includes instructions that, when executed, cause at least one processor to obtain information related to a current scenario to be evaluated by an ML / AI algorithm, the information including an initial reward function related to the current scenario. The medium also includes instructions that, when executed, cause the at least one processor to identify one or more policies associated with one or more prior scenarios similar to the current scenario. The medium further includes instructions that, when executed, cause the at least one processor to determine one or more differences between parameters optimized by the initial reward function and the one or more reward functions associated with the one or more prior scenarios. The medium also includes instructions that, when executed, cause the at least one processor to modify the initial reward function based on at least one of the one or more determined differences to generate a new reward function. The medium further includes instructions that, when executed, cause the at least one processor to generate a new policy for the current scenario based on the new reward function.

[0007] Other technical features will be readily apparent to those skilled in the art from the following drawings, descriptions, and claims.

[0008] For a more complete understanding of the present disclosure, reference is now made to the following descriptions taken in conjunction with the accompanying drawings, in which: [Brief description of the drawings]

[0009] [Figure 1] 1 illustrates an exemplary architecture for providing an explicitly ethical machine using a similar scenario according to the present disclosure. [Diagram 2] 1 illustrates an exemplary device supporting an explicitly ethical machine using a similar scenario according to the present disclosure. [Diagram 3] 1 illustrates an exemplary use of a single-agent architecture to provide an explicitly ethical machine using a similar scenario according to the present disclosure. [Figure 4] 1 illustrates an exemplary use of a multi-agent architecture to provide an explicitly ethical machine using a similar scenario according to the present disclosure. [Diagram 5] 1 illustrates an example technique for using inverse reinforcement learning in an architecture for providing an explicitly ethical machine using a similar scenario according to the present disclosure. [Figure 6] 1 illustrates an exemplary method for providing an explicitly ethical machine using a similar scenario according to the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] 1 to 6 described below and various embodiments used to explain the principles of the present disclosure are illustrative only and should not be construed as being intended to limit the scope of the present disclosure in any way. Those skilled in the art will understand that the basic principles of the present disclosure can be implemented in any type of appropriately configured device or system.

[0011] As mentioned above, with the increasing adoption of autonomous and semi-autonomous decision-making algorithms in various fields, it is common for the algorithms to encounter scenarios involving ethical dilemmas. One example of an ethical dilemma is frequently referred to as the "trolley problem," which generally considers a hypothetical situation in which a runaway trolley is heading towards multiple people in one track, but can also be deflected into another track occupied by a single person. These and other types of ethical dilemmas can arise for various decision-making algorithms, such as in the design of control systems for self-driving cars. Unfortunately, complex autonomous and semi-autonomous systems routinely use machine learning (ML) or other artificial intelligence (AI) algorithms that provide limited transparency to operators, essentially functioning as "black boxes." As a result, system engineers frequently end up designing requirements and system behavior into defined parameters that are then expected to operate correctly when in use.

[0012] Current approaches used in autonomous and semi-autonomous systems often rely on closed-world assumptions that encode a desired course of action for a given scenario. However, these approaches are often very fragile and do not scale to even minor changes in the environment or scenario, making these systems unreliable and, in effect, prone to catastrophic errors. The U.S. Department of Defense has adopted five key "ethical principles" for artificial intelligence systems that encompass five key areas, generally categorized as "accountable," "fair," "traceable," "trustworthy," and "controllable." However, the practical implementation of these five principles in the art currently relies on amorphous human judgment that is prone to error and bias. Moreover, it is not always clear whether the course of action to be adopted by an autonomous or semi-autonomous system has any ethical implications.

[0013] The present disclosure provides various techniques for implementing an explicitly ethical machine that can satisfy at least some of these types of principles using similar scenarios to provide actionable guardrails. As described in more detail below, these techniques represent actionable scenarios using a Markov decision process that encodes a goal-specific reward function. During operation, a current scenario can be represented using a Markov decision process that encodes an initial reward function. At least one prior scenario similar to the current scenario can be identified, and at least one policy associated with the similar scenario(s) can also be identified. One or more policies associated with the one or more similar scenarios can be applied to the current scenario, including modifying an initial reward function associated with the current scenario based on one or more policies associated with the one or more similar scenarios. This creates a new policy for the current scenario, where the new policy includes a modified reward function for the current scenario. The new policy for the current scenario can then be used to identify a course of action that results in the current scenario. Interaction with one or more human operators may also occur to assist in the ongoing learning of the characteristics of ethical decisions. For example, to estimate a machine's level of "moral maturity," one or more machine-generated adjustments to a selected course of action in a given scenario can be evaluated against one or more human adjustments.

[0014] In this way, prior experience in the form of similar prior scenarios can be taken, for example using statistical techniques, and similarities and analogies can be drawn for application to new (previously unexplored) scenarios. As a result, these techniques allow ML / AI algorithms to be designed to learn from other scenarios as they attempt to make decisions related to new scenarios for which the algorithm was not previously trained. Thus, similar prior scenarios act as guardrails for the current scenario, helping to ensure that the course of action selected for the current scenario is not catastrophically inaccurate. In essence, this provides a framework that allows ML / AI algorithms to discern and reason about ethics in a consistent and transparent manner. Furthermore, these techniques can be used to clearly discern how an autonomous or semi-autonomous system reaches a given conclusion and select a specific course of action for a given scenario. This can help support traceable, reliable, and controllable operation of autonomous or semi-autonomous systems. Furthermore, these techniques can provide engineers, designers, operators, and other personnel with insight into how autonomy is being used in the system, as well as whether there is over- or under-reliance on autonomy. Furthermore, transparent, trustworthy, and ethically explicit operation of autonomous and semi-autonomous systems can help increase the adoption of the systems in a variety of applications.

[0015] As a specific example of this functionality, an autonomous vehicle control system may require 5 billion miles of driving to demonstrate 95% confidence in its reliability. Using the approaches herein, rare events can be simulated and prior events in similar scenarios can be used to determine how the autonomous vehicle control system responds. Among other things, this allows the autonomous vehicle control system or other autonomous / semi-autonomous systems to self-assess and accelerate validation and assurance evaluations. This also allows engineers, designers, operators or other personnel associated with the autonomous vehicle control system or other autonomous / semi-autonomous systems to predict the ethical impact of possible courses of action based on prior scenarios, even when the current scenario does not appear to involve any ethical considerations.

[0016] FIG. 1 illustrates an exemplary architecture 100 for providing an explicit ethical machine using a similar scenario according to the present disclosure. As shown in FIG. 1, the architecture 100 here includes various operations 102-108 and a database 110 that can be used to train at least one ML / AI algorithm 112 to perform inferences on both task-based decisions and ethical decisions simultaneously. The operations 102-108 here can be performed iteratively, thereby iteratively training the ML / AI algorithm 112 to perform inferences of increasing complexity. Ideally, this allows the ML / AI algorithm 112 to handle increasingly complex task-based ethical decision-making problems.

[0017] Operation 102 generally represents a policy identification operation that includes identifying, generating, or otherwise obtaining an initial policy associated with a current scenario to be evaluated by the ML / AI algorithm 112. In some embodiments, the current scenario may be represented using a Markov decision process (MDP) that includes a set of possible states (S), a set of possible actions (A), and a reward function (R) that defines the reward for transitioning between states via actions. The initial policy represents an initial function that specifies the action to take when in a particular state. The initial policy is typically based on an initial set of one or more parameters to be optimized as defined by the reward function. A combination of the Markov decision process and the policy can identify actions for each state of the process, and ideally the policy can be selected to maximize a function of the reward. In certain embodiments, the Markov decision process can obtain an initial reward function from one or more users that represents the current scenario and identifies one or more parameters to be optimized.

[0018] Operation 104 generally represents a similarity identification operation that includes identifying one or more prior scenarios that are similar in some respect to an initial policy associated with the current scenario being evaluated by the ML / AI algorithm 112. The prior scenarios may be similar to the initial policy with respect to a Markov decision process, a set of parameters to be optimized, or both. Identification of any similar prior scenarios may be done in any suitable manner, for example, using suitable statistical correlation techniques. Operation 104 may output suitable information related to one or more prior scenarios that are similar to the current scenario.

[0019] In this example, operation 104 may include the use of database 110, which may store various information regarding prior scenarios. The information in database 110 may include information regarding actual prior scenarios encountered by ML / AI algorithm 112 (or another ML / AI algorithm). The information in database 110 may also or alternatively include information regarding simulated scenarios, such as scenarios that may be encountered in a given environment. Database 110 may store any suitable information regarding each prior scenario, such as the prior scenario's Markov decision process, reward function, and other information of the prior scenario's policy. The output of operation 104 may include the policy or other information of each similar prior scenario contained in database 110.

[0020] Operation 106 generally represents a conflict identification operation that includes identifying differences (conflicts) between an initial policy associated with the current scenario and a policy associated with each similar prior scenario. For example, the initial policy associated with the current scenario and the policy associated with the similar prior scenario may be related to a similar problem but include different sets of parameters to be optimized (and thus have different reward functions). Thus, operation 106 here can identify reasoning used in one or more similar scenarios that can be applied in the current scenario. Operation 106 can output information identifying the reasoning, for example in the form of similar reward functions.

[0021] In some embodiments, operation 106 includes the use of inverse reinforcement learning (IRL), which is a process of extracting a reward function based on observed behavior. In other words, operation 106 may use IRL here to identify a reward function based on behavior that occurred in one or more similar prior scenarios. Operation 106 may then generate a similar reward function for the current scenario to be evaluated. The similar reward function herein may include various parameters to be optimized, including one or more parameters not included in the original reward function and / or excluding one or more parameters that were included in the original reward function.

[0022] Operation 108 generally represents a reward function modification operation, which includes comparing the initial reward function to the similar reward function to identify any differences. Operation 108 may then generate a new reward function based on the comparison. For example, operation 108 may determine that one or more particular parameters optimized in the similar reward function were not included in the original reward function and should be included. Thus, operation 108 may generate and output a new reward function updated to include the one or more particular parameters. The new reward function may be used to form an updated policy associated with the current scenario being evaluated.

[0023] At this point, human interaction can optionally occur to verify whether the updated policy is acceptable. For example, the updated policy can be provided to the ML / AI algorithm 112 for use in identifying a proposed course of action for the current scenario, and one or more humans can evaluate whether the proposed course of action is acceptable. The updated policy can also be stored in the database 110 along with other information about the current scenario, allowing the updated policy to be used with subsequent scenarios. Additionally, the updated policy can be fed back through another iteration of the process described herein, allowing for iterative updating of the policy. This can allow for more complex situations to be learned as the number of prior scenarios increases.

[0024] Overall, this architecture 100 results in a machine learning framework that can iteratively adapt different reward functions based on increasingly complex reasoning, such as by including more parameters in an iteration. This can generally correspond to human moral development, as defined, for example, by Kohlberg's theory of moral development, which models ethical development that generally occurs in six stages. Stage 1 - Punishment and obedience orientation (following rules to avoid punishment) Stage 2 - Moralistic relativistic orientation (complying to obtain rewards or to gain favor) Stage 3 - Interpersonal conformity orientation (conforming to avoid being rejected by others) Stage 4 - Law and Order Oriented (complying to avoid punishment by authority figures) Stage 5 - Social Contract Orientation (conformity to maintain society) Stage 6 - Universal Ethical Principles Orientation (consider how other people will be affected by the decision) The first and second stages are frequently referred to as "pre-conventional," the third and fourth stages are frequently referred to as "conventional," and the fifth and sixth stages are frequently referred to as "post-conventional." The ability of architecture 100 to perform increasingly complex reasoning aids the ability of ML / AI algorithms 112 to become more effective at making ethical decisions over time, consistent with this type of model (of course, other models may be used to illustrate this principle). By comparing the decisions of ML / AI algorithms 112 and the basis for those decisions against human justification, it is possible to assess the level of ethical maturity of ML / AI algorithms 112.

[0025] Additionally, the architecture 100 herein supports the use of analog reasoning (possibly with heuristics) to identify additional parameters from prior experience, which can provide an ethical justification or explanation for the selected action. This helps support the use of transparent, trustworthy, and ethically explicit behavior by the ML / AI algorithm 112. Additionally, the architecture 100 provides a framework for evaluating the moral reasoning of the ML / AI algorithm 112, rather than simply evaluating the selected actions of the ML / AI algorithm 112. This is useful because the same observable outcome may emerge from different levels of ethical development, such as when the outcome "not stealing" results from low ethical development (to avoid punishment) versus high ethical development (to follow universal ethical principles). The architecture 100 gives the ability to determine or explain why the ML / AI algorithm 112 chooses a selected course of action, rather than simply evaluating whether the ML / AI algorithm 112 chose the "right" course of action. This can help reduce or eliminate catastrophic errors by the ML / AI algorithm 112. Additionally, the architecture 100 herein can be used to train explicit ethical agents that are not entirely rule-based, which helps avoid the pitfalls associated with implicit agents. For example, explicit ethical agents can be extensible, may not rely on designers to predict every possible scenario (thus helping humans with unanticipated risks), and can provide specific explanations of decisions to justify those decisions.

[0026] It should be noted that operations 102-108 described above with reference to FIG. 1 may be implemented in any suitable manner in one or more devices. For example, in some embodiments, operations 102-108 may be implemented using dedicated hardware or using a combination of hardware and software / firmware instructions. Also, in some embodiments, operations 102-108 may be implemented using hardware or hardware and software / firmware instructions embedded in a larger system, such as a system that uses one or more ML / AI algorithms 112 to perform one or more functions. However, the present disclosure is not limited to any particular physical implementation of operations 102-108.

[0027] Although Figure 1 illustrates an example of an architecture 100 for providing an explicitly ethical machine using similar scenarios, various modifications may be made to Figure 1. For example, any suitable number of iterations of the process performed by architecture 100 may occur. Additionally, architecture 100 need not necessarily perform all of operations 102-108 in each iteration; for example, architecture 100 may skip operations 106-108 if no prior scenarios similar to the current scenario have been identified.

[0028] FIG. 2 illustrates an exemplary device 200 for assisting an explicitly ethical machine using a similar scenario according to the present disclosure. One or more examples of the device 200 may be used, for example, to at least partially implement the operations 102-108 illustrated in FIG. 1. However, the operations 102-108 may be implemented in any other suitable manner. According to some embodiments, the device 200 illustrated in FIG. 2 may form at least a part of a computing system, such as a desktop computer, a laptop computer, a server computer, or a tablet computer. However, any other suitable device or devices may be used to perform the operations 102-108.

[0029] 2, device 200 illustrates a computing device including at least one processing device 202, at least one storage device 204, at least one communication unit 206, and at least one input / output (I / O) unit 208. Processing device 202 is capable of executing instructions that can be loaded into memory 210. Processing device 202 includes any suitable number(s) and type(s) of processors or other processing devices in any suitable arrangement. Exemplary types of processing device 202 include one or more microprocessors, microcontrollers, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or discrete circuits.

[0030] Memory 210 and persistent storage 212 are examples of storage device 204, which represents any structure(s) that enable the storage and retrieval of information (such as data, program code, and / or other suitable information on a temporary or persistent basis). Memory 210 may represent random access memory or any other suitable volatile or non-volatile storage device(s). Persistent storage 212 may include one or more components or devices, such as read-only memory, a hard drive, a flash memory, or an optical disk, that support longer-term storage of data.

[0031] The communication unit 206 supports communication with other systems or devices. For example, the communication unit 206 may include a network interface card or a wireless transceiver that facilitates communication over a wired or wireless network or direct connection. The communication unit 206 may support communication over any suitable physical or wireless communication link(s).

[0032] I / O unit 208 allows for the input and output of data. For example, I / O unit 208 may provide a connection for user input via a keyboard, mouse, keypad, touch screen, or other suitable input device. I / O unit 208 may also send output to a display or other suitable output device. It should be noted, however, that I / O unit 208 may be omitted if device 200 does not require local I / O, such as when device 200 represents a remotely accessible server or other device.

[0033] In some embodiments, the instructions executed by the processing device 202 include instructions to perform operations 102-108 above. Thus, for example, the instructions executed by the processing device 202 may cause the processing device 202 to obtain an initial policy for a current scenario and identify any prior scenarios that are similar to the current scenario. The instructions executed by the processing device 202 may also cause the processing device 202 to identify similar reward functions and generate a new reward function and an updated policy for the current scenario. The updated policy may be provided to the ML / AI algorithm 112 to identify a course of action selected by the ML / AI algorithm 112 for the current scenario. The updated policy may also be used in any other suitable manner.

[0034] Although Figure 2 illustrates an example of a device 200 supporting explicitly ethical machines using a similar scenario, various modifications can be made to Figure 2. For example, computing and communication devices and systems are provided in a wide variety of configurations, and Figure 2 is not intended to limit this disclosure to any particular computing or communication device or system.

[0035] FIG. 3 illustrates an exemplary use of a single-agent architecture 300 to provide an explicitly ethical machine using a similar scenario according to the present disclosure. Here, the single-agent architecture 300 uses a single instance of the architecture 100 to support the use of at least one ML / AI algorithm 112. As shown in FIG. 3, inputs regarding the current scenario are provided to a policy identification operation 102 within the single instance of the architecture 100. Here, the inputs are generated using a Markov decision process (MDP オリジナル )302a and the original reward function (R オリジナル and an initial set of one or more parameters for optimization, which may be represented by input 302b. The input may be obtained from any suitable source(s), e.g., a user, in any suitable manner.

[0036] The policy identification operation 102 provides the initial policy containing that information to a similarity identification operation 104, which accesses a database 110 to identify any prior scenarios that are similar to the current scenario. The similarity identification operation 104 generates one or more similar policies (π 類似 ) 304. As noted above, identification of one or more similar prior scenarios may occur in any suitable manner, such as via statistical correlation of information of the current scenario with information related to the prior scenarios.

[0037] The conflict identification operation 106 can identify whether the one or more similar policies 304 are optimizing any parameters that are not optimized by the original reward function 302b. The conflict identification operation 106 can identify whether the one or more similar policies 304 are optimizing any parameters that are not optimized by the original reward function 302b. In some embodiments, the conflict identification operation 106 can perform an IRL using a Markov decision process 302a and one or more similar policies 304, which identifies any parameters associated with similar prior scenarios that are or are not optimized by the original reward function 302b (or vice versa). The result is a similar reward function (R 類似 ) 306.

[0038] The reward function modification operation 108 examines the difference between the original reward function 302b and the similar reward function 306. Based on this, the reward function modification operation 108 determines one or more parameters from a similar prior scenario that can be inserted into the original reward function 302b for optimization, and / or the reward function modification operation 108 determines one or more parameters in the original reward function 302b that can be removed from the original reward function 302b. This helps to analogize the current scenario to the prior scenario(s) by modifying the reward function of the current scenario. The reward function modification operation 108 thus updates the original reward function 302b to produce a new reward function (R 新 The policy identification operation 102 can combine the Markov decision process 302a with the new reward function 308 to generate a new policy (π 新 ) 310 can be generated, which (ideally) represents the initial policy modified by incorporating one or more analogs from one or more prior scenarios.

[0039] The new policy 310 may be used in any suitable manner. For example, the new policy 310 may be provided from the policy identification operation 102 to the similarity identification operation 104 for another iteration of the process (where R オリジナル is the R from the previous iteration 新 (representing a new policy 310). The new policy 310 can be stored in the database 110 for use in discovering analogies for other input policies. The new policy 310 can be fed to the ML / AI algorithm 112, which can identify a proposed course of action for the current scenario. In some cases, the proposed course of action can be evaluated by comparison with human justification to determine whether the new policy 310 enables the selection of the right course of action for the right reason(s).

[0040] The single-agent approach shown in Figure 3 can be used in any number of applications, such as the following examples: Note that the following examples are for illustrative purposes only and are not intended to limit the scope of the disclosure to the specific examples or types of examples described below.

[0041] In one application, the ML / AI algorithm 112 may be trained to select which of a plurality of patients should receive dialysis. The architecture 300 may receive as inputs 302a-302b an indication of the number of dialysis machines available in a given area, a list of patients in need of receiving dialysis in a given area (including demographic information), and an indication that the selection should be optimized based on the age of the patients. Using statistical correlation or other techniques, the architecture 300 may determine that selecting patients for dialysis has similar characteristics to selecting patients for kidney transplants, and that a prior scenario has been defined in relation to selecting patients for kidney transplants. A policy regarding the selection of patients for kidney transplants may be selected as the similar policy 304, and the architecture 300 may analyze the similar policy 304 and determine that the similar policy 304 includes two parameters (smoking and drug use) that were not included in and optimized by the original reward function 302b. Thus, these parameters may be included in the similar reward function 306, other or additional parameters may be included, or some existing parameters from the original reward function 302b may be excluded from the similar reward function 306. The architecture 300 may then generate a new reward function 308 based on the original reward function 302b and the similar reward function 306, e.g., where the new reward function 308 includes one or both of the parameters (smoking and drug use) from the similar policy 304. The new reward function 308 may be used to generate a new policy 310, and the ML / AI algorithm 112 may apply the new policy 310 to the problem of selecting which of multiple patients should receive dialysis. The ML / AI algorithm 112 may also provide an explanation of why the selected patient was selected for dialysis, e.g., by outputting information indicating that the similar policy 304 was selected and two parameters from the similar policy 304 were included in the new policy 310.

[0042] According to another application example, the ML / AI algorithm 112 may be trained to determine where hospitals should be built within a specified region. The architecture 300 may receive as inputs 302-302b an indication of the number of hospitals to build, demographic information for the specified region, and an indication that the decision should be optimized based on population size. Using statistical correlation or other techniques, the architecture 300 may determine that identifying which hospitals should be built has similar properties to identifying which schools should be built, and that a prior scenario has been defined with respect to identifying which schools should be built. A policy regarding identifying where the schools should be built may be selected as a similar policy 304, and the architecture 300 may analyze the similar policy 304 and determine that the similar policy 304 includes one parameter (connectivity to other towns) that was not included in and optimized by the original reward function 302b. Thus, this parameter may be included in the similarity reward function 306, other or additional parameters may be included, or some existing parameters from the original reward function 302b may be excluded from the similarity reward function 306. The architecture 300 may then generate a new reward function 308 based on the original reward function 302b and the similarity reward function 306, for example, where the new reward function 308 includes the new parameters from the similarity policy 304. Using the new reward function 308, a new policy 310 may be generated, and the ML / AI algorithm 112 may apply the new policy 310 to the problem of identifying where the hospital will be built. The ML / AI algorithm 112 may also provide an explanation of why the identified hospital location was selected, for example, by outputting information indicating that the similarity policy 304 was selected and parameters from the similarity policy 304 were included in the new policy 310.

[0043] FIG. 4 illustrates an exemplary use of a multi-agent architecture 400 to provide an explicitly ethical machine using a similar scenario according to the present disclosure. The multi-agent architecture 400 here uses multiple instances of the architecture 100 to support the use of at least one ML / AI algorithm 112, although multiple instances of the architecture 100 may share the same database 110 (note, however, that this is not required). Different instances of the architecture 100 may have different biases or preferences regarding how a task should be performed. As shown in FIG. 4, inputs regarding the current scenario are provided to the policy identification operation 102 of each single instance of the architecture 100. The inputs here include a Markov decision process 402a and an initial set of one or more parameters for optimization, which may be represented by an original reward function 402b. The inputs may be obtained in any suitable manner from any suitable source(s), e.g., a user.

[0044] Each policy identification operation 102 provides this information to an associated similarity identification operation 104, which accesses database 110 to identify any prior scenarios that are similar to the current scenario. Note that the prior scenarios accessed by each similarity identification operation 104 may be relevant only to that particular instance of architecture 100, or prior scenarios may be shared across multiple instances of architecture 100. Each similarity identification operation 104 may output one or more similar policies 404 that represent one or more policies associated with one or more similar prior scenarios. As noted above, identification of one or more similar prior scenarios may occur in any suitable manner, such as via a statistical correlation between information of the current scenario and information related to the prior scenarios.

[0045] Each conflict identification operation 106 can identify whether one or more similar policies 404 optimize any parameters that are not optimized by the associated original reward function 402b. Each conflict identification operation 106 can identify whether one or more similar policies 404 optimize any parameters that are not optimized by the original reward function 402b. In some embodiments, each conflict identification operation 106 can perform an IRL using a Markov decision process 402a and one or more similar policies 404, which identifies any parameters associated with similar prior scenarios that are optimized or not optimized by the original reward function 402b (or vice versa). The results can be output from each conflict identification operation 106 as a similar reward function 406. Note that each conflict identification operation 106 can output its own similar reward function 406, or multiple conflict identification operations 106 can share a common similar reward function 406.

[0046] Each reward function modification operation 108 examines the difference between the original reward function 402b and the similar reward function 406. Based on this, each reward function modification operation 108 determines one or more parameters from a similar prior scenario that can be inserted into the original reward function 402b for optimization, and / or each reward function modification operation 108 determines one or more parameters in the original reward function 402b that can be removed from the original reward function 402b. This helps to analogize the current scenario to the prior scenario(s) by modifying each reward function of each current scenario. Each reward function modification operation 108 can thus update the original reward function 402b to generate a new reward function 408. Each policy identification operation 102 combines the Markov decision process 402a with the new reward function 408 to determine a new policy (π 新 ) 410 can be generated, which (ideally) represents an initial policy that has been modified by incorporating one or more likes from one or more prior scenarios.

[0047] The new policies 410 from the multiple instances of the architecture 100 can be fed to an evaluation operation 412, which evaluates the new policies 410 to determine whether the multiple instances of the architecture 100 reach a consensus on how a course of action is selected. Consensus herein can be defined as all instances 410 of the architecture 100 that identify the same course of action, a majority or other specified number / percentage of instances of the architecture 100 that produce policies 410 that identify the same course of action, or any other suitable criteria. Note that multiple instances of the architecture 100 may be used to select the same course of action, possibly for different reasons. If consensus is not achieved, the evaluation operation 412 can adjust one or more of the original reward functions 402b used by one or more instances of the architecture 100, and the process can be repeated. If consensus is achieved, the new policies 410 can be used as the final set of policies 414 for the ML / AI algorithm(s) 112.

[0048] The new policy 414 may be used in any suitable manner. For example, the new policy 414 may be provided from the policy identification operation 102 to the similarity identification operation 104 for another iteration of the process. The new policy 414 may be stored in the database 110 for use in finding analogies for other input policies. The new policy 414 may be provided to the ML / AI algorithm(s) 112, which may identify a proposed course of action for the current scenario. In some cases, the proposed course of action may be evaluated by comparison with human justification to determine whether the new policy 414 enables the selection of the correct course of action for the correct reason(s).

[0049] The multi-agent approach shown in Figure 4 can be used in any number of applications, such as the following examples: Note that the following examples are for illustrative purposes only and are not intended to limit the scope of the disclosure to the specific examples or types of examples described below.

[0050] In one application, the at least one ML / AI algorithm 112 can be trained to operate a missile defense system and select incoming missiles to be struck and destroyed by the missile defense system. Different locations can be targeted by the incoming missiles, each location having a unique instance of the architecture 100. The architecture 400 can receive as inputs 402a-402b an indication of the number of incoming missiles and an indication that the selection of incoming missiles to be struck should be optimized for the protection of a civilian population. Using statistical correlation or other techniques, each instance of the architecture 100 can determine that another prior scenario was defined relative to the current scenario. A policy associated with the prior scenario can be selected as a similar policy 404, and each instance of the architecture 100 can analyze the similar policy 404 and determine that the similar policy 404 includes one or more parameters not included in its original reward function 402b. These parameters may be included in each similar reward function 406, other or additional parameters may be included, or some existing parameters from the original reward function 402b may be excluded from the associated similar reward function 406. Each instance of the architecture 100 may then generate a new reward function 408 based on its original reward function 402b, and the new reward function 408 may be used to generate new policies 410, which may be compared to seek consensus. If consensus is achieved, the ML / AI algorithm(s) 112 may apply the new policies 410 as a final set of policies 414 to the problem of selecting which incoming missiles to crash. If consensus is not achieved, an evaluation operation 412 may adjust one or more of the original reward functions 402b and repeat this process until consensus is achieved (or some other specified criteria or criteria are met).The ML / AI algorithm(s) 112 may also provide an explanation of why a selected incoming missile was selected for impact, for example by outputting appropriate information explaining the process.

[0051] 3 and 4 illustrate examples of the use of single-agent and multi-agent architectures 300, 400 to provide explicitly ethical machines using similar scenarios, various modifications may be made to FIGS. 3 and 4. For example, there may be multiple databases 110 used in either architecture 300 or 400, and / or one or more databases 110 may be local or remote to the device(s) implementing the instance(s) of architecture 100. Also, any suitable number of instances of architecture 100 may be used in a multi-agent system, and results from instances of architecture 100 may be combined or otherwise used in any other suitable manner.

[0052] 5 illustrates an example architecture 500 for using inverse reinforcement learning in an architecture for providing an explicitly ethical machine using a similar scenario according to the present disclosure. The technique 500 illustrated in FIG. 5 may be used to refine a reward function based on the similar scenario, for example, according to operation 106. However, it should be noted that operation 106 may be implemented in any other suitable manner.

[0053] 5, the technique 500 includes a sequence of tasks 502a-502n that occur over a period of time, where task 502a occurs first, followed by other tasks 502b-502n. Each task 502a-502n is associated with a version of a reward function 504a-504n, respectively, which may represent a common reward function that varies over time.

[0054] Each version of the reward function 504a-504n here includes or is otherwise associated with a task-independent portion 506a-506n and a task-dependent portion 508a-508n, respectively. The task-independent portions 506a-506n of the reward functions 504a-504n can be used to help define a reward system that enforces ethical boundaries regardless of the task being performed. The task-dependent portions 508a-508n of the reward functions 504a-504n can be used to help define a reward system that drives the contextual behavior of the reward system. Each reward function 504a-504n here can be used to generate a result 510a-510n, which represents the application of the reward function 504a-504n to data associated with the respective task 502a-502n. The task-independent portions 506a-506n and the task-dependent portions 508a-508n of each reward function 504a-504n can be used to generate associated results 510a-510n.

[0055] Here, line 512 defines the improvement of ethical behavior that may occur in the task-independent portions 506a-506n over time. This may, for example, allow for the improvement of the ethical components of gradual inheritance and rewards to achieve a higher level of ethical development (e.g., moving up the stages of the Kohlberg model). Here, line 514 defines the analogy and transfer learning that may occur in the task-dependent portions 508a-508n over time. This may, for example, be used similarly to the preceding scenario to facilitate learning over time, as may occur in the techniques described above using the database 110. Analogy with the preceding scenario may help to enhance learning and reduce the training requirements of the ML / AI algorithm(s) 112. Here, the Bayesian formulation may provide a logically sound framework for learning and acting under uncertainty, allowing trade-off analysis for conflicting ethical norms and rewards.

[0056] Figure 5 illustrates an example of a technique 500 for using inverse reinforcement learning in an architecture for providing an explicitly ethical machine using similar scenarios, although various modifications may be made to Figure 5. For example, the reward function may evolve in various ways over time based on changes to its task-independent portions and / or its task-dependent portions.

[0057] 6 illustrates an example method 600 for providing an explicitly ethical machine using a similar scenario according to the present disclosure. For ease of explanation, the method 600 is described as being performed by the architecture 100, which may be implemented using at least one device 200 of FIG. 2. However, the method 600 may be performed by any suitable device(s) and in any suitable system(s).

[0058] As shown in Figure 6, input information defining a current scenario and an initial reward function is obtained in step 602 and used to generate an initial policy for the current scenario in step 604. This may include, for example, the processing device 202 performing the policy identification operation 102 to receive information from a user or other source(s) that specifies a Markov decision process and an original reward function. This may also include the processing device 202 performing the policy identification operation 102 to use the Markov decision process and the original reward function as an initial policy for the current scenario or to generate an initial policy for the current scenario. The original reward function may identify one or more parameters to be optimized when selecting a course of action for the current scenario.

[0059] At step 606, the database is searched for any prior scenarios that are similar to the current scenario, and at step 608, a similar policy associated with each similar prior scenario is identified. This may include, for example, the processing device 202 performing the similarity identification operation 104 to search the database 110 for any prior scenarios that are similar to the current scenario. In some cases, the similarity may be determined statistically, for example, based on the similarity of the Markov decision processes and reward functions of the current and prior scenarios. This may also include the processing device 202 performing the similarity identification operation 104 to extract policy information associated with each of the identified similar prior scenarios from the database 110, and using the policy information as the similarity policy or policies.

[0060] Inverse reinforcement learning is applied to one or more policies associated with one or more similar prior scenarios at step 610, and a similar reward function is generated based on the IRL results at step 612. This may include, for example, the processing device 202 performing conflict identification operation 106, where the processing device 202 may use a Markov decision process for the current scenario and information about the similar policies to identify a similar reward function. The similar reward function may include one or more parameters not included in the original reward function and / or the similar reward function may omit one or more parameters included in the original reward function.

[0061] The initial reward function and the similar reward function are compared in step 614, and a new reward function for the current scenario is generated in step 616. This may include, for example, the processing device 202 performing reward function modification operations 108 to identify differences between the original reward function and the similar reward function to identify (i) one or more parameters of the similar reward function that are not included in the original reward function, and / or (ii) one or more parameters of the original reward function that are not included in the similar reward function. This may also include the processing device 202 performing reward function modification operations 108 to generate a new reward function, which may represent, as modified, the original reward function that (i) includes at least one parameter from the similar reward function, and / or (ii) excludes at least one parameter from the original reward function.

[0062] In step 618, a new policy based on the new reward function is obtained. This may include, for example, the processing device 202 performing the policy identification operation 102 to use the Markov decision process of the current scenario and the new reward function of the current scenario as a new policy for the current scenario. At this point, optionally, a determination may be made in step 620 as to whether to repeat the process. In some cases, the determination in step 620 is used when there are multiple agents (multiple instances of the architecture 100) each performing steps 602-618 and the resulting policies from the multiple agents do not match. In other cases, the determination in step 620 is used to determine whether the new policy should be subjected to another iteration of the process. This allows, for example, more parameters to be added to the policy and taken into account during inference. If an iteration is desired for whatever reason, the process returns to step 602 (or some other step) to repeat one or more operations.

[0063] In step 622, the new policy may be applied using the ML / AI algorithm or stored, output, or used in some manner. This may include, for example, the processing device 202 providing the new policy to the ML / AI algorithm 112, which may be executed by the same processing device 202 or a different processing device. This may also include the ML / AI algorithm 112 using the new policy to identify a selected course of action that occurs as a result of the current scenario. However, it should be noted that the new policy may be stored, output, or used in any other suitable manner, including the various aspects described above.

[0064] Although Figure 6 illustrates one example of a method 600 for providing an explicitly ethical machine using a similar scenario, various modifications may be made to Figure 6. For example, although shown as a series of steps, various steps in Figure 6 may overlap, occur in parallel, occur in a different order, or occur any number of times.

[0065] In some embodiments, the various functions described in this patent document are realized or supported by a computer program formed from computer readable program code and embodied in a computer readable medium. The phrase "computer readable program code" includes any type of computer code, including source code, object code, and executable code. The phrase "computer readable medium" includes any type of medium accessible by a computer, such as read only memory (ROM), random access memory (RAM), hard disk drive (HDD), compact disk (CD), digital video disk (DVD), or any other type of memory. "Non-transitory" computer readable medium excludes wired, wireless, optical, or other communication links that transmit transient electrical or other signals. Non-transitory computer readable medium includes media on which data is permanently stored and media on which data can be stored and later overwritten, such as rewritable optical disks or erasable storage devices.

[0066] It is deemed advantageous to provide definitions of certain words and phrases used throughout this patent document. The terms "application" and "program" refer to one or more computer programs, software components, sets of instructions, procedures, functions, objects, classes, instances, associated data, or portions thereof, adapted for implementation in suitable computer code (including source code, object code, or executable code). The term "communicate" and its derivatives encompass both direct and indirect communication. The terms "include" and "comprise" and their derivatives mean inclusion without limitation. The term "or" also includes the meaning of "and / or". The term "associated with" and its derivatives may mean including, contained in, interconnected, containing, stored in, connected, coupled, communicable, cooperating, interleaving, juxtaposing, adjacent, joined, having, having an attribute, relating thereto, and the like. The phrase "at least one of," when used in conjunction with a list of items, means that different combinations of one or more of the listed items may be used, and only one item in the list is required. For example, "at least one of A, B, and C" includes any combination of the following: A, B, C, A and B, A and C, B and C, and A and B and C.

[0067] The description in this disclosure should not be construed as implying that any particular element, step, or function is an essential or critical element that must be included in the scope of the claims. The scope of patented subject matter is defined solely by the claims as allowed. Moreover, no claim invokes 35 U.S.C. 112(f) with respect to any of the appended claims or claim elements, unless the precise phrase "means for" or "step for" followed by a specific phrase identifying the function is expressly used in the particular claim. The use of terms such as "mechanism," "module," "device," "unit," "component," "element," "apparatus," "machine," "system," "processor," or "controller" in the claims, without limitation, is understood and intended to refer to structures known to those skilled in the relevant art, as further modified or enhanced by the features of the claims themselves, and no invocation of 35 U.S.C. 112(f) is intended.

[0068] While this disclosure has described certain embodiments and generally associated methods, modifications and permutations of these embodiments and methods will be apparent to those skilled in the art. Thus, the above description of exemplary embodiments does not define or constrain this disclosure. Other modifications, substitutions, and alterations are possible without departing from the spirit and scope of this disclosure as defined by the following claims.

Claims

1. 1. A method comprising: obtaining, using at least one processor, information related to a current scenario to be evaluated by a machine learning / artificial intelligence (ML / AI) algorithm, the information including an initial reward function related to the current scenario; identifying, using the at least one processor, one or more policies associated with one or more prior scenarios that are similar to the current scenario; determining, using the at least one processor, one or more differences between parameters optimized by the initial reward function and one or more reward functions associated with the one or more prior scenarios; modifying the initial reward function based on at least one of the one or more determined differences, using the at least one processor, to generate a new reward function; and generating, using the at least one processor, a new policy for the current scenario based on the new reward function; A method comprising:

2. 2. The method of claim 1, further comprising: applying the new policy to the current scenario using the ML / AI algorithm to determine a selected course of action for the current scenario.

3. 2. The method of claim 1 , further comprising: repeating at least some of the obtaining, identifying, determining, modifying, and generating using the new reward function as the initial reward function.

4. the information relating to the current scenario includes a first Markov decision process; The method of claim 1 , wherein the database stores one or more second Markov decision processes associated with the one or more prior scenarios.

5. 2. The method of claim 1 , wherein determining the one or more differences between the optimized parameters comprises using inverse reinforcement learning to identify inferences used in the one or more prior scenarios that apply to the current scenario.

6. The obtaining, identifying, determining, modifying, and generating are performed using a plurality of agents; The method further comprising: comparing the new policies generated by the multiple agents for a match; and modifying the initial reward function used by at least one of the plurality of agents in response to the new policy generated by the plurality of agents not conforming, and repeating at least some of the obtaining, identifying, determining, modifying, and generating. The method of claim 1 further comprising:

7. Each of the initial reward function and the new reward function is a task independent portion configured to enforce one or more ethical boundaries regardless of the task; a task-dependent portion configured to drive a contextual behavior of the associated reward function; and The method of claim 1 further comprising:

8. An apparatus comprising: at least one memory configured to store information related to a current scenario to be evaluated by a machine learning / artificial intelligence (ML / AI) algorithm, the information including an initial reward function associated with the current scenario; and At least one processor, identifying one or more policies associated with one or more prior scenarios that are similar to the current scenario; determining one or more differences between parameters optimized by the initial reward function and one or more reward functions associated with the one or more prior scenarios; modifying the initial reward function based on at least one of the one or more determined differences to generate a new reward function; and generating a new policy for the current scenario based on the new reward function; The processor is configured to:

13. An apparatus comprising:

9. The at least one processor further comprises:

10. The apparatus of claim 8, configured to apply the new policy to the current scenario using the ML / AI algorithm to determine a selected course of action for the current scenario.

10. The at least one processor 10. The apparatus of claim 8, further configured to repeat at least some of the identifying, determining, refining, and generating using the new reward function as the initial reward function.

11. the information relating to the current scenario includes a first Markov decision process; 10. The apparatus of claim 8, wherein the at least one processor is configured to access a database configured to store one or more second Markov decision processes associated with the one or more prior scenarios.

12. 10. The apparatus of claim 8, wherein to determine the one or more differences between the optimized parameters, the at least one processor is configured to use inverse reinforcement learning to identify inferences used in the one or more prior scenarios to apply to the current scenario.

13. The at least one processor performing said identifying, said determining, said modifying, and said generating using a plurality of agents; comparing the new policies generated by the multiple agents for a match; and modifying the initial reward function used by at least one of the plurality of agents in response to the new policy generated by the plurality of agents not conforming, and repeating at least some of the storing, identifying, determining, modifying, and generating. The apparatus of claim 8 , further configured to:

14. Each of the initial reward function and the new reward function is a task independent portion configured to enforce one or more ethical boundaries regardless of the task; a task-dependent portion configured to drive a contextual behavior of the associated reward function; and The apparatus of claim 8 , comprising:

15. 1. A non-transitory computer-readable storage medium that, when executed, causes at least one processor to: obtaining information related to a current scenario to be evaluated by a machine learning / artificial intelligence (ML / AI) algorithm, the information including an initial reward function related to the current scenario; identifying one or more policies associated with one or more prior scenarios that are similar to the current scenario; determining one or more differences between parameters optimized by the initial reward function and one or more reward functions associated with the one or more prior scenarios; modifying the initial reward function based on at least one of the one or more determined differences to generate a new reward function; and generating a new policy for the current scenario based on the new reward function; A non-transitory computer-readable storage medium comprising instructions to cause

16. When executed, the at least one processor is caused to:

20. The non-transitory computer-readable storage medium of claim 15, further comprising instructions for applying the new policy to the current scenario using the ML / AI algorithm to determine a selected course of action for the current scenario.

17. When executed, the at least one processor is caused to:

16. The non-transitory computer-readable storage medium of claim 15, further comprising instructions to repeat at least some of the obtaining, identifying, determining, modifying, and generating using the new reward function as the initial reward function.

18. the information relating to the current scenario includes a first Markov decision process; 16. The non-transitory computer-readable storage medium of claim 15, wherein the instructions, when executed, cause the at least one processor to access a database configured to store one or more second Markov decision processes associated with the one or more prior scenarios.

19. The instructions, which when executed, cause the at least one processor to determine the one or more differences between the optimized parameters, 16. The non-transitory computer-readable storage medium of claim 15, comprising instructions that, when executed, cause the at least one processor to use inverse reinforcement learning to identify inferences used in the one or more prior scenarios that apply to the current scenario.

20. When executed, the at least one processor is caused to: performing said obtaining, identifying, determining, modifying, and generating using a plurality of agents; comparing the new policies generated by the plurality of agents for a match; modifying the initial reward function used by at least one of the plurality of agents in response to the new policy generated by the plurality of agents not conforming, and repeating at least some of the obtaining, identifying, determining, modifying, and generating.

20. The non-transitory computer-readable storage medium of claim 15, further comprising instructions to: generating a first Markov decision process (MDP) representation of the current scenario as a set of possible states, possible actions, and an initial reward function that defines rewards for transitions between the possible states via the possible actions; the one or more prior scenarios are associated with one or more second MDP representations including one or more second sets of possible states and possible actions that satisfy a statistical correlation criterion to an MDP representation of the current scenario; 2. The method of claim 1 , wherein the new reward function adds at least one new parameter from one or more reward functions associated with the one or more prior scenarios, the new parameter not included in the initial reward function.

22. The at least one processor is further configured to generate a first Markov decision process (MDP) representation of the current scenario as a set of possible states, possible actions, and an initial reward function that defines rewards for transitions between the possible states via the possible actions; the one or more prior scenarios are associated with one or more second MDP representations including one or more second sets of possible states and possible actions that satisfy a statistical correlation criterion to an MDP representation of the current scenario; 10. The apparatus of claim 8, wherein the new reward function adds at least one new parameter from one or more reward functions associated with the one or more prior scenarios, the new parameter not included in the initial reward function.

23. The method of claim 20, wherein when executed, the method generates, on the at least one processor, a first Markov decision process (MDP) representation of the current scenario as a set of possible states, possible actions, and an initial reward function that defines rewards for transitions between the possible states via the possible actions; the one or more prior scenarios are associated with one or more second MDP representations including one or more second sets of possible states and possible actions that satisfy a statistical correlation criterion to an MDP representation of the current scenario; 16. The non-transitory computer-readable storage medium of claim 15, wherein the new reward function adds at least one new parameter from one or more reward functions associated with the one or more prior scenarios, the new parameter not included in the initial reward function.

Images