Computer implementation methods, computer programs, and systems (faster adapted Q-iterations using zero-suppression decision diagrams)

Abstract

To provide a faster Fitted Q-iteration using a zero-suppressed decision diagram in off line reinforcement learning.SOLUTION: A method includes a step 100 of obtaining a set of tuples D and a discount factor γ, each of the set of tuples including a state s, an action a, a reward r, and a resulting state s', a step 102 of constructing a zero-suppressed decision diagram (ZDD) of feature vectors for each of the resulting states s' of the set of tuples, a step 104 of updating parameters w, θ of a state-action value function Q (s, a; w, θ); and a step 106 of repeating the updating step a predetermined times by incrementing t.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates, in general, to machine learning, and more specifically, to a method and system for faster fitted Q-iteration using a zero-suppressed decision diagram. [Background technology]

[0002] Q-learning is a popular reinforcement learning algorithm with several applications in automation and robotics. Reinforcement learning algorithms compute control policies through agents, which can learn directly from the system (online control) or through interaction with a simulator of the system (offline or batch control). A desirable control policy is one that selects the action that maximizes the reward accumulated over time by the agent, starting from any initial state. Q-learning uses an action-value model, which creates a function that deals with different states. Q-learning was conceived to determine the optimal policy in a stepwise manner.

[0003] Through Q-learning in states where the entire state representation has been shown to converge, a suitable approximation method is required to find the optimal policy when the state of the environment is partially observable (e.g., inaccuracy or delay of a sensor device). For finite and sufficiently small state and action spaces, the Q-function can be expressed in tabular form, and its approximation and derived control policy (in batch and online modes) are simple. However, this method cannot be successfully used when dealing with continuous or very large discrete state-action or action spaces, or both. One important step in most regression problems is the selection of variables that describe the response, while removing others from the model [1][7]. Such procedures are usually called feature selection techniques, and they facilitate the learning of a good and simple regressor with the reduced variable space. Another important problem that needs to be addressed when training a regressor is so-called hyperparameter optimization. These hyperparameters are specific to certain types of machine learning (ML) models and may, depending on the case, be the number of neurons, kernel functions and their parameters, regularization constants, etc. [2] These must be tuned to obtain the optimal model given a performance or cost function. Both of these techniques, namely feature selection and hyperparameter optimization, have a significant impact on model quality, interpretability, training speed, and model evaluation speed, where the optimal solution is usually obtained as a compromise between these model characteristics. [Overview of the project] [Problems that the invention aims to solve]

[0004] This invention provides a method and device for faster fitted Q-iterations using zero-suppression decision diagrams in offline reinforcement learning. [Means for solving the problem]

[0005] According to one embodiment, a computer implementation method for estimating a state-action value function for a fitted Q-iteration is provided. The computer implementation method includes the steps of obtaining a set of tuples D and a discount rate γ, where each of the set of tuples includes a state s, an action a, a reward r, and a resulting state s', and for each of the resulting states s' of the set of tuples, a feature vector

number

number

number

[0006] According to another embodiment, a computer program product is provided for estimating a state-action value function for fitted Q iterations. The computer program product comprises a computer-readable storage medium in which program instructions are embodied, the program instructions comprising a procedure for obtaining a set of tuples D and a discount rate γ, each of the set of tuples comprising a state s, an action a, a reward r, and a resulting state s', and a feature vector for each of the resulting states s' of the set of tuples.

number

number

Number

[0007] According to yet another embodiment, a system for estimating a state-action value function for adaptive Q-iteration is provided. The system includes a memory and one or more processors in communication with the memory. The one or more processors are to obtain a set D of tuples and a discount rate γ, each of the set of tuples including a state s, an action a, a reward r, and a resulting state s', and for each resulting state s' of the set of tuples, to construct a zero-suppressed decision diagram (ZDD) of a feature vector

Number

Number

Number

[0008] According to another embodiment, a computer implementation method is provided for estimating a state-action value function for fitted Q iterations. The computer implementation method includes a step of obtaining a set of tuples D and a discount rate γ, each of which the set of tuples includes a state s, an action a, a reward r, and a resulting state s', and for each of the resulting states s' of the set of tuples, a feature vector

number

number

number

[0009] In yet another embodiment, a computer implementation method is provided for estimating a state-action value function for a fitted Q iteration. The computer implementation method is a step of obtaining a set of tuples D and a discount rate γ, each of which the set of tuples includes a state s, an action a, a reward r, and a resulting state s', and for each of the resulting states s' of the set of tuples, a feature vector

number

number

number

[0010] It should be noted that exemplary embodiments are described with reference to different subject matter. In particular, some embodiments are described with reference to method-type claims, while others are described with reference to apparatus-type claims. However, those skilled in the art will infer from the above and below descriptions that, unless otherwise notified, any combination of features belonging to one type of subject matter, as well as any combination of features relating to different subject matter, in particular, any combination of features of method-type claims and features of apparatus-type claims, are also described herein.

[0011] These and other features and advantages will become apparent from the following detailed description of exemplary embodiments of the present invention when read in conjunction with the accompanying drawings. [Brief explanation of the drawing]

[0012] The present invention provides further details in the following description of preferred embodiments with reference to the following drawings.

[0013] [Figure 1] This is an exemplary decomposition of machine learning in artificial intelligence (AI) and a block / flow diagram of where the fitted Q iterations reside, relating to one embodiment of the present invention.

[0014] [Figure 2] This figure shows a method for implementing a fitted Q-iteration using a zero-suppression decision diagram (ZDD) according to one embodiment of the present invention.

[0015] [Figure 3]This figure shows a practical application of a machine learning workflow for computational material discovery according to one embodiment of the present invention.

[0016] [Figure 4] This figure shows a method for utilizing adapted Q-iterations using ZDD according to one embodiment of the present invention.

[0017] [Figure 5] This is an algorithm that utilizes adapted Q-iterations using ZDD, relating to one embodiment of the present invention.

[0018] [Figure 6] This is a block / flow diagram of an exemplary practical application for chemical discovery of a method for predicting chemical properties or generating a new molecule, according to one embodiment of the present invention.

[0019] [Figure 7] This is a block / flow diagram of an exemplary processing system that utilizes adapted Q iterations using ZDD according to one embodiment of the present invention.

[0020] [Figure 8] This figure shows a practical application of an artificial intelligence (AI) accelerator chip that utilizes adapted Q-iteration using ZDD, according to one embodiment of the present invention.

[0021] [Figure 9] This is a block / flow diagram of an exemplary cloud computing environment relating to one embodiment of the present invention.

[0022] [Figure 10] This is a schematic diagram of an exemplary abstraction model layer according to one embodiment of the present invention.

[0023] Throughout the drawing, the same or similar reference numerals represent the same or similar elements. [Modes for carrying out the invention]

[0024] Embodiments of the present invention provide a method and device for faster adaptive Q-iteration using zero-suppressed decision diagrams in offline reinforcement learning. Reinforcement learning aims to determine an optimal control policy from observations collected from or interacting with a system. In batch mode, this can be achieved by approximating a so-called Q-function based on a set of four tuples (x t , u t , r t , x t+1 ), where x t represents the system state at time t, u t represents the control action taken, r t represents the instantaneous reward obtained, and X t+1 represents the subsequent state of the system), and determining a control policy from this Q-function. Q-function approximation can be obtained from the limitations of a sequence of supervised learning problems (in batch mode). For a finite and sufficiently small state space and action space, the Q-function can be represented in tabular form, so its approximation (in batch and online mode) and the derived control policy are simple. However, this approach cannot be successfully used when dealing with continuous or very large discrete state spaces or action spaces, or both.

[0025] To address these problems, the Fitted Q-Iteration (FQI) algorithm was introduced. FQI is a batch-mode reinforcement learning algorithm that provides an approximation of the Q-function corresponding to an infinite-horizon optimal control problem with discounted rewards by iteratively expanding the optimization horizon. In the first iteration, the FQI algorithm generates an approximation of the expected reward using a training set with inputs as state-action pairs and outputs as observed rewards. In subsequent iterations, only the output values ​​are updated using the Q-function values ​​generated in the previous stage, as well as information about the rewards reached and subsequent states in each tuple. Since all updates are performed offline, the approximation of the Q-function can be viewed as a separate supervised learning problem. This raises the question of whether there are function approximators that are particularly suited to offline updates.

[0026] The concept of FQI is derived from the pioneering work of Ormoneit and Sen, which combined the concept of fitted value iterations with kernel-based reinforcement learning, reformulating the Q-function decision problem as a sequence of kernel-based regression problems. FQI was introduced by Ernst to fit any (parametric or nonparametric) approximate architecture to the Q-function (using a set of four tuples).

[0027] However, FQI alone may not be sufficient for certain applications, such as the discovery of high-performance functional materials or computational material discovery.

[0028] Finding new materials with superior performance is an eternal challenge in materials science. Currently, experimental and computational screening for the discovery of new materials involves elemental substitution and structural transformation. However, the compositional search space, the structural search space, or both tend to be severely constrained. Furthermore, both of these screening methods require enormous amounts of computation or experimentation, and often lead to the misguided pursuit of "exhaustive search," which consumes significant time and resources. Considering this fact and the advantages of machine learning, a method combining machine learning with computational simulation is proposed for the evaluation and screening of new materials, providing suggestions for novel and better materials.

[0029] Exemplary embodiments of the present invention mitigate such problems in discovering new materials by implementing faster fitted Q iterations using zero-suppression decision diagrams (ZDDs). Such configurations may be beneficial, for example, in computational materials discovery for generating new molecular structures that satisfy target properties.

[0030] While the present invention is described in relation to a given exemplary architecture, it should be understood that other architectures, structures, substrate materials, process features, and stages / blocks may vary within the scope of the invention. Note that for clarity, certain features may not be shown in all drawings. This is not intended to be construed as limiting any particular embodiment, illustration, or claim.

[0031] Figure 1 is an exemplary decomposition of machine learning in artificial intelligence (AI) and a block / flow diagram showing where the fitted Q iterations reside, according to one embodiment of the present invention.

[0032] Artificial intelligence 10 includes machine learning 12. Machine learning 12 can be divided into supervised learning 20, unsupervised learning 30, semi-supervised learning 40, deep learning 50, and reinforcement learning 60.

[0033] Supervised learning 20 can include, for example, classification 22 and regression 24.

[0034] Unsupervised learning 30 may include, for example, clustering 32 and dimensionality reduction 34.

[0035] Semi-supervised learning 40 can include, for example, the application of a Boltzmann machine 42.

[0036] Deep learning 50 may include, for example, a convolutional neural network (CNN) 52 and a recurrent neural network (RNN) 54.

[0037] Reinforcement learning 60 may include a temporal difference 64 and a deep adversarial network 62. Reinforcement learning 60 may further include Q-learning 66, which includes an adapted Q-iteration 68 combined with a zero-suppression decision diagram 70 of an exemplary embodiment of the present invention. The adapted Q-iteration 68 combined with the zero-suppression decision diagram 70 may implement Q(s,a;w,θ)=w·φ(s,a)+f(s;θ), specified as equation 69, where φ(s,a)∈{0,1} D This is a sparse bit vector,

number

[0038] In an alternative embodiment, the adapted Q iteration 68 can be combined with a binary decision diagram (BDD) 72. A binary decision diagram (BDD) provides a compact way to uniquely represent a given Boolean function. A BDD is a rooted, directed, acyclic graph consisting of decision nodes and terminal nodes.

[0039] Figure 2 shows a method for implementing a fitted Q-iteration using a zero-suppression decision diagram (ZDD) according to one embodiment of the present invention.

[0040] In the offline reinforcement learning diagram 80, the input data 82 is fed into the adapted Q-iteration component 84 using a zero-suppression decision diagram (ZDD) to obtain the optimal policy 86. The input data is,

number

number

number

number

[0041] Traditionally, reinforcement learning (RL) has been considered an online learning paradigm, where the interaction between the RL agent and its environment is a fundamental concern regarding how the agent learns. In offline RL (known as batch RL), the agent learns from a fixed-size dataset collected by some arbitrary and possibly unknown process. Eliminating the need to interact with the environment is noteworthy, as data collection can often be costly, risky, or otherwise difficult, especially in real-world applications. Thus, offline RL allows the use of previously logged data or the utilization of experts such as human operators without any of the risks associated with untrained RL agents. However, the absence of environment interaction, which is the main benefit of offline RL, also makes it a challenging task.

[0042] In this case, the difficulty regarding the fitting Q iteration is that this is

number

number

[0043]

number

[0044] Here, γ ∈ (0,1), which is the discount rate.

[0045] An exemplary embodiment solves such problems by incorporating a zero-suppression decision diagram (ZDD) into the adapted Q iteration.

[0046] In particular, the action-value function is parameterized by the following:

[0047] Q(s,a;w,θ)=w·φ(s,a)+f(s;θ)

[0048] Here, φ(s,a)∈{0,1} D This is a sparse bit vector,

number

[0049] A zero-suppression decision diagram (ZDD) is a compact data structure for sparse bit vectors.

number

number

number

number

[0050] Regarding ZDDs, a ZDD is a specific type of binary decision diagram (BDD) with a fixed order of variables. This data structure provides a standard, compact representation of a set, particularly well-suited to certain combinatorial problems. In a ZDD, nodes are removed if their positive edges point to the terminal node 0. This provides an improved compression of sparse sets to an alternative, powerful, conventional form.

[0051] When BDDs are applied to combinatorial problems, it can be observed that the majority of positive edges of decision nodes simply point towards the zero-terminus. This can be particularly true in the case of matching arrays. In these cases, zero-suppressed binary decision diagrams (ZSDDs, ZBDDs, or ZDDs) may perform better than standard BDDs. A ZDD is a type of BDD designed to encode a set of combinations or a family of sets of primitive elements. A ZDD is a rooted, directed, acyclic graph (DAG) containing terminal and non-terminal nodes. Each non-terminal node has two outward-pointing edges to its child nodes, labeled by a variable and referred to as negative edges (or LO edges) and positive edges (or HI edges).

[0052] Like a standard BDD, a ZDD has two terminal or leaf nodes labeled FALSE and TRUE (or 0-terminated and 1-terminated) that do not have outward-pointing edges. Furthermore, the population of all variables (or primitive elements) is ordered, and the order of variables appearing on a node in any path through the ZDD is consistent with the global order. In addition, each path through a ZDD terminating at a TRUE-terminated node defines a set of variables in a family of sets.

[0053] Figure 3 shows a practical application of a machine learning workflow for computational material discovery according to one embodiment of the present invention.

[0054] In computational materials discovery, experiments have traditionally played a crucial role in discovering and characterizing new materials. Experimental research imposes high resource and equipment requirements, meaning it must be conducted over long periods for a very limited number of materials. Due to these limitations, significant discoveries have largely occurred through human intuition. The first computational revolution in materials science was accelerated by the emergence of computational methods, particularly density functional theory (DFT), Monte Carlo simulation, and molecular dynamics, which enabled researchers to explore phase and compositional spaces far more efficiently. Indeed, a combination of both experiments and computer simulations has made it possible to drastically reduce the time and cost of materials design. The continuous increase in computing power and the development of more efficient codes have similarly enabled high-throughput computational studies of large groups of materials to screen for ideal experimental candidates. These large-scale simulations and calculations, combined with high-throughput experimental studies, generate enormous amounts of data, enabling the use of machine learning methods in materials science.

[0055] The availability of large datasets, combined with algorithmic improvements and the exponential growth of computing power, has led to an unparalleled surge in interest in the topic of machine learning. As these algorithms begin to find their place, they herald the arrival of a second computational revolution. The number of possible materials is... (googol(10) 100 Since the estimated value is quite high, this revolution is undoubtedly essential.

[0056] Referring back to Figure 3, in the offline reinforcement learning figure 90, the molecular structure and chemical property values ​​92 are used as a dataset for the offline RL as input data 94. The extracted chemical features are processed by a chemical optimization procedure including fitted Q iterations 96 using a zero-suppression decision diagram (ZDD) to obtain an optimal policy 98 that results in the generation of a new molecular structure 99.

[0057] Therefore, machine learning provides new means to screen for novel materials with good performance, develop quantitative structure-activity relationships (QSARs) and other models, predict material properties, discover new materials, and conduct other materials-related research. One exemplary method of machine learning involves fitted Q iterations with zero-suppression decision diagrams (ZDDs) for computational material discovery (CMD), as described herein.

[0058] Figure 4 shows a method using adapted Q-iterations with ZDD according to one embodiment of the present invention.

[0059] In block 100, a set of tuples D and a discount rate γ are obtained, each tuple containing a state s∈S, an action a∈A, a reward r, and the resulting state s'∈S.

[0060] In block 102, for each state s' obtained as a result of the tuple in set D, feature vector

number

number

[0061] In block 104,

number

number

[0062] In block 106, the update step is repeated a predetermined number of times by incrementing t.

[0063] Therefore, the assumptions regarding the action-value function are as follows:

[0064] Q(s,a;w,θ)=w·φ(s,a)+f(s;θ)

[0065] Here, φ is the feature vector of a state S and an action a applicable to state s. The state is a molecule, and the action is a modification applied to the molecule. An exemplary embodiment uses a Morgan fingerprint (bit vector) for φ. A molecular fingerprint is a way of encoding the structure of a molecule. The most common type of fingerprint is a set of binary numbers (bits) that represent the presence or absence of a particular substructure in the molecule. By comparing fingerprints, it becomes possible to determine the similarity between two molecules to find matches for querying substructures, etc. The Morgan fingerprint is essentially a reimplementation of the Extended Connectivity Fingerprint (ECFP). The Extended-Connectivity Fingerprint (ECFP) is a circular topological fingerprint designed for molecular characterization, similarity search, and structure-activity modeling.

[0066] The main characteristics of ECFPs are that they represent molecular structure via circular atom neighborhoods, they can be computed very quickly, their features represent the presence of specific substructures, they are not predefined and can represent a vast number of different molecular features (including stereochemical information), they are designed to represent both the presence and absence of functionality (as both are essential for analyzing molecular activity), and their generation methods can be flexibly customized to generate various types of circular fingerprints for diverse applications.

[0067] Returning to the action-value function, we can now combine a ZDD with such a specific action-value function, where ZDD is a data structure for a set of sparse bit vectors. The combination of a specific action-value function and ZDD enables a compact representation that supports several operations and faster max / min for arbitrary element-level weights.

[0068] As a result, state st In this scenario, if the agent wishes to stop, the environment will stop, and the molecules s t Returns the characteristic. Otherwise, perform action a. t Given, the environment is the current molecular s t Chemical reaction a t Apply the following state s t+1 This is derived, and this is the product of the chemical reaction. The chemical reaction can yield multiple candidate products. In this example, the exemplary embodiment considers the candidate that has the lowest synthetic accessibility score as the product.

[0069] Figure 5 shows an algorithm 110 that utilizes adapted Q-iterations using ZDD according to one embodiment of the present invention.

[0070] Figure 6 is a block / flow diagram of an exemplary practical application for chemical discovery of a method in which chemical properties are predicted or a new molecule is generated, according to one embodiment of the present invention.

[0071] In one practical application related to materials discovery, a molecular structure 120 with chemical properties can be used as a dataset for generating a property prediction model 124 through the extraction of design knowledge 122. This can be achieved by utilizing offline RL 121 with fitted Q iterations using ZDD. A molecular design 125 with the target properties can then be used to generate a new molecular structure 126. Thus, one example is shown in which artificial intelligence is applied to the molecular structure 130 to generate predicted properties 132. The applied artificial intelligence may be via offline RL 134 through fitted Q iterations 136 with ZDD.

[0072] In an alternative embodiment, the agent recommends a set of items, and the action is a feasible combination of those items.

[0073] In further alternative embodiments, a text-based action space is provided in which an agent interacts with the environment by using text. The action can be a grammatically correct sentence.

[0074] In a further alternative embodiment, a graph-based action space is provided in which the action is to select a path within the knowledge graph to enable path planning and reasoning on the knowledge graph.

[0075] The benefits of all embodiments are, in particular,

number

[0076] In summary, material properties such as hardness, melting point, ionic conductivity, glass transition temperature, molecular atomization energy, and lattice constant can be described at either a macroscopic or microscopic level. Two common methods exist for studying material properties: computational simulation and experimental measurement. Both methods involve complex operation and experimental setup. Therefore, constructing computational simulations that fully capture the complex logical relationships between material properties and their contributing factors is quite difficult, and some of these relationships may even be unknown. Moreover, experiments performed to measure the properties of compounds are generally conducted in the later stages of material selection.

[0077] Therefore, if the results are unsatisfactory, a vast amount of time and experimental resources are invested until it is proven to be wasted. In addition, in many cases, studying material properties is difficult or nearly impossible, even through a large amount of computational or experimental effort. Thus, there is an urgent need to develop intelligent and high-performance predictive models that can accurately predict material properties at low time and computational costs. Machine learning concerns the construction and research of algorithms that can learn patterns from data. The basic concept of using machine learning methods for material property prediction is to analyze and map the relationships (mostly nonlinear) between material properties and their related factors by extracting knowledge from existing empirical data. Figures 1 to 6 illustrate the fundamental framework for applying machine learning to material property prediction by utilizing fitted Q iterations with ZDD.

[0078] In particular, large action spaces are a common problem in reinforcement learning, and many techniques have been developed to mitigate this. These techniques can be used in both online and offline settings, however, few methods focus on offline settings. The inventiveness of the exemplary embodiment lies in the offline RL setting and a specific action-value function of the form Q(s,a;w,θ)=w·φ(s,a)+f(s;θ), where φ(s,a) is a bit vector. ZDD cannot be combined until the exemplary embodiment focuses on this particular case, i.e., the specific action-value function Q(s,a;w,θ)=w·φ(s,a)+f(s;θ). Furthermore, it should be noted that decision diagrams are often combined with planning rather than reinforcement learning, where the environment is assumed to be known. Therefore, the combination of ZDD with adapted Q-iterations in offline RL focused on a specific action-value function is unique.

[0079] In an alternative embodiment, a single ZDD can be used instead of multiple ZDDs.

[0080] For any set B of bit vectors, let ZDD(B) be the ZDD that represents that set.

[0081] In a single ZDD embodiment, an exemplary embodiment is:

number

[0082]

number

number

[0083] In another embodiment, the method is as follows:

number

[0084] A set of D-dimensional one-hot vectors D ⊂{0,1} D And, one-hot vectors are used for each state

number

number

number

number

[0085] In that case, the exemplary embodiment can construct the following:

[0086]

number

[0087] any

number

number

[0088] Therefore, the maximization step can be performed by first extracting the sub-ZDDs.

[0089] In a further alternative embodiment, the ZDD can be replaced with a Binary Decision Diagram (BDD) because BDDs also support maximization operations.

[0090] In yet another alternative embodiment, ZDD construction consumes a lot of memory, even when the resulting ZDD is small. Therefore, in one example, a fixed dataset

number

[0091] Figure 7 is a block / flow diagram of an exemplary processing system that utilizes adapted Q iterations using ZDD according to one embodiment of the present invention.

[0092] Figure 7 shows a block diagram of the components of system 200, including computing device 205. It should be understood that Figure 7 provides merely an example of one implementation and does not imply any limitations regarding environments in which different embodiments can be implemented. Many modifications can be made to the illustrated environment.

[0093] The computing device 205 includes a communication fabric 202, which provides communication between the computer processor 204, memory 206, persistent storage 208, communication unit 210, and input / output (I / O) interface 212. The communication fabric 202 can be implemented using any architecture designed to pass data or control information, or both, between processors (e.g., microprocessors, communication and network processors, etc.), system memory, peripheral devices, and any other hardware components in the system. For example, the communication fabric 202 can be implemented using one or more buses.

[0094] Memory 206, cache memory 216, and persistent storage 208 are computer-readable storage media. In this embodiment, memory 206 includes random access memory (RAM) 214. In another embodiment, memory 206 may be flash memory. Generally, memory 206 may include any suitable volatile or non-volatile computer-readable storage media.

[0095] In some embodiments of the present invention, program 225 is included and operates by the AI ​​accelerator chip 222 as a component of the computing device 205. In other embodiments, program 225 is stored in persistent storage 208 for execution by the AI ​​accelerator chip 222 (implementing adapted Q iterations using ZDDs) via one or more of the memories 206 in combination with one or more of the respective computer processors 204. In this embodiment, persistent storage 208 includes a magnetic hard disk drive. Instead of, or in addition to, a magnetic hard disk drive, persistent storage 208 may include a solid-state hard drive, a semiconductor storage device, read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, or any other computer-readable storage medium capable of storing program instructions or digital information.

[0096] The media used by persistent storage 208 can also be removable. For example, a removable hard drive can be used for persistent storage 208. Other examples include optical and magnetic disks, thumb drives, and smart cards, which are inserted into the drive for transfer to another computer-readable storage medium that is also part of persistent storage 208.

[0097] In these examples, the communication unit 210 provides communication with other data processing systems or devices, including resources of the distributed data processing environment. In these examples, the communication unit 210 includes one or more network interface cards. The communication unit 210 can provide communication through the use of either or both physical and wireless communication links. The deep learning program 225 can be downloaded to persistent storage 208 through the communication unit 210.

[0098] The I / O interface 212 enables data input and output to other devices that can be connected to the computing system 200. For example, the I / O interface 212 can provide connection to an external device 218 such as a keyboard, keypad, touchscreen, or any other suitable input device, or a combination thereof. The external device 218 may also include portable computer-readable storage media such as thumb drives, portable optical disks or magnetic disks, and memory cards.

[0099] The display 220 provides a mechanism for displaying data to the user and can, for example, function as a computer monitor.

[0100] Figure 8 shows a real-world application of an artificial intelligence (AI) accelerator chip that utilizes adapted Q-iteration using ZDD, according to one embodiment of the present invention.

[0101] The artificial intelligence (AI) accelerator chip 222 can implement adapted Q iterations 301 using ZDD and can be used in a wide variety of practical applications, including, but not limited to, robotics 310, industrial applications 312, mobile or Internet of Things (IoT) 314, personal computing 316, consumer electronics 318, server data centers 320, physical and chemical applications 322, healthcare applications 324, and financial applications 326.

[0102] Figure 9 is a block / flow diagram of an exemplary cloud computing environment according to one embodiment of the present invention.

[0103] While this invention includes a detailed description of cloud computing, it should be understood that implementations of the teachings described herein are not limited to cloud computing environments. Rather, embodiments of this invention can be implemented in combination with any other type of computing environment that is currently known or may be developed in the future.

[0104] Cloud computing is a service delivery model that enables convenient on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and deployed with minimal administrative effort or interaction with service providers. This cloud model may include at least five characteristics, at least three service models, and at least four deployment models.

[0105] The characteristics are as follows:

[0106] On-demand self-service: Cloud consumers can unilaterally provision computing power, such as server time and network storage, automatically as needed, without requiring human interaction with service providers.

[0107] Broad network access: This capability is available over the network and accessed through standard mechanisms that facilitate use by heterogeneous thin or thick client platforms (e.g., mobile phones, laptops, and PDAs).

[0108] Resource pooling: A provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, with different physical and virtual resources dynamically allocated and reallocated according to demand. While consumers generally have no control or knowledge of the exact location of the resources provided, there is location independence in that they may be able to specify the location at a higher level of abstraction (e.g., country, state, or data center).

[0109] Rapid resilience: This capability allows for rapid and elastic provisioning, sometimes automatically, enabling quick scaling out and rapid release and rapid scaling in. To consumers, the capacity available for provisioning often appears unlimited and can be purchased in any quantity at any time.

[0110] Services measured: Cloud systems automatically control and optimize resource usage by leveraging metric capabilities at a level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, thereby providing transparency to both service providers and consumers.

[0111] The service model is as follows:

[0112] Software as a Service (SaaS): The ability provided to consumers is the use of a provider's applications running on cloud infrastructure. These applications are accessible from various client devices through thin client interfaces such as web browsers (e.g., web-based email). Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, storage, or even individual application capabilities, with the possible exception of limited user-specific application configuration settings.

[0113] Platform as a Service (PaaS): The ability provided to consumers is to deploy applications they have created or acquired, written using programming languages ​​and tools supported by the provider, on cloud infrastructure. Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but they do control the deployed applications and, in some cases, the configuration of the application hosting environment.

[0114] Infrastructure as a Service (IaaS): The ability provided to consumers is to provision processing, storage, networking, and other basic computing resources, where consumers can deploy and run any software, including operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but they do control the operating system, storage, and deployed applications, and in some cases have limited control over selected networking components (e.g., host firewalls).

[0115] The deployment model is as follows:

[0116] Private Cloud: This cloud infrastructure operates solely for a specific organization. A private cloud may be managed by that organization or a third party and may reside on-premises or off-premises.

[0117] Community Cloud: This cloud infrastructure is shared by several organizations and supports specific communities that share common interests (e.g., mission, security requirements, policies, and compliance considerations). The community cloud may be managed by those organizations or third parties and may reside on-premises or off-premises.

[0118] Public Cloud: This cloud infrastructure is made available to the general public or large industry groups and is owned by an organization that sells cloud services.

[0119] Hybrid Cloud: This cloud infrastructure is a combination of two or more clouds (private, community, or public) that remain independent entities but are joined together by standard or proprietary technologies that enable data and application portability (e.g., cloud bursting for load balancing between clouds).

[0120] Cloud computing environments are service-oriented, focusing on statelessness, low coupling, modularity, and semantic interoperability. At the core of cloud computing lies an infrastructure that includes a network of interconnected nodes.

[0121] Referring here to Figure 9, an exemplary cloud computing environment 450 is shown to enable a use case of the present invention. As shown, the cloud computing environment 450 comprises one or more cloud computing nodes 410 to which local computing devices used by a cloud consumer, such as a personal digital assistant (PDA) or mobile phone 454A, a desktop computer 454B, a laptop computer 454C, or an automotive computer system 454N, or a combination thereof, can communicate. The nodes 410 can communicate with each other. The nodes 410 can be physically or virtually grouped within one or more networks, such as a private cloud, community cloud, public cloud or hybrid cloud, or a combination thereof, as described above in this specification (not shown). This makes it possible for the cloud computing environment 450 to provide infrastructure, platform or software, or a combination thereof, as a service to cloud consumers, without requiring them to maintain resources on their local computing devices for that purpose. The types of computing devices 454A-N shown in Figure 9 are for illustrative purposes only, and it should be understood that the computing node 410 and the cloud computing environment 450 can communicate with any type of computerized device via any type of network, or a network addressable connection, or both (for example, using a web browser).

[0122] Figure 10 is a schematic diagram of an exemplary abstraction model layer according to one embodiment of the present invention. The components, layers, and functions shown in Figure 10 are for illustrative purposes only and should be understood in advance that embodiments of the present invention are not limited thereto. As shown, the following layers and corresponding functions are provided:

[0123] The hardware and software layer 560 comprises hardware components and software components. Examples of hardware components include a mainframe 561, a RISC (Reduced Instruction Set Computer) architecture-based server 562, a server 563, a blade server 564, a storage device 565, and network and networking components 566. In some embodiments, the software components include network application server software 567 and database software 568.

[0124] The virtualization layer 570 provides an abstraction layer that may provide examples of virtual entities, including virtual servers 571, virtual storage 572, virtual networks 573 including virtual private networks, virtual applications and operating systems 574, and virtual clients 575.

[0125] In one example, the management layer 580 may provide the following functions: Resource provisioning 581 provides dynamic procurement of computing and other resources used to perform tasks within the cloud computing environment. Measurement and pricing 582 provides cost tracking as resources are used within the cloud computing environment and charges or invoices for the consumption of these resources. In one example, these resources may include application software licenses. Security provides identity verification for cloud consumers and tasks, as well as protection for data and other resources. The user portal 583 provides consumers and system administrators with access to the cloud computing environment. Service level management 584 provides cloud computing resource allocation and management to ensure that required service levels are met. Service level agreement (SLA) planning and execution 585 provides pre-arrangements and procurement of cloud computing resources where future requirements are expected to conform to the SLA.

[0126] The workload layer 590 provides examples of functions that can be utilized in a cloud computing environment. Examples of workloads and functions that can be provided from this layer include mapping and navigation 541, software development and lifecycle management 592, virtual classroom education delivery 593, data analysis processing 594, transaction processing 595, and adapted Q iterations 301 using ZDD.

[0127] The present invention may be a system, method, or computer program product, or a combination thereof. The computer program product may include a computer-readable storage medium (or a set of mediums) having computer-readable program instructions that cause a processor to execute an aspect of the present invention.

[0128] A computer-readable storage medium can be a tangible device capable of holding and storing instructions for use by an instruction execution device. A computer-readable storage medium can be, but is not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of those described above. A non-exhaustive list of more specific examples of computer-readable storage media includes, namely, portable computer diskettes, hard disks, random access memory, read-only memory, erasable programmable read-only memory (EPROM or flash memory), static random access memory, portable compact disk read-only memory (CD-ROM), digital multipurpose disks (DVDs), memory sticks, floppy disks, mechanically encoded devices such as punch cards or grooved raised structures recording instructions, and any suitable combination of those described above. When used herein, computer-readable storage media should not be interpreted as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through optical fiber cables), or transient signals such as electrical signals transmitted through wires.

[0129] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers or edge servers, or a combination thereof. A network adapter card or network interface within each computing / processing device receives computer-readable program instructions from the network and transfers such computer-readable program instructions for storage in a computer-readable storage medium within each computing / processing device.

[0130] The computer-readable program instructions that perform the operation of the present invention may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, the one or more programming languages ​​including object-oriented programming languages ​​such as Smalltalk®, C++, etc., and conventional procedural programming languages ​​such as the C programming language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer as a standalone software package, partially on the user's computer and partially on a remote computer, or fully on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or wide area network (WAN), and the connection may be to an external computer (for example, via the Internet using an Internet service provider). In some embodiments, for example, an electronic circuit including a programmable logic circuit, a field-programmable gate array (FPGA), or a programmable logic array (PLA) can be personalized by executing computer-readable program instructions using state information of computer-readable program instructions in order to perform an aspect of the present invention.

[0131] Aspects of the present invention are described herein with reference to flowcharts or block diagrams, or both, of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It will be understood that each block in a flowchart or block diagram, or both, and any combination of blocks in a flowchart or block diagram, or both, can be implemented by computer-readable program instructions.

[0132] These computer-readable program instructions can be provided to at least one processor of a general-purpose computer, a dedicated computer, or another programmable data processing device to generate a machine, thereby creating means for implementing functions / operations specified in one or more blocks or modules of a flowchart or block diagram, or both, through the instructions executed via the processor of the computer or other programmable data processing device. Furthermore, these computer-readable program instructions can be stored in a computer-readable storage medium, and such instructions can be used to instruct a computer, a programmable data processing device, or another device, or a combination thereof, to function in a specific manner, thereby the computer-readable storage medium storing the instructions can contain a product containing instructions that implement modes of functions / operations specified in one or more blocks or modules of a flowchart or block diagram, or both.

[0133] Furthermore, computer-readable program instructions can be loaded into a computer, other programmable data processing device, or other device to execute a series of operational blocks / stages on the computer, other programmable device, or other device, thereby generating a computer implementation process in which the instructions executed on the computer, other programmable device, or other device implement the functions / operations specified by one or more blocks or modules in a flowchart or block diagram, or both.

[0134] The flowcharts and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions containing one or more executable instructions that implement a specified logical function. In some alternative implementations, the functions described in a block may be performed in an order different from the order shown in the drawings. For example, two blocks shown consecutively may actually be executed substantially simultaneously, and blocks may be executed in reverse order depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, or both, and any combination of blocks in a block diagram or flowchart, or both, may be implemented by a dedicated hardware-based system that performs a specified function or operation, or a combination of dedicated hardware and computer instructions.

[0135] Any reference in this specification to “one embodiment” or “one embodiment” of the Principle and other variations means that the specific features, structures, characteristics, etc. described in relation to that embodiment are included in at least one embodiment of the Principle. Therefore, the phrases “in one embodiment” or “in one embodiment” appearing in various places throughout this specification, and the appearance of any other variations, do not necessarily all refer to the same embodiment.

[0136] Please understand that the use of any of the following " / ", "and / or", and "at least one of" is intended to include, for example, "A / B", "A and / or B", and "at least one of A and B", the selection of only the first enumerated option (A), or only the second enumerated option (B), or the selection of both options (A and B). As a further example, in the case of "A, B, and / or C", and "at least one of A, B, and C", such phrasing is intended to include the selection of only the first enumerated option (A), or only the second enumerated option (B), or only the third enumerated option (C), or only the first and second enumerated options (A and B), or only the first and third enumerated options (A and C), or only the second and third enumerated options (B and C), or the selection of all three options (A, B, and C). This can be extended to many of the listed items, as would be readily apparent to those skilled in the art in the relevant and related fields.

[0137] Preferred embodiments (exemplary and not intended to be limiting) of a method for faster fitted Q iterations using zero-suppression decision diagrams have been described, but it should be noted that modifications and variations can be made by those skilled in the art in view of the above teachings. Therefore, it should be understood that in the specific embodiments described, modifications can be made that fall within the scope of the invention, as outlined by the appended claims. Thus, aspects of the invention have been described with the details and specificity required by patent law, but what is claimed and desired to be protected by patent certificate is described in the appended claims.

Claims

1. A computer implementation method for estimating a state-action value function for fitted Q-iterations, A step of obtaining a set of tuples D and a discount rate γ, wherein each of the tuples in the set includes a state s, an action a, a reward r, and a resulting state s'. For each of the resulting states s' of the set of tuples, feature vector [Math 1] The step of constructing a zero-suppression decision diagram (ZDD), wherein the feature vector φ(s, a) is a sparse bit vector {0, 1} Ｄ And, [Math 2] The steps are a set of actions applicable in state s'. Parameters of the state-action value function Q(s, a; w, θ) [Math 3] , the step of updating θ, The step of repeating the updating step a predetermined number of times by incrementing t. Equipped with, The updating step includes an operation to find the maximum value of the state-action value function Q for the set of actions applicable in state s', and in the operation to find the maximum value, the ZDD of the constructed feature vector is used. Computer implementation method.

2. The computer implementation method according to claim 1, wherein the updating step includes an operation to find max a'∈A(s') w・φ(s', a') for the set of actions A(s') applicable in the state s', the operation being calculated using the ZDD of the constructed feature vector.

3. The update of the aforementioned parameters is [Math 4] Calculated by \(Q(s,a;w,\theta)=w\cdot\varphi(s,a)+f(s;Q), max\) ａ'∈Ａ（ｓ'） \(Q(s',a';w\) ｔ ,\theta ｔ ) = f(s';\theta ｔ ) + max ａ'∈Ａ（ｓ'） w ｔ \(\cdot\varphi(s',a')\), and \(max\) ａ'∈Ａ（ｓ'） w ｔ \(\cdot\varphi(s',a')\) is calculated using the ZDD, the computer-implemented method according to claim 1.

4. The computer implementation method according to any one of claims 1 to 3, wherein the adapted Q iteration using the ZDD is used for computational material discovery to generate a new molecular structure that satisfies target characteristic values.

5. The computer implementation method according to any one of claims 1 to 3, wherein the state is the current molecule, the action is a chemical reaction, and the reward is the characteristic to be maximized.

6. The computer implementation method according to claim 5, wherein the chemical reaction yields a plurality of candidates.

7. The computer implementation method according to claim 6, wherein the candidate having the lowest ease of synthesis score among the plurality of candidates is selected as the product of the chemical reaction.

8. The computer implementation method according to any one of claims 1 to 3, wherein the adapted Q-iterations using the ZDD are used in offline reinforcement learning.

9. A computer program for estimating a state-action value function for fitted Q-iterations, wherein the computer... A procedure for obtaining a set of tuples D and a discount rate γ, wherein each of the tuples in the set includes a state s, an action a, a reward r, and a resulting state s'. For each of the resulting states s' of the set of tuples, feature vector [Math 5] A procedure for constructing a zero-suppression decision diagram (ZDD), wherein the feature vector φ(s, a) is a sparse bit vector {0, 1} Ｄ And, [Math 6] This is a set of actions applicable in state s', which is a procedure, Parameters of the state-action value function Q(s, a; w, θ) [Number 7] The procedure for updating θ, The procedure of repeating the updating procedure a predetermined number of times by incrementing t. Have them do it, The updating procedure includes an operation to find the maximum value of the state-action value function Q for the set of actions applicable in state s', and in the operation to find the maximum value, the ZDD of the constructed feature vector is used. Computer program.

10. The computer program according to claim 9, wherein the updating procedure includes an operation to find max a'∈A(s') w・φ(s', a') for the set of actions A(s') applicable in the state s', the operation being calculated using the ZDD of the constructed feature vector.

11. The update of the aforementioned parameters is [Number 8] It is calculated by Q(s, a; w, θ) = w・φ(s, a) + f(s; Q), max ａ'∈Ａ（ｓ'） Q(s', a'; w ｔ , θ ｔ ) = f(s'; θ ｔ ) + max ａ'∈Ａ（ｓ'） lol ｔ φ(s', a') and max ａ'∈Ａ（ｓ'） lol ｔ The computer program according to claim 9, wherein φ(s', a') is calculated using the ZDD.

12. The computer program according to any one of claims 9 to 11, wherein the adapted Q iteration using the ZDD is used for computational material discovery to generate a new molecular structure that satisfies a target characteristic value.

13. The computer program according to any one of claims 9 to 11, wherein the state is the current molecule, the action is a chemical reaction, and the reward is a characteristic to be maximized.

14. The computer program according to claim 13, wherein the chemical reaction yields a plurality of candidates.

15. The computer program according to claim 14, wherein the candidate having the lowest ease of synthesis score among the plurality of candidates is selected as the product of the chemical reaction.

16. The computer program according to any one of claims 9 to 11, wherein the adapted Q-iteration using the ZDD is used in offline reinforcement learning.

17. A system for estimating a state-action value function for fitted Q-iterations, Memory and One or more processors that communicate with the aforementioned memory The one or more processors are provided with, The method involves obtaining a set of tuples D and a discount rate γ, wherein each of the tuples in the set includes a state s, an action a, a reward r, and the resulting state s'. For each of the resulting states s' of the set of tuples, feature vector [Number 9] The method involves constructing a zero-suppression decision diagram (ZDD), where the feature vector φ(s, a) is a sparse bit vector {0, 1} Ｄ And, [Number 10] This is the set of actions applicable in state s', Parameters of the state-action value function Q(s, a; w, θ) [Math 11] , updating θ, The process of updating is repeated a predetermined number of times by incrementing t. It is configured to do the following: The updating includes an operation to find the maximum value of the state-action value function Q for the set of actions applicable in state s', and in the operation to find the maximum value, the ZDD of the constructed feature vector is used. system.

18. The system according to claim 17, wherein the updating includes an operation to find max a'∈A(s') w・φ(s', a') for the set of actions A(s') applicable in the state s', the operation being calculated using the ZDD of the constructed feature vector.

19. The update of the aforementioned parameters is [Math 12] It is calculated by Q(s, a; w, θ) = w・φ(s, a) + f(s; Q), max ａ'∈Ａ（ｓ'） Q(s', a'; w ｔ , θ ｔ ) = f(s'; θ ｔ ) + max ａ'∈Ａ（ｓ'） lol ｔ φ(s', a') and max ａ'∈Ａ（ｓ'） lol ｔ The system according to claim 17, wherein φ(s', a') is calculated using the ZDD.

20. The system according to any one of claims 17 to 19, wherein the adapted Q iteration using the ZDD is used for computational material discovery to generate a new molecular structure that satisfies target characteristic values.

21. The system according to any one of claims 17 to 19, wherein the state is the current molecule, the action is a chemical reaction, and the reward is the characteristic to be maximized.

22. The system according to claim 21, wherein the chemical reaction yields a plurality of candidates.

23. The system according to claim 22, wherein the candidate having the lowest ease of synthesis score among the plurality of candidates is selected as the product of the chemical reaction.

24. The adapted Q-iteration using the ZDD is used in offline reinforcement learning, according to any one of claims 17 to 19.

25. A computer implementation method for estimating a state-action value function for fitted Q-iterations, A step of obtaining a set of tuples D and a discount rate γ, wherein each of the tuples in the set includes a state s, an action a, a reward r, and a resulting state s'. For each of the resulting states s' of the set of tuples, feature vector [Number 17] The step of constructing multiple zero-suppression decision diagrams (ZDDs), wherein the feature vector φ(s, a) is a sparse bit vector {0, 1} Ｄ And, [Number 18] The steps are a set of actions applicable in state s'. Parameters of the state-action value function Q(s, a; w, θ) [Number 19] , the step of updating θ, The step of repeating the updating step a predetermined number of times by incrementing t. Equipped with, The updating step includes an operation to find the maximum value of the state-action value function Q for the set of actions applicable in state s', and in the operation to find the maximum value, the plurality of ZDDs of the constructed feature vector are used. Computer implementation method.

26. The computer implementation method according to claim 25, wherein the updating step includes an operation to find max a'∈A(s') w・φ(s', a') for the set of actions A(s') applicable in the state s', the operation being calculated using the plurality of ZDDs of the constructed feature vector.

27. The update of the aforementioned parameters is [Number 20] It is calculated by Q(s, a; w, θ) = w・φ(s, a) + f(s; Q), max ａ'∈Ａ（ｓ'） Q(s', a'; w ｔ , θ ｔ ) = f(s'; θ ｔ ) + max ａ'∈Ａ（ｓ'） lol ｔ φ(s', a') and max ａ'∈Ａ（ｓ'） lol ｔ The computer implementation method according to claim 25, wherein φ(s', a') is calculated using the plurality of ZDDs.

Images