Information processing system

Abstract

This information processing system for optimizing a compound on the basis of a plurality of characteristics includes: a weight generation unit that generates weights for a plurality of characteristics; a generative model learning unit that learns a generative model from compound data; and a distributed reinforcement learning execution unit for executing distributed reinforcement learning using the weights and the generative model. In this way, it is possible to optimize a compound on the basis of a plurality of characteristics so that experiments are minimized when there are limited computational resources.

Description

Information Processing Systems

[0001] The present disclosure relates to an information processing system.

[0002] In the fields of materials development and drug discovery, in order to generate promising compounds, compounds are optimized based on multiple properties using machine learning and other methods.

[0003] Patent Document 1 describes generating selection criteria based on one or more desired properties to identify compounds having the desired properties.

[0004] Special Publication No. 2001-507675

[0005] When optimizing a compound based on multiple properties, the final compound obtained will differ depending on how important each property is. For this reason, weights may be set for each property that serves as an experimental condition, and multiple experiments may be performed. Compound optimization using machine learning requires enormous computational resources, so weights must be set to minimize the number of experiments.

[0006] Patent Document 1 does not disclose the idea of ​​setting weights so as to minimize experiments when optimizing a compound based on multiple properties.

[0007] The objective of the present disclosure is to optimize compounds based on multiple properties with minimal experimentation and limited computational resources.

[0008] The information processing system disclosed herein is a system for optimizing a compound based on multiple characteristics, and includes a weight generation unit that generates weights for the multiple characteristics, a generative model training unit that trains a generative model from compound data, and a distributed reinforcement learning execution unit that performs distributed reinforcement learning using the weights and the generative model.

[0009] The present disclosure allows compounds to be optimized based on multiple properties with limited computational resources and minimal experimentation.

[0010] 1 is a block diagram showing a hardware configuration of an information processing system according to a first embodiment. FIG. 2 is a flowchart showing a weight generation process according to the first embodiment. FIG. 3 is a table showing an example of weights generated in the weight generation process according to the first embodiment. FIG. 4 is a flowchart showing a generative model learning process according to the first embodiment. FIG. 5 is a block diagram showing a configuration of data learning for a learner in the distributed reinforcement learning execution process according to the first embodiment. FIG. 6 is a block diagram showing a configuration of data generation for an actor in the distributed reinforcement learning execution process according to the first embodiment. FIG. 7 is a block diagram showing a configuration of a reward calculation unit 603 in the distributed reinforcement learning execution process according to the first embodiment. FIG. 8 is a flowchart showing a distributed reinforcement learning execution process according to the first embodiment. FIG. 9 is a table showing an example of a learning result acquired in the distributed reinforcement learning execution process according to the first embodiment. FIG. 10 is a diagram showing an example of an input / output screen related to the weight generation process according to the first embodiment. FIG. 11 is a diagram showing an example of an input / output screen related to the distributed reinforcement learning execution process according to the first embodiment. FIG. 12 is a block diagram showing a hardware configuration of an information processing system according to a second embodiment. FIG. 13 is a flowchart showing an updated weight generation process according to the second embodiment. FIG. 14 is a diagram showing an example of an input / output screen related to the updated weight generation process according to the second embodiment.

[0011] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0012] First Embodiment In the first embodiment, an example will be described in which weights for a plurality of characteristics are generated and distributed reinforcement learning is performed to optimize a compound based on a plurality of characteristics.

[0013] Distributed reinforcement learning consists of a large number of actors that generate data through distributed processing and a learner that learns the data obtained from each actor. At a given time step t, each actor selects an action a(t) using a Q-network from a state s(t), which is a feature of the compound, and outputs a state s(t+1) and a reward r(t) updated by the environment from the action a(t). The learner learns this experience data (state s(t), state s(t+1), action a(t), reward r(t)) and updates the learning parameters of the Q-network. By repeating the above process, each actor can explore the feature space of the compound and optimize the compound.

[0014] When performing distributed reinforcement learning based on multiple characteristics, it is necessary to set the degree of importance each actor places on each characteristic when exploring the feature space of compounds.

[0015] Therefore, in this embodiment, a weight to be assigned to each actor is generated based on the D-optimality criterion used to optimize experimental conditions in experimental design. Specifically, a matrix X is defined, with actors as rows and weight values ​​for multiple characteristics as columns. The D-optimality criterion is calculated using the determinant of the product of the transposed matrix of X and X, and weight values ​​are generated so as to maximize the D-optimality criterion. This allows each actor to search the feature space of compounds over a wide range without bias for each characteristic.

[0016] FIG. 1 is a block diagram showing the hardware configuration of an information processing system according to this embodiment.

[0017] The information processing system shown in this figure includes a server 101 and a database 102 .

[0018] The server 101 is connected to a database 102 so as to be able to access data stored therein.

[0019] The server 101 is a computer having an input device 103, an output device 104, an arithmetic unit 105 for executing a program, a memory 106 for storing the program, and a storage device 107.

[0020] The input device 103 is an interface that receives input to the server 101, and is a mouse, keyboard, or the like.

[0021] The output device 104 is a display device, a printer, or the like, and outputs the results of the calculations performed by the calculation device 105 .

[0022] The arithmetic unit 105 is a central processing unit (CPU) and a graphics processing unit (GPU), and executes a program loaded into the memory 106 .

[0023] The memory 106 includes a non-volatile memory element, ROM (Read Only Memory), and a volatile memory element, RAM (Random Access Memory). The ROM stores unchanging programs (e.g., BIOS (Basic Input / Output System)). The RAM is a high-speed, volatile memory element, such as DRAM (Dynamic Random Access Memory), and temporarily stores programs stored in the storage device 107 and data used when the programs are executed.

[0024] The storage device 107 is a non-volatile storage device such as a magnetic storage device (HDD) or a flash memory (SSD), and stores programs executed by the arithmetic device 105 and data used when the programs are executed.

[0025] Specifically, the storage device 107 stores programs for realizing each of the weight generation unit 108, the generative model training unit 109, and the distributed reinforcement learning execution unit 110. In other words, the programs for realizing the weight generation unit 108, the generative model training unit 109, and the distributed reinforcement learning execution unit 110 are executed by the calculation device 105. That is, it can be said that the weight generation unit 108, the generative model training unit 109, and the distributed reinforcement learning execution unit 110 are built into the calculation device 105 as hardware.

[0026] The weight generating unit 108 generates weights for each characteristic by executing a predetermined program (see FIG. 2).

[0027] The generative model learning unit 109 learns a generative model from compound data by executing a predetermined program (see FIG. 4).

[0028] The distributed reinforcement learning execution unit 110 executes a predetermined program to perform distributed reinforcement learning using the weights of each characteristic and the generative model (see FIG. 8).

[0029] The database 102 stores data to be analyzed by the server 101, i.e., a compound data storage unit 111. The compound data stored in the compound data storage unit 111 is a chemical structure expressed in SMILES notation or the like. Here, "SMILES" is an abbreviation for Simplified Molecular Input Line Entry System.

[0030] FIG. 2 is a flowchart showing the weight generation process of this embodiment.

[0031] This weight generation process is executed by the weight generation unit 108 of the server 101 shown in FIG.

[0032] 2, first, weight candidates are generated by random number generation (step S201). Specifically, the weight candidates take values ​​from 0 to 1 for each characteristic, and a predetermined number of random numbers are generated for each characteristic so that the sum of the values ​​for each characteristic is 1.

[0033] Next, the weight acquisition range is set (step S202). Specifically, the difference between the maximum and minimum values ​​of the weight candidates for each characteristic is calculated, and a value is obtained by multiplying this difference by a predetermined ratio. For example, if the maximum value of the weight candidates for a certain characteristic is Wmax, the minimum value is Wmin, and the predetermined ratio is r, the acquisition range for that characteristic is calculated as (Wmax - Wmin) x r. This makes it possible to determine, for each characteristic, how widely the extracted weights are distributed compared to the weight candidates in step S206, which will be described later.

[0034] Next, weights are randomly selected from the weight candidates (step S203), and the D-optimality criterion is calculated (step S204). Specifically, when the rows of a matrix are the weights selected at random so that the number is the same as the number of actors, and the columns are the weight values ​​for multiple characteristics, X, the D-optimality criterion is calculated using the determinant of the product of the transposed matrix of X and X.

[0035] Next, it is determined whether the calculated D-optimization criterion is greater than the maximum value up to the previous time (step S205). If this condition is not met, the process returns to step S203. On the other hand, if this condition is met, it is determined whether the extracted weights are within the obtainable range (step S206). Specifically, for the extracted weights, the obtainable range of the weights is calculated for each characteristic using the same method as in step S202, and it is determined whether this obtainable range is greater for all characteristics than the obtainable range set in step S202. If this condition is not met, the process returns to step S203. On the other hand, if this condition is met, the extracted weights are obtained (step S207), and this process ends.

[0036] FIG. 3 is a table showing an example of weights generated in the weight generation process of this embodiment.

[0037] As shown in this figure, the weight includes data of an actor ID 301, a weight of characteristic 1 302, a weight of characteristic 2 303, and a weight of characteristic 3 304.

[0038] The actor ID 301 is an identifier that uniquely identifies an actor.

[0039] A weight 302 for characteristic 1, a weight 303 for characteristic 2, and a weight 304 for characteristic 3 are weights assigned to each actor. Here, an example is shown in which weights for three characteristics are assigned to each actor.

[0040] FIG. 4 is a flowchart showing the generative model learning process of this embodiment.

[0041] This generative model learning process is executed by the generative model learning unit 109 of the server 101 shown in FIG.

[0042] 4, first, compound data is acquired from the compound data storage unit 111 (step S401).

[0043] Next, a generative model is learned from the compound data by machine learning (step S402), and this process ends. Specifically, a generative model such as a variational autoencoder is learned. This enables encoding of the compound data, which converts the chemical structure into feature quantities, and decoding of the feature quantities, which converts the chemical structure, in the distributed reinforcement learning execution process described below.

[0044] FIG. 5 is a block diagram showing the configuration of data learning for a learner in the distributed reinforcement learning execution process of this embodiment.

[0045] As shown in this figure, the data learning of the learner 501 is composed of the learner 501 and the replay memory 502. The learner 501 is composed of the Q network 503, the Q ＊ The Q network 503 includes a learning parameter θ 506. ＊ The network 504 uses the learning parameters θ ＊ Includes 507.

[0046] The learner 501 is a machine learning model that executes TD learning (Temporal Difference Learning).

[0047] The replay memory 502 stores multiple pieces of experience data (state s(t), state s(t+1), action a(t), reward r(t)). Prioritized Experience Replay assigns a high priority to experience data with a large TD error (Temporal Difference Error) calculated in TD learning.

[0048] Q network 503 and Q ＊Network 504 is an action value function of the same configuration. Specifically, it is a neural network model that calculates the expected value of reward, such as Amortized Q-Learning, and performs machine learning to maximize the value of action a(t) for state s(t).

[0049] The learning parameter update unit 505 updates the learning parameter θ506 of the Q network 503 by the gradient method. The learning parameter θ506 of the Q network 503 is updated by the Q ＊ The learning parameter θ of the network 504 ＊ Copy to 507.

[0050] FIG. 6 is a block diagram showing the configuration for generating actor data in the distributed reinforcement learning execution process of this embodiment.

[0051] As shown in this figure, the data generation of the actor 601 is composed of the actor 601 and the replay memory 502. The actor 601 includes a Q network 503 and an environment 602. The environment 602 includes a reward calculation unit 603.

[0052] The actor 601 generates experience data (state s(t), state s(t+1), action a(t), reward r(t)) and stores it in the replay memory 502. There may be multiple actors 601.

[0053] The environment 602 outputs a state s(t+1) and a reward r(t) from the action a(t) selected by the Q-network 503 for the state s(t). The state s(t+1) is output from the action a(t) with a predetermined transition probability. The reward r(t) is output from the state s(t+1) by the reward calculation unit 603, which will be described later.

[0054] FIG. 7 is a block diagram showing the configuration of the reward calculation unit 603 in the distributed reinforcement learning execution process of this embodiment.

[0055] As shown in this figure, a generative model 701 is a generative model learned in the generative model learning process, and decodes a state s(t+1) into a chemical structure 702.

[0056] The characteristic value calculation 703 calculates characteristic values ​​704 for a plurality of characteristics from the chemical structure 702. For example, characteristic values ​​such as LogP, which indicates the lipophilicity of a compound, and the Tanimoto coefficient, which indicates the similarity to other compounds, are calculated.

[0057] The dot product calculation 706 calculates the dot product of the characteristic values ​​704 and the weights 705 as the reward r(t). For example, if the characteristic values ​​704 for three characteristics are r1, r2, and r3, and the weights 705 are w1, w2, and w3, the reward r(t) is calculated using the following formula:

[0058] r(t)=w1×r1+w2×r2+w3×r3 FIG. 8 is a flowchart showing the distributed reinforcement learning execution process of this embodiment.

[0059] This distributed reinforcement learning execution process is executed by the distributed reinforcement learning execution unit 110 of the server 101 shown in FIG.

[0060] 8, first, initialization is performed (step S801). Specifically, the learning parameters θ506 and Q of the Q network 503 are ＊ The learning parameter θ of the network 504 ＊ 507 is initialized with a random value. Furthermore, a predetermined number of actors 601 are generated. Furthermore, the compound data acquired from the compound data storage unit 111 is encoded using the generative model learned in the generative model learning process, and an initial state s (t=0) is generated.

[0061] Next, the weights generated in the weight generation process are set for each actor (step S802).

[0062] Next, at time step t, the learner is executed (step S803), the actor is executed (step S804), and the experience data (state s(t), state s(t+1), action a(t), reward r(t)) is saved in the replay memory (step S805).

[0063] Next, it is determined whether to execute learning (step S806). Specifically, if the time step t has not reached the predetermined maximum number of time steps, the process returns to step S803, and the time step t is updated. If the time step t has reached the predetermined maximum number of time steps, learning is executed (step S807). Specifically, based on the priority assigned by prioritized experience playback, the learner 501 acquires a predetermined number of experience data (state s(t), state s(t+1), action a(t), reward r(t)) from the replay memory 502, and executes TD learning in the learning parameter update unit 505. TD learning is performed by setting a predetermined discount rate γ, the action value function of the Q network 503 for state s(t) as Q(s(t)), and the action value function for state s(t+1) as Q(s(t)). ＊ The maximum value of the action value function of the network 504 is maxQ ＊ When (s(t+1)) is used, the TD error is r(t) + γ × maxQ ＊ (s(t+1))-Q(s(t)) is calculated, and the learning parameter θ506 of the Q network 503 is updated using the gradient method so that the TD error becomes smaller.

[0064] Next, it is determined whether to end the optimization of the compound (step S808). If the predetermined number of learning times has not been reached, the process returns to step S803 and the time step t is initialized. If the predetermined number of learning times has been reached, the learning results are output (step S809) and this process ends.

[0065] FIG. 9 is a table showing an example of a learning result acquired in the distributed reinforcement learning execution process of this embodiment.

[0066] The learning result includes data of actor ID 301, weight of characteristic 1 302, weight of characteristic 2 303, weight of characteristic 3 304, characteristic value 901 of characteristic 1, characteristic value 902 of characteristic 2, characteristic value 903 of characteristic 3, and reward 904.

[0067] The actor ID 301 is an identifier that uniquely identifies an actor.

[0068] A weight 302 for characteristic 1, a weight 303 for characteristic 2, and a weight 304 for characteristic 3 are weights assigned to each actor. Here, an example is shown in which weights for three characteristics are assigned to each actor.

[0069] A characteristic value 901 of characteristic 1, a characteristic value 902 of characteristic 2, and a characteristic value 903 of characteristic 3 are characteristic values ​​calculated by characteristic value calculation for each actor.

[0070] The reward 904 is calculated by calculating the inner product of the characteristic value and the weight for each actor.

[0071] FIG. 10 is a diagram showing an example of an input / output screen related to the weight generation process of this embodiment.

[0072] The input / output screen shown in this figure is composed of an analysis area 1001 and a display area 1002 .

[0073] The analysis area 1001 is composed of an analysis condition area 1003 , an analysis execution button 1004 , and an analysis result area 1005 .

[0074] When the user inputs information into the analysis condition area 1003 and clicks the analysis execution button 1004, the weight generation process is executed and the optimum criteria and weights are output in the analysis result area 1005.

[0075] The display area 1002 is composed of a display condition area 1006 , a display execution button 1007 , and a display result area 1008 .

[0076] When the user inputs information into the display condition area 1006 and clicks the display execution button 1007, the analysis results are displayed graphically in the display result area 1008. Specifically, when the horizontal and vertical axes of the weight distribution to be displayed are input in the display condition area 1006 and the display execution button 1007 is clicked, the weight distribution is output in the display result area 1008. This makes it easy to check how evenly and widely the weights have been generated.

[0077] FIG. 11 is a diagram showing an example of an input / output screen related to the distributed reinforcement learning execution process of this embodiment.

[0078] The input / output screen shown in this figure is composed of an analysis area 1101 and a display area 1102 .

[0079] The analysis area 1101 is composed of an analysis condition area 1103 , an analysis execution button 1104 , and an analysis result area 1105 .

[0080] When the user inputs information in the analysis condition area 1103 and clicks the analysis execution button 1104, the distributed reinforcement learning execution process is executed, and the learning results of the distributed reinforcement learning are output in the analysis result area 1105.

[0081] The display area 1102 is composed of a display condition area 1106 , a display execution button 1107 , and a display result area 1108 .

[0082] When the user inputs information into the display condition area 1106 and clicks the display execution button 1107, the analysis results are displayed graphically in the display result area 1108. Specifically, by inputting the horizontal and vertical axes of the distribution of characteristic values ​​to be displayed and a predetermined actor ID in the display condition area 1106 and clicking the display execution button 1107, the distribution of characteristic values ​​and the compound optimized for the predetermined actor are output in the display result area 1108. This makes it easy to check what characteristics the compound that has been generated has.

[0083] As described above, in this embodiment, by generating weights for multiple characteristics and performing distributed reinforcement learning, each actor can search the feature space of compounds over a wide range without bias for each characteristic.

[0084] Second Embodiment In the second embodiment, in addition to optimizing compounds based on multiple characteristics by distributed reinforcement learning as described in the first embodiment, weights for each characteristic are generated from the learning results of distributed reinforcement learning. Specifically, weight candidates that are expected to have high rewards are extracted, and weights are generated from the extracted weight candidates so as to maximize the D-optimality criterion. By re-executing distributed reinforcement learning using these weights, each actor can search the feature space of compounds over a wide range without bias for each characteristic and within a range that is expected to have high rewards.

[0085] 12 is a block diagram showing the hardware configuration of an information processing system according to the second embodiment. The same components as those in the first embodiment are given the same reference numerals, and the description thereof will be omitted.

[0086] 1 , storage device 107 stores a program for implementing update weight generation section 1201. In other words, the program for implementing update weight generation section 1201 is executed by calculation device 105. That is, it can be said that update weight generation section 1201 is built into calculation device 105 as hardware.

[0087] The update weight generation unit 1201 generates weights for each characteristic from the learning results of distributed reinforcement learning by executing a predetermined program (see FIG. 13).

[0088] The weight generation process, generative model learning process, and distributed reinforcement learning execution process of the second embodiment are the same as those of the first embodiment described above, and therefore will not be described again.

[0089] In the weight generation process of the second embodiment, a weight is generated for each characteristic.

[0090] In the generative model learning process of this embodiment, a generative model is learned from compound data.

[0091] In the distributed reinforcement learning execution process of this embodiment, distributed reinforcement learning is executed using the weights of each characteristic and the generative model.

[0092] FIG. 13 is a flowchart showing the update weight generation process of this embodiment.

[0093] This update weight generation process is executed by update weight generation unit 1201 of server 101 shown in FIG.

[0094] 13 , first, weight candidates and the learning results of the distributed reinforcement learning are obtained (step S1301). The weight candidates are obtained from the weight generation unit 108. The learning results of the distributed reinforcement learning are obtained from the distributed reinforcement learning execution unit 110.

[0095] Next, a regression model is trained by machine learning (step S1302). Specifically, a regression model such as linear regression is trained using the weights of each characteristic in the learning result of distributed reinforcement learning as explanatory variables and the reward as a target variable.

[0096] Next, the rewards for the weight candidates are estimated using a regression model (step S1303), and weight candidates for which the estimated rewards fall within a predetermined percentile range are extracted (step S1304). This makes it possible to extract weight candidates that are expected to have high rewards from the original weight candidates.

[0097] Next, the weight acquisition range is set (step S1305). Specifically, the difference between the maximum and minimum values ​​of the weight candidates for each characteristic is calculated, and a value is obtained by multiplying this difference by a predetermined ratio. For example, if the maximum value of the weight candidates for a certain characteristic is Wmax, the minimum value is Wmin, and the predetermined ratio is r, the acquisition range for that characteristic is calculated as (Wmax - Wmin) x r. This makes it possible to determine, for each characteristic, how widely the extracted weights are distributed compared to the weight candidates in step S1309, which will be described later.

[0098] Next, weights are randomly extracted from the weight candidates extracted in step S1304 (step S1306), and the D-optimality criterion is calculated (step S1307). Specifically, when a matrix X is defined with the rows of the weights randomly extracted so that the number is the same as the number of actors, and the columns of the weight values ​​for multiple characteristics, the D-optimality criterion is calculated using the determinant of the product of the transposed matrix of X and X.

[0099] Next, it is determined whether the calculated D-optimal criterion is greater than the maximum value up to the previous time (step S1308). If this condition is not met, the process returns to step S1306. On the other hand, if this condition is met, it is determined whether the extracted weights are within the obtainable range (step S1309). Specifically, for the extracted weights, the obtainable range of the weights is calculated for each characteristic using a method similar to that of step S1305, and it is determined whether this obtainable range is greater for all characteristics than the obtainable range set in step S1305. If this condition is not met, the process returns to step S1306. On the other hand, if this condition is met, the extracted weights are obtained as updated weights (step S1310), and this process ends. As a result, the updated weights can be set in the distributed reinforcement learning execution process, and distributed reinforcement learning can be re-executed.

[0100] FIG. 14 is a diagram showing an example of an input / output screen related to the update weight generation process of this embodiment.

[0101] The input / output screen shown in this figure is composed of an analysis area 1401 and a display area 1402 .

[0102] The analysis area 1401 is composed of an analysis condition area 1403 , an analysis execution button 1404 , and an analysis result area 1405 .

[0103] When the user inputs information into the analysis condition area 1403 and clicks the analysis execution button 1404, the update weight generation process is executed, and the estimated accuracy of the regression model, the optimal criteria, and the weights are output in the analysis result area 1405. Here, the information in the analysis condition area 1403 is that the regression model is linear regression, the accuracy index is root mean square error (RMSE), and the percentile value is the 75th percentile.

[0104] The display area 1402 is composed of a display condition area 1406 , a display execution button 1407 , and a display result area 1408 .

[0105] When the user inputs information into the display condition area 1406 and clicks the display execution button 1407, the analysis results are displayed graphically in the display result area 1408. Specifically, by inputting the horizontal and vertical axes of the weight distribution to be displayed in the display condition area 1406 and clicking the display execution button 1407, the weight distribution is output in the display result area 1408. This makes it easy to check how evenly and widely the weights have been generated.

[0106] As described above, in this embodiment, by generating weights for each characteristic from the learning results of distributed reinforcement learning, each actor can search the feature space of compounds for each characteristic over a wide range without bias and within a range where high rewards are expected.

[0107] In the first and second embodiments, the weights are generated based on the D-optimality criterion in order to set the weights so as to minimize the number of experiments, but the weights may be generated based on other optimality criteria such as the A-optimality criterion, which allows for flexible experimental planning according to predetermined experimental conditions.

[0108] In the first and second embodiments, the reward r(t) is calculated using only the extrinsic reward obtained from the environment for a plurality of characteristics, but it may also be calculated by adding intrinsic rewards obtained from past experiences, such as NGU (Never Give Up) and Agent 57. This makes it possible to optimize a compound even when it is difficult to obtain rewards from the environment in the short term.

[0109] Preferred embodiments of the present disclosure will be described below.

[0110] In addition to the weight generation unit, generative model learning unit, and distributed reinforcement learning execution unit, the information processing system preferably further includes an update weight generation unit that generates weights for multiple characteristics from the learning results of the distributed reinforcement learning.

[0111] The weight generation unit and the updated weight generation unit generate candidates for weights for a plurality of characteristics, calculate an optimum criterion using weights extracted from the candidates for weights for a plurality of characteristics, and generate weights based on the optimum criterion.

[0112] Here, the weight generation unit and the update weight generation unit are described separately as follows.

[0113] The weight generating unit generates candidate weights for a plurality of characteristics, calculates an optimum criterion using weights extracted from the candidate weights for the plurality of characteristics, and generates weights based on the optimum criterion.

[0114] The update weight generation unit generates candidate weights for a plurality of characteristics, calculates an optimality criterion using weights extracted from the candidate weights for the plurality of characteristics, and generates weights (update weights) based on the optimality criterion.The distributed reinforcement learning execution unit then executes distributed reinforcement learning again using the generated update weights.

[0115] The distributed reinforcement learning execution unit calculates, as a reward, an inner product of weights for a plurality of characteristics and characteristic values ​​for a plurality of characteristics.

[0116] The update weight generation unit uses machine learning to learn a regression model from the learning results of distributed reinforcement learning, estimates rewards for candidate weights for multiple characteristics using the regression model, and extracts candidate weights whose rewards fall within a predetermined percentile value range.

[0117] The weight generating unit and the updated weight generating unit set an acquisition range of weights for a plurality of characteristics, and generate weights until the weights are within the acquisition range.

[0118] 101: Server, 102: Database, 103: Input device, 104: Output device, 105: Computing device, 106: Memory, 107: Storage device, 108: Weight generation unit, 109: Generative model learning unit, 110: Distributed reinforcement learning execution unit, 111: Compound data storage unit, 1201: Update weight generation unit.

Claims

1. A system for optimizing a compound based on multiple characteristics, comprising: a weight generation unit that generates weights for the multiple characteristics; a generative model training unit that trains a generative model from compound data; and a distributed reinforcement learning execution unit that executes distributed reinforcement learning using the weights and the generative model.

2. The information processing system according to claim 1, further comprising an update weight generation unit that generates weights for the plurality of characteristics from the learning results of the distributed reinforcement learning.

3. The information processing system according to claim 2, wherein the weight generation unit and the updated weight generation unit generate candidate weights for the plurality of characteristics, calculate an optimum criterion using weights extracted from the candidate weights for the plurality of characteristics, and generate the weights based on the optimum criterion.

4. The information processing system according to claim 2, wherein the distributed reinforcement learning execution unit calculates, as a reward, an inner product of the weights for the plurality of characteristics and characteristic values ​​for the plurality of characteristics.

5. The information processing system of claim 2, wherein the update weight generation unit learns a regression model by machine learning from the learning results of the distributed reinforcement learning, estimates rewards for candidate weights for the plurality of characteristics using the regression model, and extracts candidate weights for which the rewards are within a predetermined percentile value range.

6. The information processing system according to claim 2, wherein the weight generation unit and the updated weight generation unit set an acquisition range of weights for the plurality of characteristics and generate weights until the weights are within the acquisition range.

Images