A one-step reinforcement learning method for constructing an adiabatic environment to simulate crystal structures.

Abstract

This invention provides a one-step reinforcement learning-based adiabatic environment construction method for simulating crystal structures, belonging to the fields of reinforcement learning and materials synthesis technology. This method searches for possible phase transition paths that a crystal may undergo during experiments by approximating the relationship between the phase transition process and the energy supply. Specifically, the method first obtains all possible structures of the crystal material undergoing phase transitions to construct the simulation environment. Then, it processes the specific information contained in all possible structures, performs structural transformations, and formulates transformation rules to simulate the phase transition process of the crystal material. Therefore, this method can effectively assist experiments, reduce the number of experiments, lower costs, and has high economic value for the field of materials synthesis.

Description

Technical Field

[0001] This invention belongs to the field of reinforcement learning and materials synthesis technology, specifically providing a one-step reinforcement learning adiabatic environment construction method that simulates crystal structure. Background Technology

[0002] Crystals with different structures possess different properties. In the process of material synthesis, we need to prepare materials with crystal structures that have a certain optimal performance. The method is to transform an existing crystal structure into the target crystal structure through a phase transition process. This process requires verification through multiple experiments, but the cost of these experiments is too high.

[0003] An adiabatic environment generally refers to an ideal simulated environment, that is, one that does not consider energy dissipation from interaction with the external environment. A one-step preparation process refers to the process of injecting all the raw materials at once, slowly heating and maintaining a constant temperature, and finally cooling at a suitable rate; the reaction process is difficult to control. Summary of the Invention

[0004] This invention is made to solve the above-mentioned problems, and aims to provide a one-step reinforcement learning adiabatic environment construction method that simulates crystal structure.

[0005] In a first aspect, the present invention provides a one-step reinforcement learning adiabatic environment construction method for simulating crystal structure, characterized by: acquiring all possible structures of a crystal material undergoing a phase transition process; processing the specific information contained in all possible structures and then performing structural transformation and formulating transformation rules.

[0006] The construction method provided by this invention also has the following features: the transformation rules include a first transformation rule and a second transformation rule. The first transformation rule includes setting a transformation radius. The second transformation rule, based on the first transformation rule, includes randomly selecting a transformable structural point and determining the available energy value of the transformable structural point. The formula for calculating the available energy value of the transformable structural point is as follows: available energy value = energy supply value of the simulated environment + difference between the initial energy values ​​of the structure before and after the transformation.

[0007] The construction method provided by this invention also has the following feature: wherein the energy supply value E of the simulated environment... i The range is: 0≤E i <E max E max It represents the maximum difference in initial energy values ​​across all possible structures.

[0008] The construction method provided by this invention also has the following feature: wherein the initial energy supply value E0 of the simulated environment is 0.

[0009] The construction method provided by this invention also has the following characteristics: the process of preparing crystal materials in a one-step manner under simulated adiabatic conditions includes a first simulation stage, a second simulation stage, and a third simulation stage. In the first simulation stage, the energy supply value E1 of the simulated environment ranges from E0 to E1. max .

[0010] The construction method provided by this invention also has the following feature: wherein, in the second simulation stage, the energy supply value of the simulation environment E2 = E max .

[0011] The construction method provided by this invention also has the following feature: wherein, in the third simulation stage, the energy supply value E3 of the simulation environment is < E max In the third simulation phase, the energy supply value of the simulated environment decays in a linear manner.

[0012] Secondly, the present invention provides a reinforcement learning adiabatic environment construction device for simulating crystal structure in a one-step method, characterized by comprising: a crystal all-possible structure acquisition unit, used to acquire all possible structures of the crystal material undergoing a phase transition process; and a transformation rule formulation unit, used to process the specific information contained in all possible structures, perform structural transformation, and formulate transformation rules.

[0013] Thirdly, the present invention provides an interaction method for a reinforcement learning environment, characterized by: mapping all possible structures corresponding to the reinforcement learning environment into state representations and then interacting with a reinforcement learning agent; or encoding all possible structures corresponding to the reinforcement learning environment into quantum states and mapping them into state representations and then interacting with a reinforcement learning agent, wherein the reinforcement learning environment is constructed by the one-step reinforcement learning adiabatic environment construction method of simulating crystal structures of any of the above-mentioned methods.

[0014] The interactive method provided by this invention also has the following features: the quantum state encoding includes: normalizing the high-dimensional data of each possible structure to obtain a normalized vector; processing the normalized vector to obtain the quantum state right vector; conjugating and transposing the quantum state right vector to obtain the corresponding quantum state left vector; and taking the outer product of the quantum state right vector and the quantum state left vector to obtain the structure density matrix obtained after quantum state encoding. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating the overall process of building and executing a one-step reinforcement learning adiabatic environment for simulating crystal structures, according to an embodiment of the present invention.

[0016] Figure 2 This is a first structural schematic diagram of a visualized two-dimensional plane according to an embodiment of the present invention;

[0017] Figure 3This is a schematic diagram of the second structure of a visualized two-dimensional plane according to an embodiment of the present invention;

[0018] Figure 4 This is a schematic diagram of the third structure of a visualized two-dimensional plane according to an embodiment of the present invention;

[0019] Figure 5 This is a line graph illustrating the energy supply value of the simulated environment in a one-step heating process according to an embodiment of the present invention.

[0020] Figure 6 This is a schematic diagram of the quantum encoding process in an embodiment of the present invention.

[0021] The role and effect of invention

[0022] The reinforcement learning adiabatic environment construction method for simulating a one-step crystal structure according to the present invention simulates the process of preparing crystal materials using a one-step method under adiabatic conditions. By approximately establishing the relationship between the phase transition process of the crystal structure and the energy supply, the method searches for the possible phase transition paths that the crystal may undergo during the experiment. Specifically, the method first obtains all possible structures of the crystal material undergoing phase transition to construct a simulation environment. Then, it processes the specific information contained in all possible structure information, performs structural transformation, and formulates transformation rules to simulate the phase transition process of the crystal material. Therefore, this method can effectively assist in the experiment, reduce the number of experiments and lower costs, and has high economic value for the discipline of materials synthesis.

[0023] The reinforcement learning adiabatic environment construction device for simulating crystal structure in one step according to the present invention utilizes computer simulation of the phase transition process of crystal structure to obtain the intermediate structure of the change process. The device includes a crystal all-possible structure acquisition unit for acquiring all possible structures of crystal material undergoing phase transition process; and a transformation rule formulation unit for processing the specific information contained in all possible structures, performing structural transformation and formulating transformation rules. Therefore, the device can simulate the phase transition process of crystal material structure, reduce experimental costs, and has high application prospects in the field of materials science.

[0024] According to the interaction method of the reinforcement learning environment provided by the present invention, since the method interacts and learns with the reinforcement learning agent built by the present invention, and encodes all possible structures constituting the reinforcement learning environment into quantum states, and then maps them into state representations and interacts with the reinforcement learning agent, this interaction method can use quantum circuits to replace classical neural networks as decision agents, thereby improving the decision efficiency of quantum reinforcement algorithms. Detailed Implementation

[0025] To make the technical means, creative features, objectives and effects of this invention easy to understand, the following describes in detail the one-step reinforcement learning adiabatic environment construction method for simulating crystal structures provided by this invention, in conjunction with embodiments and accompanying drawings.

[0026] <Example>

[0027] Figure 1 This is a flowchart illustrating the one-step reinforcement learning adiabatic environment construction method and execution method for simulating crystal structures provided in this embodiment of the invention.

[0028] like Figure 1 As shown, the construction and execution method of the one-step reinforcement learning adiabatic environment (hereinafter referred to as reinforcement learning environment) simulation crystal structure construction method in this embodiment of the invention includes the following steps:

[0029] Step S1: Obtain all possible structures for the crystal to undergo a phase transition, as well as the atomic three-dimensional spatial coordinates and initial energy values ​​corresponding to each structure.

[0030] Step S2 involves reducing the dimensionality of the atomic three-dimensional spatial coordinates of each structure to obtain the corresponding dimensionality-reduced two-dimensional array, which is then visualized.

[0031] Step S3: Based on the dimensionality reduction coordinates of each structure corresponding to the two-dimensional data points in the visualized two-dimensional plane, set the range of the transformation radius, that is, set the first transformation rule.

[0032] Step S4: Based on the set range of transformation radius, randomly select a transformable structural point within the radius region of a certain structural point, determine the available energy value of the transformable structural point, that is, set the second transformation rule.

[0033] Step S5: Determine whether the available energy value of the transformable structure point is positive.

[0034] Step S6: If the available energy value of the transformable structural point is positive, the transformation is successful, and the system switches to the region of the transformable structural point and the transformation radius to continue structural transformation.

[0035] Step S7: If the available energy value of the transformable structure point is negative, then keep the structure unchanged and return to step S4.

[0036] Step S8: Determine whether the successfully transformed transformable structure point is the endpoint structure or whether the maximum number of transformations is met. If it is met, the structure transformation ends; if it is not met, return to step S4 until the transformable structure point is the endpoint structure or the maximum number of transformations is met.

[0037] The reinforcement learning environment built in this embodiment is used to simulate the possible structural phase transition paths of materials in the synthesis experiment. The overall idea of ​​building the environment is to generate all possible material structures with structural information and perform structural transformations within them, while defining the specific rules for the transformations.

[0038] Furthermore, this embodiment uses DFT (Density Functional Theory) calculations to obtain all structural information of a crystal cell, including at least atomic three-dimensional spatial coordinate information and initial energy value information. The three-dimensional spatial coordinate information of each cell structure is processed into a set of high-dimensional data. Then, each set of high-dimensional data is used to calculate a set of two-dimensional data using a manifold learning dimensionality reduction algorithm. The purpose of this process is to find a reasonable low-dimensional feature representation by maintaining a certain invariant feature quantity between the high-dimensional and low-dimensional data, which is beneficial for visualization while reducing computational requirements. This embodiment uses a manifold learning dimensionality reduction algorithm to reduce high-dimensional data to two-dimensional data. In other embodiments, existing dimensionality reduction algorithms can also be used for dimensionality reduction processing. The two-dimensional data points of all structures obtained after dimensionality reduction are saved as an array, with each data point representing a material structure. All data points are visualized on a two-dimensional planar graph. The maximum difference between the initial energy values ​​of all structure points is calculated as a reference value for energy replenishment in the simulated phase transition environment. The energy value replenished in each step of the simulated environment is adjusted according to the heating method used in the actual experimental material synthesis.

[0039] The properties of the simulation environment defined in this embodiment are as follows:

[0040] Furthermore, this embodiment specifies the relationship between points (two-dimensional data points): the closer two points are on the visualized two-dimensional plane, the higher their similarity.

[0041] Furthermore, this embodiment defines the meaning of transformation: transformation refers to the change of a crystal from one structure to another, corresponding to the phase transition process of materials in reality. This embodiment sets two transformation rules to realize the structural transformation, namely the first transformation rule and the second transformation rule.

[0042] See Figure 2 The first transformation rule is to set the transformation radius. In this embodiment, the range of the transformation radius is set based on the maximum difference of the dimensionality reduction coordinates corresponding to all possible structures. Figure 2 Let X1 and Xn be the minimum and maximum values ​​of the dimension-reduced coordinates among all possible structures, respectively. The range of the transformation radius is set based on the maximum difference X between them (not shown in the figure). Figures 2-4 Mid-radius The specific transformation radius can be set randomly according to the specific situation. This embodiment includes other transformation radii set based on the maximum difference X.

[0043] Furthermore, this embodiment specifies the variable range: the radius value r of the variable range is determined based on the size of the visualized two-dimensional planar diagram, the density of points on the diagram, and the material synthesis heating method. During the structural transformation process, the radius r scales up or down with the amount of heating energy replenishment. The scaling method of the r value is adjusted through multiple simulations according to the accuracy requirements.

[0044] Furthermore, in this embodiment, based on the first transformation rule, a second transformation rule is set to determine whether the structural transformation is successful. The second transformation rule set in this embodiment is to randomly select a transformable structural point and determine the available energy value of the transformable structural point. If the available energy value of the transformable structural point is positive, the transformation is successful; if the available energy value of the transformable structural point is negative, it remains unchanged, and other transformable structural points are searched within the set transformation radius.

[0045] Furthermore, the expression for the available energy value of any transformable structure point is as follows:

[0046] Available energy value = replenishment value of simulated environmental energy + difference between the initial value of structural energy before and after transformation.

[0047] In this embodiment, the maximum difference between the initial energy values ​​of all possible structures is used as a reference for replenishing the energy of the simulated environment.

[0048] Specifically, this embodiment sets initial energy values ​​for all possible structural information from the outset, and also sets initial energy values ​​for the simulation environment. See [link to documentation]. Figure 2 Using the first possible structure (initial possible structure 0) as the center and a predefined phase transition radius (r0) as the search radius, an arbitrary variable structure search is performed. If the randomly searched possible structure (the possible structure where 1 is located) is within the search radius r0, and the available energy value of the possible structure where 1 is located is positive, then the structure transformation is successful, and the structure where 1 is located is the first transformable structure point. If the available energy value of the randomly searched possible structure is negative, then the structure remains unchanged, and other transformable structure points are searched within the predefined transformation radius until the available energy value of the first transformable structure point is found to be positive. A reward is then set according to the predefined reward function to make the structure transformation approach or reach the endpoint structure. Figure 2 In the diagram, r0 and r1 are the search radii, and r0 = r1. 1 is the first transformable structure point, and 2 is the second transformable structure point obtained in the same way as 1.

[0049] Furthermore, the energy replenishment value for the simulated environment in this embodiment is shown in [reference]. Figure 5 The diagram shown is a broken line diagram, as follows: Figure 5 As shown, the simulated environmental energy supply value E i The range is 0≤E i <Emax E max The maximum difference between the initial energy values ​​set for all possible structural information. In this embodiment, the initial energy supply value of the simulation environment is E0 = 0. During the first simulation phase 0-T1, the energy supply value of the simulation environment, E1, slowly increases from E0 to E1. max In the second simulation phase T1-T2, the energy supply value E2 of the simulated environment remains constant at E max The energy supply value E3 of the simulated environment remains unchanged. In the third simulation stage T2-T3, the energy supply value E3 of the simulated environment decreases moderately with the increase of the number of steps. The decrease can be linear with the number of steps or it can be decreased in a way that conforms to the specific experiment. The number of steps is determined according to the specific synthetic material.

[0050] See Figure 3 , Figure 3 In this example, the circle containing 0 represents the initial structure, and the circle containing 1 represents the first transformable structure point. In this embodiment, the initial structure 0 is used as the center, and a set transformation radius r0 (r0 is referenced) is used. Figure 2 The search radius is set to r0 for structure searching. The two possible structure points indicated by the dashed arrows in the diagram are within the transformation radius r0. Assuming the available energy values ​​of these two possible structure points are both negative, the initial structure 0 is kept unchanged, and the first transformable structure point is searched using the same transformation radius r0. If the available energy value of structure 1 is positive, then this structure point is the first transformable structure point. Subsequently, following the same method, the second transformable structure point is searched using the first transformable structure point (structure 1) as the center and the set phase transition radius r1 (r1 = r0) as the search radius. Based on the set reward function R, the search proceeds to or near the endpoint structure (not shown). In this embodiment... Figure 2 and Figure 3 A schematic diagram of the phase transition path that can be used as a known starting and ending structure.

[0051] Furthermore, this embodiment sets a reward function based on the phase transition environment. This reward function is used to evaluate the transformation of the crystal structure. A good transformation will give the reinforcement learning agent positive feedback, and a bad transformation will give the reinforcement learning agent negative feedback. Specifically, the quality of the transformation is defined according to the tendency of the crystal material to undergo phase transition in the experiment. In addition, the reinforcement learning algorithm influences how the agent chooses the transformation path through iteration. The transformation path selected by the algorithm that converges after multiple iterations often conforms to the process of phase transition of crystal materials in the real environment. The specific method of setting the reward function varies depending on whether there is a specific known starting structure and ending structure.

[0052] Furthermore, if the starting and ending structures are known, a reward function is set according to a typical reinforcement learning environment, and is divided into a first type of reward function and a second type of reward function. The first type of reward function sets the reward for a successful structure transformation to 1, and the reward for any intermediate transition structures to 0, to facilitate rapid algorithm iteration in finding a reasonable phase transition path. The second type of reward function can use the energy absorption or release value during the phase transition process as a reference value for reward setting. Although this embodiment lists the above two specific reward functions, the specific reward function settings can be set according to the phase transition environment. This embodiment also includes other reward functions in the prior art.

[0053] See Figure 4 If the starting and ending structures are unknown, since each round of structural transformation sets a maximum number of transformations, such as... Figure 4 If possible structure 0 is any randomly selected possible structure that is not the starting structure, then a structure search is performed by transforming the radius r0. Assuming the structure search is performed in a counterclockwise direction, the available energy values ​​of the possible structures pointed to by the dashed arrows in the figure are all negative. If the possible structure where 1 is located has reached the preset maximum number of times (threshold) of this round of structure transformation, then regardless of whether the available energy value of structure 1 is negative, structure 1 is the ending structure, and this round of structure transformation terminates.

[0054] Furthermore, the phase transition environments simulated in this embodiment include absolutely free environments, adiabatic environments, and dissipative environments. The first type, the absolutely free environment, does not consider the case of insufficient phase transition energy, allowing the crystal to freely search for phase transition paths in the structural space, thus imposing the lowest degree of constraint on the transformation rules. The second type simulates an adiabatic environment, which only considers the interaction between the crystal and the energy supply source, constraining the structural transformation based on the total energy of both. The third type is a dissipative environment, which considers not only the relationship between the crystal and the energy supply source but also the energy dissipation generated by both in the environment, thus imposing the highest degree of constraint on the transformation rules.

[0055] On the other hand, this embodiment also provides an interaction method for a reinforcement learning environment, which maps all possible structures corresponding to the reinforcement learning environment built above into state representations, interacts and learns with the reinforcement learning agent, and updates the state representations based on the actions output by the reinforcement agent.

[0056] Furthermore, in this embodiment, each state(s) in quantum reinforcement learning is defined as a structural representation, and all structures constitute the state set of the simulated phase transition system. At a certain moment, a structure is obtained from the constructed reinforcement learning environment, which is mapped to an input state representation and input to the reinforcement learning agent. Based on the action output by the reinforcement learning agent, the crystal structure changes from one state to another, and the algorithm is iteratively updated based on the feedback obtained.

[0057] Furthermore, in this embodiment, high-dimensional data from the established reinforcement learning environment can be input into a dimensionality reduction algorithm for dimensionality reduction before being input into the reinforcement learning algorithm to interact with the reinforcement learning agent. However, due to the high-dimensional complexity of material structure data, the dimensionality reduction algorithm may not be able to fully express the similarity between different structures. In this case, it is necessary to directly perform computational processing on a large amount of high-dimensional data of structures. Direct computation on high-dimensional data requires enormous computing power, while quantum computers can meet the computing power requirements. Therefore, this embodiment also provides a quantum state encoding method for representing structural data.

[0058] Figure 6 This is a schematic diagram of the quantum encoding process according to an embodiment of the present invention.

[0059] See Figure 6 Quantum encoding includes the following steps:

[0060] Step S102: First, normalize the high-dimensional data of the straightened structure with length n to obtain a normalized vector.

[0061] Step S104: Transform the normalized vector into complex form, i.e., the right vector of the quantum state in n*1 dimensions.

[0062] Step S106: Take the conjugate transpose of the n*1 dimensional quantum state right vector to obtain the 1*n dimensional quantum state left vector.

[0063] Step S108: Perform an outer product of the n*1 dimensional quantum state right vector and the 1*n dimensional quantum state left vector to obtain an n*n dimensional quantum state density matrix, thus completing the quantum state encoding.

[0064] The role and effect of the embodiments

[0065] The reinforcement learning adiabatic environment construction method for simulating crystal structure provided in this embodiment searches for possible phase transition paths of crystals during experiments by approximately establishing the relationship between the phase transition process of crystal structure and the energy supply. Specifically, this method first obtains all possible structures of crystal materials undergoing phase transition processes to construct a simulation environment, then processes the specific information contained in all possible structure information to perform structure transformation and formulate transformation rules, thereby simulating the phase transition process of crystal materials. Therefore, this method can effectively assist experiments, reduce the number of experiments and lower costs, and has high economic value for the discipline of materials synthesis.

[0066] The foregoing description and accompanying drawings have provided typical embodiments of specific structures for specific implementations. The above-described invention presents preferred embodiments, but these are not intended to be limiting. Various changes and modifications will undoubtedly be apparent to those skilled in the art after reading the foregoing description. Therefore, the appended claims should be considered to cover all changes and modifications that encompass the true intent and scope of the invention. Any and all equivalent scope and content within the scope of the claims should be considered to still fall within the intent and scope of the invention.

Claims

1. A method for building a reinforcement learning adiabatic environment for simulating a one-step method of crystal structure, characterized in that, The method comprises: acquiring all possible structures of a crystal material during a phase transition process; processing specific information contained in the all possible structures, performing structure transformation, and formulating transformation rules; wherein the transformation rules include a first transformation rule and a second transformation rule, the first transformation rule includes setting a transformation radius; based on the first transformation rule, the second transformation rule includes randomly selecting a transformable structure point and judging an available energy value of the transformable structure point; the calculation formula of the available energy value of the transformable structure point is as follows: Available energy value = Energy supply value of the simulated environment + Difference between the initial structure energy values before and after transformation; wherein the energy supply value E of the simulation environment i ranges from: 0 ≤ E i < E max , E max is the maximum difference in the energy initial values of all possible structures; wherein i is equal to 1, 2, or 3. wherein the initial energy supply value E0 of the simulated environment = 0; wherein the process of preparing the crystal material in the simulated adiabatic environment by the one-step method includes a first simulation stage, a second simulation stage, and a third simulation stage, the energy supply value E1 of the simulated environment in the first simulation stage ranges from: E0≤E1 < E max ; wherein the energy supply value E2 of the simulation environment in the second simulation phase = E max ; wherein the energy supply value E3 of the simulation environment in the third simulation phase is < E max , the attenuation mode of the energy supply value of the simulated environment in the third simulation stage includes linear attenuation.

2. An interaction method of a reinforcement learning environment, characterized in that: all possible structures corresponding to the reinforcement learning environment are mapped into state representations and interacted with a reinforcement learning agent; or all possible structures corresponding to the reinforcement learning environment are quantum state encoded and mapped into state representations and interacted with a reinforcement learning agent, wherein the reinforcement learning environment is built by the method for building a one-step simulated crystal structure method of a reinforcement learning adiabatic environment according to any one of claim 1.

3. The method of claim 2, characterized in that: wherein the quantum state encoding includes: normalizing the high-dimensional data of each possible structure to obtain a normalized vector; processing the normalized vector to obtain a quantum state right vector; conjugate transposing the quantum state right vector to obtain a corresponding quantum state left vector; performing outer product on the quantum state right vector and the quantum state left vector, i.e., obtaining the structure density matrix after the quantum state encoding.

Images