A reinforcement learning method of direct spatial crystal flow generation model

Abstract

The application discloses a reinforcement learning method of a direct space crystal flow generation model, and relates to the technical field of artificial intelligence assisted material design, and comprises the following steps: obtaining a pre-trained direct space crystal flow generation model; initializing strategy parameters according to the direct space crystal flow generation model; generating multi-channel random generation trajectories by using the direct space crystal flow generation model and the strategy parameters, and calculating corresponding joint logarithmic scores; decoding the termination state of each trajectory into a candidate crystal structure, and performing material evaluation to obtain a final thermodynamic evaluation; calculating a trajectory-level advantage of the candidate crystal structure according to the final thermodynamic evaluation; recalculating the joint logarithmic scores of the multi-channel random generation trajectories by using the strategy parameters, combining the trajectory-level advantage to construct a pruning strategy loss, and updating and optimizing the direct space crystal flow generation model; and predicting and outputting a de novo crystal structure by using the optimized direct space crystal flow generation model. The application can improve the accuracy of structure prediction of the model.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence-assisted material design technology, and more specifically to a reinforcement learning method for a direct spatial crystal flow generation model. Background Technology

[0002] Flow matching, rectification, or other continuous flow generation models learn the time-varying velocity field. This involves progressively mapping easily sampled prior distributions to the crystal data distribution. In direct space crystal generation, the joint state can be defined as: ,in, For continuous states of atomic type, In fractional coordinate state, It is a lattice state. The conventional deterministic discrete integral can be written as... The deterministic integral itself does not provide explicit random action probabilities applicable to policy gradients. If it is desired to perform reinforcement learning post-training based on the stability, effectiveness, novelty, diversity, or target function properties of the final generated crystal, the multi-step generation process needs to be constructed as a policy process with random actions and computable transition scores, and behavior policy sampling, behavior score caching, and current policy recalculation should be for the same physical action and the same transition semantics.

[0003] The existing representative method Chemeleon2 introduces reinforcement learning into the crystal latent space denoising diffusion model. Its technical approach is as follows: it uses a variational autoencoder to encode the crystal structure as a continuous latent variable z, and uses the latent space back-diffusion kernel as the strategy. This paper proposes a method that guides generation through Group Relative Policy Optimization (GRPO) and multi-objective material rewards (creativity, stability, diversity), and employs machine learning force fields to reduce evaluation costs. However, the aforementioned existing schemes and general direct space generation techniques still suffer from the following significant drawbacks:

[0004] First, there is a lack of direct spatial multichannel stochastic policy construction schemes. Chemeleon2's policy relies entirely on VAE latent variables and latent space diffusion kernels. Existing technologies do not provide a method to transform this deterministic flow integral into a joint policy with explicit stochastic actions and computable transition scores without introducing a latent space encoder. Furthermore, the consistency problem of direct spatial transition semantics between behavior trajectory sampling, score buffering, and current policy recalculation has not been resolved.

[0005] Second, there is semantic inconsistency in the transition scores at the periodic boundaries of the fractional coordinates. In the direct space, if only the fractional coordinate state is wrapped around (the action is placed back into the fundamental domain), and the difference between the action and the transition center is still calculated using ordinary Euclidean residuals, small physical displacements near the periodic boundaries will be misjudged as large residuals approaching one lattice period, leading to abnormal amplification of Gaussian scores and policy gradients. Chemeleon2's probability is defined in the latent variable space and does not involve the selection of equivalent residuals for the direct space periods. More importantly, existing technologies fail to recognize that the behavior-policy cache score and the current policy recalculation score must strictly reuse the same component-rounded minimum image rule; otherwise, even if the state wraps around correctly, the probability ratio between the old and new policies will still be distorted due to the difference in coordinate representation, failing to truly reflect policy changes.

[0006] Third, it fails to consider the scale differences and interference from invalid atoms in multiple channels. The three channels—atom type, fractional coordinates, and lattice—have different state dimensions, physical dimensions, and effective degrees of freedom. A single diffusion nucleus in latent space cannot adapt to the random perturbation scale of each channel, nor can it solve the problem of changes in cumulative diffusion intensity caused by variations in discrete time steps. Furthermore, if cell-filling atoms (invalid slots) are not excluded from the transfer score and activity dimension statistics, they will dominate the joint score and interfere with the policy gradient direction; existing technologies do not provide an effective mechanism to address this.

[0007] Fourth, the relaxation budget allocation in material evaluation is unreasonable. Existing schemes uniformly perform a fixed number of structural relaxation steps on candidate crystals without dynamically allocating the calculation budget based on the preliminary thermodynamic evaluation results. Applying high-cost deep relaxation to all candidates will significantly reduce the training throughput of reinforcement learning; while uniformly applying shallow relaxation may lead to misjudgments of rewards for candidates close to the stable or metastable thresholds because the structure has not yet fully converged, affecting the reliability of policy updates.

[0008] Therefore, how to construct periodic consistency transfer scores, adapt multi-channel stochastic strategies, and conduct reliable material evaluation at controllable costs, thereby improving the accuracy of model-generated crystal structures, is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0009] In view of the above problems, this invention is proposed to provide a reinforcement learning method for a direct spatial crystal flow generation model that overcomes or at least partially solves these problems. The method constructs the multi-step generation process of the crystal flow model as a randomly generated trajectory with a computable transition score, and updates the model parameters using policy gradients based on the material evaluation results of the generated crystal. This invention is applied to ab initio crystal structure generation, including the joint generation of atom types, fractional coordinates, and lattice parameters, as well as techniques such as random flow transition construction, periodic fractional coordinate processing, behavior trajectory caching, current policy recalculation, multi-objective reward calculation, group relative advantage estimation, reference policy constraints, and structural relaxation evaluation.

[0010] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, embodiments of the present invention provide a reinforcement learning method for a direct spatial crystal flow generation model, comprising the following steps: Step 1: Obtain a pre-trained direct spatial crystal flow generation model; Step 2: Initialize the strategy parameters based on the direct space crystal flow generation model; copy the parameters of the direct space crystal flow generation model to the strategy parameters; Step 3: Generate multi-channel randomly generated trajectories using the direct spatial crystal flow generation model and strategy parameters, and calculate the joint logarithm score for each trajectory; Step 4: Decode the termination state of each trajectory into a candidate crystal structure, and perform material evaluation on the candidate crystal structure to obtain the final thermodynamic evaluation; Step 5: Calculate the trajectory-level advantage of the candidate crystal structure based on the final thermodynamic evaluation; Step 6: Recalculate the joint logarithmic score of the cached trajectory using the current policy in the policy parameters, and construct the pruning policy loss by combining the trajectory-level advantages to update and optimize the direct spatial crystal flow generation model; Step 7: Predict the output ab initio crystal structure using the optimized direct space crystal flow generation model.

[0011] Preferably, the direct spatial crystal flow generation model pre-trained on the crystal structure dataset is represented as follows: ; ; in, Indicates the joint state at step k; For the continuous state of the atom type in step k, Let the periodic fraction coordinates be the coordinates of the k-th step. This represents the lattice state at step k; Indicates the joint state based on k steps and the generation time of step k Output velocity field; This represents the velocity field corresponding to the continuous state channel of the atom type in the k-th step; This represents the velocity field corresponding to the periodic fractional coordinate channel at step k; This represents the velocity field corresponding to the lattice state channel at step k.

[0012] Preferably, the strategy parameters include behavioral strategies. Current strategy and reference strategy Behavioral strategies Used to generate and cache trajectories; current policy The strategy to be updated is initially the same as the behavioral strategy; reference strategy. Used to constrain policy drift, initially the same as the behavioral policy.

[0013] Preferably, the specific process of generating each generation step k of each trajectory in step 3 is as follows: Step 311: Read the current joint state at step k. and generation time Adopting behavioral strategies The velocity fields corresponding to three channels were obtained, including atomic type, fractional coordinates, and lattice. Step 312: Based on the joint state in step k Velocity field and preset discrete time step The conditional transfer center is calculated and represented as: ; in, This represents the conditional transition center of the behavioral policy before adding a random perturbation to channel c; when c=F, This represents the conditional transition center in the fractional coordinate channel before the addition of random perturbation; The states are represented by H, F, and L, which represent atomic type channels, fractional coordinate channels, and lattice channels, respectively. This represents the state component of the joint state at step k on channel c; The velocity field of channel c is represented by the output of the behavioral policy based on the joint state at step k and the generation time.

[0014] Step 313: Set an independent random disturbance standard deviation for each channel, expressed as: ; in, This represents the standard deviation of the random perturbation in channel c; s represents the channel diffusion coefficient; s() represents the time scheduling function; Step 314: Sample independent standard Gaussian noise for each channel, and calculate the random joint state at step k+1 based on the one-step conditional transfer center and the standard deviation of the random perturbation. Obtain the random state component at step k+1 for each channel without periodic folding, expressed as: ; in, This represents the standard Gaussian noise of channel c at step k; Represents the random joint state at step k+1. Random state components on channel c; Step 315: Based on the random joint state of step k+1, perform state updates on the atom type state and lattice state, perform periodic folding and state updates on the fractional coordinates, and obtain the updated joint state of step k+1. The joint state at step k+1 includes the atom-type state component, lattice-state component, and fractional coordinate state component at step k+1; the expression is: ; ; ; in, This represents the atom type state component at step k+1; This represents the random state component of the atom type at step k+1; The lattice state components at step k+1; The lattice random state components at step k+1; Represents the fractional coordinate state component at step k+1; represents the random state component of the fractional coordinates at step k+1; wrap represents the folding operation.

[0015] Preferably, step 3 further includes calculating the joint logarithmic score of the actual action executed by the behavior policy for each generation step k of each trajectory, and associating and caching all necessary information; the specific process is as follows: Step 321: Calculate the residual for each channel based on the state and conditional transition center of step k+1 corresponding to each channel. The expression is: ; ; ; ; ; in, To represent the atomic type residual, Euclidean residuals are used; Lattice residuals are represented using Euclidean residuals; The original fractional coordinate residual between the (k+1)th step fractional coordinate state component and the one-step conditional transition center without periodic equivalence processing is used to determine the periodic equivalence representative element. This represents the smallest image integer translation matrix obtained by rounding the original fractional coordinate residuals to the nearest integer. Represents a set of integers. This represents all atomic slots after filling; This represents the effective residual of the fractional coordinates after periodic equivalence processing; Step 322: Sum the effective atom dimensions of each channel based on the residuals, random perturbation standard deviations, and effective atom masks, and calculate the logarithmic score for each channel, expressed as: ; in, Let represent the logarithmic score of channel c at step k in the i-th trajectory; The actual score residual of channel c at step k in the i-th trajectory is represented as follows: (The residual can be expressed as:) (The remaining text is incomplete and cannot be translated.) , , ; This represents the standard deviation of the random perturbation in channel c; This represents the set of valid scalar dimension indices determined by the valid atomic mask, used to select the first... The scalar dimension of the trajectory participating in the summation in channel c, and the dimension corresponding to the filling atom are not included in this index set; according to Determine the mask length and construct an effective atom mask based on the actual effective atom count of the i-th trajectory. The mask value for effective atomic slots is 1, and the mask value for filled atomic slots is 0. Step 323: Sum the logarithmic scores of each channel according to the joint logarithmic probability form to calculate the joint logarithmic score, expressed as: ; in, Let represent the joint logarithmic score at step k in the i-th trajectory; This represents the logarithmic score of the atom type at the k-th step in the i-th trajectory; The logarithmic score of the fractional coordinates at the k-th step in the i-th trajectory; Let represent the lattice logarithm score at the k-th step in the i-th trajectory; Step 324: For the k-th step of the i-th trajectory, combine the joint logarithmic score corresponding to the behavior policy with the joint state at the k-th step. The next joint state obtained in step 315 Generation time Discrete time step Standard deviation of random disturbances for each channel The effective atomic mask and the minimum image rule with component rounding used in step 321 are associated and cached to obtain cached data. The cached data is used in steps 53 and 54 to recalculate the current policy score under the condition of fixed sampled adjacent state transitions; This represents the k-th step state of the i-th trajectory; This indicates the next state.

[0016] Preferably, the minimum image integer translation matrix is ​​calculated based on the original fractional coordinate residuals using the minimum image rule of rounding down by components. The minimum image integer translation matrix is ​​then used to calculate the effective residuals of the fractional coordinates after periodic equivalence processing. These effective residuals are the periodic equivalent residuals of the fractional coordinates, expressed as: ; ; Here, round represents the rounding operation.

[0017] Preferably, the material evaluation process in step 4 is as follows: Step 41: Determine the crystal structure of each candidate crystal. Does it meet the pre-inspection rules? If it does, then the candidate crystal structure will be... Mark it as passing the pre-inspection and proceed to step 42; if it does not meet the requirements, mark it as failing the evaluation, retain the corresponding trajectory and proceed to step 5. Step 42: Apply conditionally triggered two-stage relaxation to each candidate crystal structure Perform machine learning-based interatomic potential relaxation to obtain the convex hull energy difference; the specific process is as follows: Step 421: Utilize the structural relaxation operator For candidate crystal structures Perform the first depth relaxation to obtain the first depth convex hull energy difference. If no abnormalities occur during the calculation process, the candidate crystal structure is marked as successfully evaluated, and the process proceeds to step 422; otherwise, it is marked as unsuccessfully evaluated, and the process proceeds to step 5. The expression is: ; ; in, This indicates the first deep relaxation structure; This indicates the first level of relaxed budget; This represents the energy difference of the first depth convex hull after the first depth relaxation. This represents an evaluation operator for calculating the energy difference of the first deep convex hull based on the energy, composition, and reference phase data of the first deep relaxation structure. Step 422: If the first depth convex hull has a difference , To preset the second depth trigger threshold, then for candidate crystal structures Performing a second-depth relaxation yields the energy difference of the second-depth convex hull. If no abnormalities occur during the calculation process, the candidate crystal structure is marked as successfully evaluated, and the second depth convex hull energy difference is recorded. If an error occurs during the calculation process, the process is marked as an evaluation failure and proceeds to step 5. If the first depth convex hull energy is poor Then the energy difference of the first depth convex hull will be... Transmit to step 43; The expression for the second deep relaxation is:

[0018] in, This indicates the second level of budget relaxation; This represents the energy difference of the second convex hull after the second depth relaxation; a successful evaluation means that there are no abnormalities in the calculation of structural relaxation and convex hull energy difference, and that the energy difference of the relaxed structure and convex hull are both valid finite values. Step 43: Based on each candidate crystal structure The convex hull energy difference determines the corresponding final thermodynamic evaluation, expressed as: or ; in, The final thermodynamic evaluation is indicated by the following steps: When the energy difference of the first deep convex hull is greater than the second deep trigger threshold, the energy difference of the first deep convex hull is retained as the final thermodynamic evaluation; when it is less than or equal to the threshold, the second deep relaxation is performed on the same candidate crystal structure, and when the second deep relaxation evaluation is successful, the energy difference of the second deep convex hull is used as the final thermodynamic evaluation; when the first deep relaxation or the second deep relaxation after triggering, the structural analysis or the calculation of the convex hull energy difference is abnormal, or the output is a non-finite value, the candidate crystal structure is marked as an evaluation failure, and a preset finite failure reward is given in step 5.

[0019] Preferably, the pre-detection rules include multiple discrimination constraints: structural resolvability; effective number of atoms greater than 0; positive and finite crystal volume; and periodic minimum interatomic spacing greater than a preset threshold. .

[0020] Preferably, the specific process of step 5 is as follows: Step 51: For successfully evaluated candidate crystal structures, a reward component is obtained based on one or more of the following: final thermodynamic evaluation, stability or metastability threshold, compositional effectiveness, creativity, compositional diversity, and structural diversity. A trajectory-level total reward is then constructed; the expression is: ; in, Indicates the reward component index; This represents the j-th reward component of the i-th candidate crystal structure after standardization within the same sampling batch; Indicates the first The preset weight of each reward item; This represents the total trajectory-level reward for the i-th candidate crystal structure. The reward components include one or more of the following: continuous convex hull energy reward, stability threshold reward, metastability threshold reward, compositional effectiveness reward, creativity reward, compositional diversity reward, and structural diversity reward. The continuous convex hull energy reward is obtained by truncating the final convex hull energy difference within preset upper and lower limits and taking its negative value. The stability threshold reward and metastability threshold reward are determined based on the final thermodynamic evaluation. The stability threshold reward is 1 if the final thermodynamic evaluation is not greater than 0 eV / atom, otherwise it is 0. The metastability threshold reward is 0 if the final thermodynamic evaluation is not greater than 0.1 eV / atom. The value is 1 for eV / atom, otherwise 0. The component validity reward is obtained by determining the component validity based on the combination of element oxidation states, charge neutrality, and electronegativity constraints. The creativity reward first determines the binary endpoint based on the novelty of the candidate composition relative to the reduced composition of the training set and its uniqueness within the same sampling batch. In the boundary case, under the same reduced composition, the score is continuously assigned based on the distance between the average minimum distance descriptor of the candidate structure and the descriptors of other candidate structures and the reference structure within the batch. The component diversity reward is obtained based on the contribution of the difference between the element score vector of the candidate composition and the maximum mean of the reference composition set within the same sampling batch. The structural diversity reward is obtained based on the contribution of the difference between the average minimum distance descriptor of the candidate structure and the maximum mean of the reference structure set within the same sampling batch. After each reward component is standardized within the same sampling batch, they are weighted and summed according to preset weights to obtain the trajectory-level total reward. For candidate crystal structures that fail evaluation, their total trajectory-level reward is set to the minimum total trajectory-level reward of the candidate crystal structures that succeeded evaluation within the same sampling batch minus a preset finite interval, thereby avoiding invalid values ​​from entering the gradient and ensuring that failed trajectories usually gain a negative advantage relative to the effective trajectories in the same group. Step 52: At the start of a reinforcement learning iteration, fix the behavior policy parameters from Step 2, and use the behavior policy to generate multiple trajectories for a sampling batch; divide the candidate crystal structures corresponding to the termination states of each trajectory within the sampling batch into the same comparison group, with the comparison group of the i-th candidate crystal structure identified as follows: According to the same comparison group The total trajectory-level reward for each candidate crystal structure is calculated as the trajectory-level advantage, expressed as: ; in, This represents the trajectory-level advantage of the i-th candidate crystal structure; and These represent the mean and standard deviation of the total trajectory-level reward within the same comparison group, respectively. This represents a preset positive number used for numerical stability. ; This represents the correspondence between the total trajectory-level reward and the comparison group, where Let represent the total trajectory-level reward for the m-th candidate crystal structure. Indicates the identifier of the comparison group to which the m-th candidate crystal structure belongs; in the dominance calculation formula, select the one that satisfies... =g i The total trajectory-level rewards are grouped into a set of total trajectory-level rewards for the same comparison group. This set is used to calculate the mean and standard deviation of the total trajectory-level rewards within the same comparison group; the total rewards for the same trajectory The policy gradient is then used for the selected generation step of the trajectory.

[0021] Preferably, a reinforcement learning iteration includes a trajectory sampling phase and a parameter optimization phase; in the trajectory sampling phase, a batch of randomly generated trajectories is generated and cached using a fixed behavior policy; in the parameter optimization phase, the cached trajectories themselves remain unchanged, and only the current policy parameters are updated; each batch of trajectories is used only for one parameter update; step 6 is the parameter optimization phase, and the specific process is as follows: Step 61: Extract adjacent state transition pairs from the cached data of multi-channel randomly generated trajectories. Generation time Discrete time step and the joint logarithmic score corresponding to the caching behavior strategy. ; Joint logarithmic score using cache As a baseline, and using the current strategy For the k-th step joint state and corresponding generation time Perform forward computation to obtain the current velocity field of each channel, and then calculate based on the current velocity field and the discrete time step. The current conditional transfer center is calculated as follows: ; in, Indicates the current conditional transfer center of channel c; This represents the current velocity field of channel c; This represents the k-th joint state of channel c in the i-th trajectory; where the standard deviation of random perturbation of each channel, the effective atomic mask, and the minimum image rule rounded by components are used when recalculating the score in step 62; Step 62: Without resampling the state transition results, use the cached joint state from step k+1. As the scoring object, based on the standard deviation of random perturbation of each channel in the cached data, the effective atomic mask, and the minimum image rule rounded by components, the current logarithmic score of each of the three channels is recalculated, and then summed to obtain the current joint logarithmic score. ; indicates as: ; in, Indicates the current logarithmic score of the atom type; Represents the current logarithmic score on the fractional coordinate; This represents the current logarithmic score of the lattice. For the fractional coordinate channel, the original residual is recalculated based on the current conditional transfer center, and the current fractional coordinate effective residual is determined according to the same component-based minimum image rule. The logarithmic score of the current fractional coordinate is calculated using the current fractional coordinate effective residual and the standard deviation of the random disturbance. The expression for the current fractional coordinate effective residual is: ; in, Indicates the current conditional transition center of the fractional coordinates; Represents the fractional coordinate state component at step k+1; ensures that the behavior policy score and the current policy recalculation use the same periodic semantics for the same physical action; Step 63: Score based on joint logarithm and current joint logarithmic score The difference between them constructs a stepwise importance ratio. The expression is: ; Step 64: Construct the pruning strategy loss based on the stepwise importance ratio The pruning strategy loss serves as the strategy optimization loss for a single parameter update of this batch of trajectories. The expression is: ; in, This indicates the preset clipping width that limits a single policy change; This represents the expected operator obtained by averaging the index pairs (i,k) of the trajectories and generation steps involved in this parameter optimization. This indicates the trajectory-level advantage of the candidate crystal structure corresponding to the i-th trajectory; At the start of this round of parameter updates, the current policy and the behavior policy are the same, therefore At this update starting point, the gradient of the pruning target is equivalent to the gradient of the following fractional function: ; in, This represents the gradient of the policy gradient loss with respect to the current policy parameters; This indicates calculating the gradient with respect to the current policy parameters; Step 65: Based on the joint state of the cache, generation time, discrete time step, and standard deviation of random perturbation for each channel, use the reference strategy. Forward computation is performed on the same cache joint state and generation time to obtain the reference velocity field and reference conditional transition center for each channel. The reference conditional transition centers are then mapped one-to-one with the current conditional transition centers according to the same trajectory, generation step, and channel. The stepwise reference constraints of the current policy relative to the reference policy are calculated. The current policy and the reference policy use the same random perturbation variance in the same channel. The stepwise reference constraints are expressed as the difference between the mean of the reference conditional transition center and the current conditional transition center. ; in, The stepwise reference constraint of channel c at step k of the i-th trajectory is represented; Indicates the current conditional transfer center of channel c; Indicates the reference condition transfer center for channel c; This represents the standard deviation of the random perturbation in channel c; Step 66: Averaging the stepwise reference constraints for each trajectory, generation step, and channel yields the reference constraint term. The strategy optimization loss is combined with the reference constraint term to form the total loss. , represented as: ; Step 67: Compare the reference constraint with the preset circuit breaker threshold; if it exceeds the preset circuit breaker threshold, skip this parameter update; if it does not exceed the preset circuit breaker threshold, perform backpropagation on the total loss, perform global gradient pruning on the gradient of the current policy parameters, and use the optimizer to update the current policy parameters once. Step 68: After the parameter update is completed, the reference policy remains fixed within the preset training interval; when the preset refresh interval is reached, the updated current policy parameters are copied to the reference policy; then the updated current policy is used as the behavior policy for the next round, and the process returns to step 3 to regenerate the trajectory of a sampling batch.

[0022] Preferably, the updated direct spatial crystal flow generation model is used to perform multi-step integration starting from the atom type, fractional coordinates, and lattice prior states, and the atom type, fractional coordinates, and lattice parameters of the candidate crystal are jointly generated to obtain the ab initio crystal structure optimized by reinforcement learning.

[0023] Preferably, the pre-inspection rules may also include lattice parameter ranges, lattice density ranges, or lattice composition constraints.

[0024] Preferably, the multi-objective reward, time step selection method, reference policy refresh cycle, two-level relaxation depth, and trigger threshold can all be set according to the dataset, pre-trained streaming model, and evaluation budget.

[0025] As can be seen from the above technical solution, compared with the prior art, this invention discloses a reinforcement learning method for a direct spatial crystal flow generation model. In the trajectory sampling stage, periodic folding is performed on the next state of the fractional coordinates after adding drift and noise, to place the actual action back into the fixed basic domain. In the behavior policy scoring stage and the current policy recalculation stage, the residuals between the actions and their respective policy transition centers are mapped using the same periodic equivalent representation, to define the new and old transition scores. This invention constructs the deterministic generation process of the pre-trained crystal flow model as a multi-channel stochastic policy with computable transition scores, using the material evaluation results of the generated crystal to form trajectory-level rewards, and updating the model parameters through group relative policy gradients. In the reinforcement learning process of the direct spatial flow generation model of periodic crystals, a stochastic transition score consistent with the periodic equivalence relation of the fractional coordinates is constructed, ensuring that different integer periodic images of the same physical action receive consistent evaluation, and guaranteeing that the behavior policy trajectory cache and the current policy recalculation use the same action and scoring semantics, thereby avoiding the pollution of policy gradients by pseudo-large residuals at periodic boundaries. Specific beneficial effects include: (1) Construct a differentiable, cacheable and recalcible multi-channel stochastic strategy for the originally deterministic direct space crystal flow integral; Step 3 performs local Gaussian transfer scoring on the small physical displacements across the period boundary according to the minimum image residual, so as to avoid them being misjudged as large displacements close to one lattice period; The behavior strategy scoring cache and the current strategy recalculation reuse the same component-rounded minimum image rule, so that the difference between the old and new scores mainly reflects the change of strategy parameters rather than the difference in period coordinate representation, thereby making the probability ratio and strategy gradient obtained by recalculation consistent with the period geometry of the crystal fraction coordinate; (2) Match the random perturbation scale with the discrete time step and eliminate the influence of invalid filling atoms; the random perturbation scale used in the process of generating multi-channel random trajectory includes the square root factor of the discrete time step, so that the cumulative diffusion intensity does not change arbitrarily due to the gradual fixed noise addition when the number of integration steps is changed; the diffusion scale can be set for the atom type, fractional coordinates and lattice channels respectively, and the interference of filling atoms on the transfer score and policy gradient can be eliminated by effective atom mask; (3) Perform parameter updates on each batch of cached trajectories once by using the progressive importance ratio and pruning strategy loss, so that the strategy update directly corresponds to the state transition of the sampled state in this batch; (4) Policy drift is limited by multi-channel reference policy transfer constraints, reference constraint circuit breaking, reference policy periodic refresh and global gradient pruning; (5) By using conditionally triggered two-level structure relaxation, the reliability of evaluation near the threshold and the computational cost of training are both taken into account. (6) Without migrating the direct space crystal flow model to latent space, a joint generation strategy can be adopted to update the atom type, fractional coordinates and lattice using non-differentiable material feedback such as crystal relaxation, convex hull energy difference, structure matching and database novelty. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0027] Figure 1 This is a flowchart of a reinforcement learning method for a direct spatial crystal flow generation model provided in an embodiment of the present invention; Figure 2 This is a schematic diagram comparing the Euclidean residual and the periodic equivalent residual when the fractional coordinates cross the period boundary in an embodiment of the present invention. Figure 3 This is a schematic diagram of the random transfer of atom type, fractional coordinates, and lattice three channels provided in the embodiments of the present invention; Figure 4 This is a schematic diagram of two-stage structural relaxation and material reward calculation provided in an embodiment of the present invention. Detailed Implementation

[0028] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] This invention discloses a reinforcement learning method for a direct spatial crystal flow generation model, such as... Figure 1 As shown, it includes the following steps: S1: Obtain a pre-trained direct spatial crystal flow generation model; S2: Initialize strategy parameters based on the direct space crystal flow generation model; S3: Generate multi-channel randomly generated trajectories using the direct spatial crystal flow generation model and strategy parameters, and calculate the joint logarithmic score for each trajectory; S4: Decode the termination state of each trajectory into a candidate crystal structure, and perform material evaluation on the candidate crystal structure to obtain the final thermodynamic evaluation; S5: Calculation of trajectory-level advantages for candidate crystal structures based on the final thermodynamic evaluation; S6: Recalculate the joint logarithmic score of the cached trajectory using the policy parameters, and construct the pruning policy loss by combining the trajectory-level advantages to update and optimize the direct spatial crystal flow generation model; S7: Predict the output ab initio crystal structure using the optimized direct space crystal flow generation model.

[0030] A crystal structure can be represented as , Indicates the type of atom, Represents atomic fraction coordinates. This represents the lattice matrix, lattice parameters, or their continuous representation. This represents the effective number of atoms in the unit cell. Due to periodic boundary conditions, for any integer translation matrix... All Therefore, multiple fractional coordinates with different values ​​can represent the same physical location. In direct space crystal generation, atomic type, fractional coordinates and lattice are coupled with each other. In particular, the fractional coordinates are located in toroidal space, and their state representation, action representation and transition score should satisfy integer periodic translation invariance.

[0031] Furthermore, the direct spatial crystal flow generation model pre-trained on the crystal structure dataset is represented as follows: ; ; in, Indicates the joint state at step k; For the continuous state of the atom type in step k, The periodic fractional coordinate state at step k. This represents the lattice state at step k; This indicates that based on the joint state at step k... and the generation time of step k Output velocity field; This represents the velocity field corresponding to the atom type channel at step k; This represents the velocity field corresponding to the fractional coordinate channel at step k; This represents the velocity field corresponding to the lattice channel at step k.

[0032] Furthermore, the strategy parameters include behavioral strategies. Current strategy and reference strategy Behavioral strategies Used to generate and cache trajectories; current policy The strategy to be updated is initially the same as the behavioral strategy; reference strategy. Used to constrain policy drift, initially the same as the behavioral policy.

[0033] Furthermore, the specific process of generating each generation step k of each trajectory in S3 is as follows: S311: Read the current k-th step joint state and generation time Adopting behavioral strategies The velocity fields corresponding to three channels were obtained, including atomic type, fractional coordinates, and lattice. S312: Based on the joint state at step k Velocity field and preset discrete time step The conditional transfer center is calculated and represented as: ; in, This represents the conditional transition center of channel c before the random perturbation is added at step k. When c = F, This represents the conditional transition center in the fractional coordinate channel before the addition of random perturbation; H represents the channel, F represents the atom type, F represents the fractional coordinates, and L represents the lattice. This represents the state component of the joint state at step k on channel c; The velocity field of channel c represents the output of the behavioral policy based on the joint state at step k and the generation time. S313: Set an independent standard deviation for random disturbances for each channel, expressed as: ; in, This represents the standard deviation of the random perturbation in channel c; s represents the channel diffusion coefficient; s() represents the time scheduling function; S314: Sample independent standard Gaussian noise for each channel, and calculate the random joint state at step k+1 based on the one-step conditional transfer center and the standard deviation of the random perturbation. Obtain the random state component at step k+1 for each channel without periodic folding, expressed as: ; in, This represents the standard Gaussian noise of channel c at step k; Represents the random joint state at step k+1. Random state components on channel c; S315: Based on the random joint state at step k+1, perform state updates on the atom type state and lattice, perform periodic folding and state updates on the fractional coordinates, and obtain the updated joint state at step k+1. The joint state at step k+1 includes the atom-type state component, lattice-state component, and fractional coordinate state component at step k+1; the expression is: ; ; ; in, This represents the atom type state component at step k+1; This represents the random state component of the atom type at step k+1; The lattice state components at step k+1; The lattice random state components at step k+1; Represents the fractional coordinate state component at step k+1; represents the random state component of the fractional coordinates at step k+1; wrap represents the folding operation.

[0034] Furthermore, in S3, the joint logarithmic score of the actual action executed by the behavior policy for each generation step k of each trajectory is calculated, and all necessary information is cached in conjunction with it; the specific process is as follows: S321: Calculate the residual for each channel based on the state and conditional transition center at step k+1 for each channel. The expression is: ; ; ; ; ; in, To represent the atomic type residual, Euclidean residuals are used; Lattice residuals are represented using Euclidean residuals; The original fractional coordinate residual between the (k+1)th step fractional coordinate state component and the one-step conditional transition center without periodic equivalence processing is used to determine the periodic equivalence representative element. This represents the smallest image integer translation matrix obtained by rounding the original fractional coordinate residuals to the nearest integer. Represents a set of integers. This represents all atomic slots after filling, and 3 represents the corresponding three fractional coordinate components. This represents the effective residual of the fractional coordinates after periodic equivalence processing; S322: Calculate the logarithmic score for each channel by summing the effective atom dimensions of each channel based on the residuals, standard deviation of random perturbations, and effective atom masks, as follows: ; in, Let represent the logarithmic score of channel c at step k in the i-th trajectory; This represents the actual score residual of channel c at step k in the i-th trajectory, which can be represented as the atomic type residual, lattice residual, or fractional coordinate effective residual, respectively. , , ; This represents the standard deviation of the random perturbation in channel c; This represents the set of valid scalar dimension indices determined by the valid atomic mask, used to select the first... The scalar dimension of the trajectory participating in the summation in channel c, and the dimension corresponding to the filling atom are not included in this index set; according to Determine the mask length and construct an effective atom mask based on the actual effective atom count of the i-th trajectory. The mask value for effective atomic slots is 1, and the mask value for filled atomic slots is 0. S323: Sum the logarithmic scores of each channel according to the joint logarithmic probability form to calculate the joint logarithmic score, expressed as: ; in, Let represent the joint logarithmic score at step k in the i-th trajectory; This represents the logarithmic score of the atom type at the k-th step in the i-th trajectory; The logarithmic score of the fractional coordinates at the k-th step in the i-th trajectory; Let represent the lattice logarithm score at the k-th step in the i-th trajectory; S324: For the k-th step of the i-th trajectory, combine the joint logarithmic score corresponding to the behavior policy with the joint state at the k-th step, the joint state at the (k+1)-th step, and the generation time. Discrete time step Standard deviation of random disturbances for each channel The effective atomic mask and the minimum image rule of component rounding are associated with the cache to obtain cached data.

[0035] Furthermore, based on the original fractional coordinate residuals, the minimum image integer translation matrix is ​​calculated using the minimum image rule of rounding down by components. This minimum image integer translation matrix is ​​then used to calculate the effective residuals of the fractional coordinates after periodic equivalence processing. The effective residuals of the fractional coordinates are the periodic equivalent residuals of the fractional coordinates, expressed as: ; ; Here, round represents the rounding operation.

[0036] Furthermore, the material evaluation process in S4 is as follows: S41: Determine the crystal structure of each candidate crystal structure Does it meet the pre-inspection rules? If it does, then the candidate crystal structure will be... If the pre-inspection is passed, proceed to S42; if the pre-inspection is not met, mark it as an evaluation failure, retain the corresponding trajectory, and proceed to S5. The pre-inspection rules include: structurally resolvable; effective atomic number greater than 0; positive and finite crystal volume; and periodic minimum interatomic spacing greater than a preset threshold. ; S42: Conditionally triggered two-stage relaxation is used for each candidate crystal structure. Perform machine learning to relax the potential between atoms and obtain the convex hull energy difference; S421: Utilizing structural relaxation operators For candidate crystal structures Perform the first depth relaxation to obtain the first depth convex hull energy difference. If no abnormalities occur during the calculation process, the candidate crystal structure is marked as successfully evaluated and proceeds to S422; otherwise, it is marked as unsuccessfully evaluated and proceeds to S5; the expression is: ; ; in, This indicates the first deep relaxation structure; This indicates the first level of relaxed budget; This represents the energy difference of the first depth convex hull after the first depth relaxation. This represents an evaluation operator for calculating the energy difference of the first deep convex hull based on the energy, composition, and reference phase data of the first deep relaxation structure. S422: If the first depth convex hull energy difference , To preset the second depth trigger threshold, then for candidate crystal structures Performing a second-depth relaxation yields the energy difference of the second-depth convex hull. If no abnormalities occur during the calculation process, the candidate crystal structure is marked as successfully evaluated, and the second depth convex hull energy difference is recorded. The data is transmitted to S43. If an error occurs during the calculation process, it is marked as an evaluation failure, and the process proceeds to S5. If the first depth convex hull energy is poor Then the energy difference of the first depth convex hull will be... Transmitted to S43; The expression for the second deep relaxation is:

[0037] in, This indicates the second level of budget relaxation; This represents the energy difference of the second-depth convex hull after the second-depth relaxation; a successful evaluation means that there are no anomalies in the calculation of structural relaxation and convex hull energy difference, and that both the relaxed structure and the convex hull energy difference are valid finite values. S43: Based on each candidate crystal structure The final thermodynamic evaluation corresponding to the convex hull energy difference calculation is expressed as: or ; in, The final thermodynamic evaluation is indicated. When the energy difference of the first deep convex hull is greater than the second deep trigger threshold, the energy difference of the first deep convex hull is retained as the final thermodynamic evaluation. When it is less than or equal to the threshold, the second deep relaxation is performed on the same candidate crystal structure, and when the second deep relaxation evaluation is successful, the energy difference of the second deep convex hull is used as the final thermodynamic evaluation. When the first deep relaxation or the second deep relaxation after triggering, the structural analysis or the calculation of the convex hull energy difference is abnormal, or the output is a non-finite value, the candidate crystal structure is marked as an evaluation failure, and a preset finite failure reward is assigned in S51.

[0038] Furthermore, the pre-detection rules include several discrimination constraints: structural resolvability; effective number of atoms greater than 0; positive and finite crystal volume; and periodic minimum interatomic spacing greater than a preset threshold. .

[0039] Furthermore, the specific process of S5 is as follows: S51: For successfully evaluated candidate crystal structures, calculate the reward component based on the final thermodynamic evaluation, stability or metastability threshold, compositional effectiveness, inventiveness, compositional diversity, and structural diversity, and construct the trajectory-level total reward; the expression is: ; in, Indicates the reward component index; This represents the j-th reward component of the i-th candidate crystal structure after standardization within the same sampling batch; Indicates the first The preset weight of each reward item; This represents the total trajectory-level reward for the i-th candidate crystal structure. The reward components include one or more of the following: continuous convex hull energy reward, stability threshold reward, metastability threshold reward, compositional effectiveness reward, creativity reward, compositional diversity reward, and structural diversity reward. The continuous convex hull energy reward is obtained by truncating the final convex hull energy difference within preset upper and lower limits and taking its negative value. The stability threshold reward and metastability threshold reward are determined based on the final thermodynamic evaluation. The stability threshold reward is 1 if the final thermodynamic evaluation is not greater than 0 eV / atom, otherwise it is 0. The metastability threshold reward is 0 if the final thermodynamic evaluation is not greater than 0.1 eV / atom. The value is 1 for eV / atom, otherwise 0. The component validity reward is obtained by determining the component validity based on the combination of element oxidation states, charge neutrality, and electronegativity constraints. The creativity reward first determines the binary endpoint based on the novelty of the candidate composition relative to the reduced composition of the training set and its uniqueness within the same sampling batch. In the boundary case, under the same reduced composition, the score is continuously assigned based on the distance between the average minimum distance descriptor of the candidate structure and the descriptors of other candidate structures and the reference structure within the batch. The component diversity reward is obtained based on the contribution of the difference between the element score vector of the candidate composition and the maximum mean of the reference composition set within the same sampling batch. The structural diversity reward is obtained based on the contribution of the difference between the average minimum distance descriptor of the candidate structure and the maximum mean of the reference structure set within the same sampling batch. After each reward component is standardized within the same sampling batch, they are weighted and summed according to preset weights to obtain the trajectory-level total reward. For candidate crystal structures that fail evaluation, their total trajectory-level reward is set to the minimum total trajectory-level reward of the candidate crystal structures that succeeded evaluation within the same sampling batch minus a preset finite interval, thereby avoiding invalid values ​​from entering the gradient and ensuring that failed trajectories usually gain a negative advantage relative to the effective trajectories in the same group. S52: At the start of a reinforcement learning iteration, fix the behavior policy parameters in S2 and use the behavior policy to generate multiple trajectories in a sampling batch; divide the candidate crystal structures corresponding to the termination states of each trajectory within the sampling batch into the same comparison group, and the identifier of the comparison group to which the i-th candidate crystal structure belongs is... According to the same comparison group The total trajectory-level reward for each candidate crystal structure is calculated as the trajectory-level advantage, expressed as: ; in, This represents the trajectory-level advantage of the i-th candidate crystal structure; and These represent the mean and standard deviation of the total trajectory-level reward within the same comparison group, respectively. This represents a preset positive number used for numerical stability. ; This indicates the correspondence between the total trajectory-level reward and the comparison group identifier, where... Let represent the total trajectory-level reward for the m-th candidate crystal structure. The comparison group identifier represents the m-th candidate crystal structure; in the dominance calculation formula, the one that satisfies... =g i The total trajectory-level rewards are grouped into a set of total trajectory-level rewards for the same comparison group. This set is used to calculate the mean and standard deviation of the total trajectory-level rewards within the same comparison group; the total rewards for the same trajectory The policy gradient is then used for the selected generation step of the trajectory.

[0040] Furthermore, a reinforcement learning iteration includes a trajectory sampling phase and a parameter optimization phase. In the trajectory sampling phase, a batch of randomly generated trajectories is generated and cached using a fixed behavior policy. In the parameter optimization phase, the cached trajectories themselves remain unchanged, and only the current policy parameters are updated. Each batch of trajectories is used only once for parameter updates. S6 is the parameter optimization phase, and the specific process is as follows: S61: Extract the cached behavior strategy corresponding to the joint logarithmic score and adjacent state transition pairs from the cached data of multi-channel randomly generated trajectories. Generation time Discrete time step Using joint logarithmic scoring As a baseline, and using the current strategy For the k-th step joint state and corresponding generation time Perform forward computation to obtain the current velocity field of each channel, and then calculate based on the current velocity field and the discrete time step. Calculate the current conditional transfer center; expressed as: ; in, Indicates the current conditional transfer center of channel c; This represents the current velocity field of channel c; This indicates the joint state at step k based on the i-th trajectory. The state component of the k-th step of a given channel c; S62: Without resampling the state transition results, use the cached joint state at step k+1. As the scoring object, and strictly reusing the standard deviation of random perturbation of each channel, effective atomic mask, and minimum image rule of rounding by components, the current logarithmic scores of the three channels are recalculated separately, and then summed to obtain the current joint logarithmic score. , represented as: ; in, Indicates the current logarithmic score of the atom type; Represents the current logarithmic score on the fractional coordinate; This represents the current logarithmic score of the lattice. For the fractional coordinate channel, the original residual is recalculated based on the current conditional transfer center, and the current fractional coordinate effective residual is determined according to the same component-based minimum image rule. The logarithmic score of the current fractional coordinate is calculated using the current fractional coordinate effective residual and the standard deviation of the random disturbance. The expression for the current fractional coordinate effective residual is: ; in, Indicates the current conditional transition center of the fractional coordinates; Represents the fractional coordinate state component at step k+1; ensures that the behavior policy score and the current policy recalculation use the same periodic semantics for the same physical action; S63: Scoring based on joint logarithm and current joint logarithmic score The difference between them constructs a stepwise importance ratio. The expression is: ; S64: Construct the pruning strategy loss based on the stepwise importance ratio The pruning strategy loss serves as the strategy optimization loss for a single parameter update of this batch of trajectories. The expression is: ; in, This indicates the preset clipping width that limits a single policy change; This represents the expectation operator that averages the index pairs (i,k) of the trajectories and generation steps involved in this parameter optimization. This indicates the trajectory-level advantage of the candidate crystal structure corresponding to the i-th trajectory; In this implementation, each batch of cached traces is used only for one parameter update. At the start of this round of parameter updates, the current policy and the behavior policy are the same. At this update starting point, the gradient of the pruning target is equivalent to the gradient of the following fractional function: ; in, This represents the gradient of the policy gradient loss with respect to the current policy parameters; This indicates calculating the gradient with respect to the current policy parameters; S65: Based on the joint state of the cache, generation time, discrete time step, and standard deviation of random perturbation for each channel, use the reference strategy. Forward computation is performed on the same cache joint state and generation time to obtain the reference velocity field and reference conditional transition center for each channel; the reference conditional transition centers are mapped one-to-one with the current conditional transition centers obtained in S61 according to the same trajectory, generation step, and channel, and the stepwise reference constraints of the current policy relative to the reference policy are calculated; the current policy and the reference policy use the same random perturbation variance in the same channel, and the stepwise reference constraints are expressed as the difference between the mean of the reference conditional transition center and the current conditional transition center: ; in, The stepwise reference constraint of channel c at step k of the i-th trajectory is represented; Indicates the current conditional transfer center of channel c; Indicates the reference condition transfer center for channel c; This represents the standard deviation of the random perturbation in channel c; S66: The reference constraint term is obtained by averaging the stepwise reference constraints of each trajectory, generation step, and channel. The strategy optimization loss is combined with the reference constraint term to form the total loss. , represented as: ; S67: Compare the reference constraint terms with the preset circuit breaker threshold; if the preset circuit breaker threshold is exceeded, skip this parameter update; if the preset circuit breaker threshold is not exceeded, perform backpropagation on the total loss, perform global gradient pruning on the gradient of the current policy parameters, and use the optimizer to perform an update on the current policy parameters. S68: After this parameter update is completed, the reference policy remains fixed within the preset training interval; when the preset refresh interval is reached, the updated current policy parameters are copied to the reference policy; then the updated current policy is used as the behavior policy for the next round, and S3 is returned to regenerate the trajectory of a sampling batch.

[0041] Furthermore, the updated direct spatial crystal flow generation model is used to perform multi-step integration starting from the atom type, fractional coordinates, and lattice prior states, jointly generating the atom type, fractional coordinates, and lattice parameters of the candidate crystal, and obtaining the ab initio crystal structure optimized by reinforcement learning.

[0042] Furthermore, pre-inspection rules may also include lattice parameter ranges, lattice density ranges, or lattice composition constraints.

[0043] Furthermore, the multi-objective reward, time step selection method, reference policy refresh cycle, two-level relaxation depth, and trigger threshold can all be set according to the dataset, pre-trained streaming model, and evaluation budget.

[0044] On the other hand, in one specific embodiment, a direct spatial crystal flow model pre-trained on the MP-20 dataset is used to jointly generate atom-type states. Periodic fraction coordinates and lattice state In the MP-20 dataset, each unit cell contains at most 20 atoms. Reinforcement learning is then applied to train this direct space crystal flow model, such as... Figure 1 As shown, the specific process is as follows: S1: Obtain the pre-trained direct space crystal flow model and perform policy initialization; initialize the current policy, behavior policy, and reference policy from the pre-trained checkpoints.

[0045] S2: Three-channel random trajectory generation; The following formula is used to generate a random next state for the three state channels;

[0046] in, It is the transfer center before noise injection in the fractional coordinate channel; the fractional coordinate motion gradually wraps back to the periodic fundamental domain; 96 trajectories are generated in a batch, and the final state of each trajectory is decoded into a candidate crystal.

[0047] S3: Periodic Consistency Transfer Score; Actions on fractional coordinates Difference from the corresponding transfer center calculate:

[0048] Based on this, the local Gaussian logarithmic score of the fractional coordinate channel is calculated; the Gaussian logarithmic scores of the atom type and lattice channels are calculated using Euclidean residuals, and the logarithmic scores of the three channels are added together to obtain the stepwise joint score; filling atoms are excluded by the effective atom mask; Trajectory caching is performed; after each sampling step generates a state transition, the joint state at step k, the joint state at step k+1, the generation time, the discrete time step size, the standard deviation of random perturbations for each channel, the joint score of the behavior policy, the effective atomic mask, and the minimum image rule rounded down by components are synchronously saved; when the current policy is recalculated, the same adjacent state transition pair is read, and the above minimum image rule is applied. A schematic diagram comparing Euclidean residuals and periodically equivalent residuals when fractional coordinates cross periodic boundaries is shown below. Figure 2 As shown, Figure 2 In the middle (a), the Euclidean residual is represented. Let 'a' represent the Euclidean residual and 'a' represent the action. The Euclidean residual calculation process, which represents the same small physical displacement as a large residual, involves Gaussian penalty and anomalous amplification of the policy gradient. Figure 2 In the middle (b), the periodic equivalent residual is represented. This represents the periodic equivalent residual. This represents the original residual between the actual action and the conditional transfer center without looping. The calculation process for the periodically equivalent residual involves selecting the smallest periodically equivalent residual, and the scoring of this process is consistent with the physical displacement; in the figure, it ranges from 0.49 to... Taking a motion of 0.49 as an example, the Euclidean residual is: 0.98, with a periodic equivalent residual of 0.02. Random transfer of atom type, fractional coordinates, and lattice three channels is as follows: Figure 3 As shown, the current state, drift center, independent diffusion scale, and random next state of each channel are displayed respectively.

[0049] S4: Crystal formation and material evaluation; For each trajectory termination state, candidate crystal structures are obtained through analysis. Pre-checks are performed on the resolvability, effective number of atoms, cell volume, and minimum periodic interatomic spacing of each candidate crystal structure. If the pre-check passes, 50 steps of first-order deep machine learning interatomic potential relaxation are performed on the candidate crystal structure, and calculations are performed. ;when At eV / atom, for the same candidate crystal structure, the second deep relaxation is re-executed and calculated using a 200-step budget. If the second deep relaxation is not triggered, the energy difference of the first deep convex hull is used as the final thermodynamic evaluation; if the second deep relaxation is triggered and the evaluation is successful, the energy difference of the second deep convex hull is used as the final thermodynamic evaluation; if the pre-check, structural relaxation or convex hull energy difference calculation fails, the corresponding trajectory is retained and a limited failure reward is assigned.

[0050] S5: Material Rewards and Group Relative Advantages; The continuous stability reward, metastability threshold reward, stability threshold reward, creativity reward, component diversity reward, structural diversity reward, and component effectiveness reward were standardized within the same sampling batch and then weighted and summed according to weights of 1.0, 2.0, 1.0, 1.5, 1.0, 0.25, and 1.0, respectively. The continuous stability reward is a negative value after truncating the final convex hull energy difference within the range of 0 to 3.0 eV / atom. The stability threshold reward and metastability threshold reward are set to 0 or 1 based on the 0 eV / atom and 0.1 eV / atom thresholds, respectively. The component validity reward is determined based on the component validity judgment based on oxidation state combination, charge neutrality, and electronegativity constraints. The creativity reward first determines the binary endpoint based on the novelty of the reduced composition relative to the training set and the uniqueness within the batch, and then continuously assigns scores based on the minimum distance of the average minimum distance descriptor within the same reduced composition in the boundary case. Component diversity and structural diversity are respectively calculated using the difference contribution of the element score vector and the average minimum distance descriptor relative to the corresponding reference set with one-out maximum mean. The original contributions of the two types of diversity are first normalized to 0 to 1 in the valid candidates of the same sampling batch. The total trajectory-level reward for the failed candidates is set as the minimum total trajectory-level reward of the successful candidates evaluated in this batch minus 1.0. In this embodiment, the same sampling batch is used as the comparison group, and the rewards are calculated based on the mean and standard deviation within the group. ; the same trajectory The policy gradient is used for the selected generation step of this trajectory. The conditional two-stage relaxation and reward calculation process is as follows: Figure 4 As shown, a trajectory termination state corresponds to a candidate crystal structure; the first deep structural relaxation is performed on the candidate crystal structure that passes the pre-check and the preliminary convex hull energy difference is calculated. Based on the preliminary convex hull energy difference, it is determined whether to increase the relaxation budget for the same candidate crystal structure. The candidate crystal structure that undergoes the second deep structural relaxation obtains the deep convex hull energy difference; the final convex hull energy difference is determined as the final thermodynamic evaluation according to the first stage, the second stage, or the failed branch; then the stability, metastability, effectiveness, creativity, and diversity rewards are calculated to construct the trajectory-level total reward.

[0051] S6: Strategy Update; Fixed adjacent state transition pairs and behavioral policy scores are read from the trajectory cache. Scores for all three channels are recalculated using the current policy, and the minimum image residual rule (rounded by components) in S3 is reused in the fractional coordinate channels. The stepwise importance ratio is calculated based on the difference between the old and new joint scores, and the pruning policy loss is calculated with a pruning width of 0.03. Each batch of cached trajectories is used only for one parameter update. The mean difference reference constraint between the conditional transition centers of each channel of the current policy and the reference policy is calculated on the same cached joint state, using a 5×10... 4 The coefficients and the pruning policy loss are combined to form the total loss; when the reference constraint does not exceed the preset circuit breaker threshold, backpropagation, global gradient pruning, and optimizer update are performed, where the global gradient pruning threshold is 2.0; every 5 training iterations, the current policy parameters are copied to the reference policy.

[0052] S7: Model output; Integrate from the three-channel prior state using the updated model to generate the ab initio crystal structure.

[0053] In this embodiment, the reproducible parameter values ​​used in the model training process are shown in Table 1 below.

[0054] Table 1 Reproducible parameter values

[0055] On the other hand, in one specific embodiment, a direct spatial crystal flow model pre-trained on the Alex-MP-20 dataset is trained using reinforcement learning to sequentially generate three-channel random trajectories, execute periodic transition scores, perform two-level relaxation, material rewards, and group-relative policy updates. The novelty reference library, phase diagram, reward weights, and relaxation thresholds can be adjusted according to the data distribution; the periodic scores and the behavioral policy-current policy semantic binding remain unchanged. This embodiment illustrates that the method of the present invention can be extended to a larger composition-structure space.

[0056] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0057] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A reinforcement learning method for a direct spatial crystal flow generation model, characterized in that, Includes the following steps: Step 1: Obtain a pre-trained direct spatial crystal flow generation model; Step 2: Initialize strategy parameters based on the direct space crystal flow generation model; Step 3: Generate multi-channel randomly generated trajectories using the direct spatial crystal flow generation model and strategy parameters, and calculate the joint logarithm score for each trajectory; Step 4: Decode the termination state of each trajectory into a candidate crystal structure, and perform material evaluation on the candidate crystal structure to obtain the final thermodynamic evaluation; Step 5: Calculate the trajectory-level advantage of the candidate crystal structure based on the final thermodynamic evaluation; Step 6: Recalculate the joint logarithmic score of the multi-channel randomly generated trajectory using the policy parameters, and construct the pruning policy loss by combining the trajectory-level advantages to update and optimize the direct spatial crystal flow generation model; Step 7: Predict the output ab initio crystal structure using the optimized direct space crystal flow generation model.

2. The reinforcement learning method for a direct spatial crystal flow generation model as described in claim 1, characterized in that, The direct spatial crystal flow generation model pre-trained on the crystal structure dataset is represented as: ； ； in, Indicates the joint state at step k; For the atom type state at step k, The fractional coordinate state at step k. This represents the lattice state at step k; This indicates that based on the joint state at step k... and the generation time of step k Output velocity field; This represents the velocity field corresponding to the atom type channel at step k; This represents the velocity field corresponding to the fractional coordinate channel at step k; This represents the velocity field corresponding to the lattice channel at step k.

3. The reinforcement learning method for a direct spatial crystal flow generation model as described in claim 2, characterized in that, The strategy parameters include the behavioral strategy, the current strategy, and the reference strategy.

4. The reinforcement learning method for a direct spatial crystal flow generation model as described in claim 3, characterized in that, The specific process of generating each generation step k of each trajectory in step 3 is as follows: Step 311: Read the current joint state at step k. and generation time Adopting behavioral strategies The velocity fields corresponding to three channels were obtained, including atomic type, fractional coordinates, and lattice. Step 312: Based on the joint state in step k Velocity field and preset discrete time step The conditional transfer center is calculated and represented as: ； in, This represents the conditional transition center of channel c before the random perturbation is added at step k; H represents the channel, F represents the atom type, F represents the fractional coordinates, and L represents the lattice. This represents the state component of the joint state at step k on channel c; The velocity field of channel c represents the output of the behavioral policy based on the joint state at step k and the generation time. Step 313: Set an independent random disturbance standard deviation for each channel, expressed as: ； in, This represents the standard deviation of the random perturbation in channel c; s represents the channel diffusion coefficient; s() represents the time scheduling function; Step 314: Sample independent standard Gaussian noise for each channel, and calculate the random joint state at step k+1 based on the one-step conditional transfer center and the standard deviation of the random perturbation. Obtain the random state component at step k+1 for each channel without periodic folding, expressed as: ； in, This represents the standard Gaussian noise of channel c at step k; Represents the random joint state at step k+1. Random state components on channel c; Step 315: Based on the random joint state of step k+1, perform state updates on the atom type and lattice, perform periodic folding and state updates on the fractional coordinates, and obtain the updated joint state of step k+1. The expression is: ； ； ； in, This represents the atom type state component at step k+1; This represents the random state component of the atom type at step k+1; The lattice state components at step k+1; The lattice random state components at step k+1; Represents the fractional coordinate state component at step k+1; represents the random state component of the fractional coordinates at step k+1; wrap represents the folding operation.

5. The reinforcement learning method for a direct spatial crystal flow generation model as described in claim 4, characterized in that, Step 3 also includes using a transition scoring method to calculate the joint logarithmic score of the actual actions performed by the behavioral strategy at each generation step k of each trajectory; the specific process is as follows: Step 321: Calculate the residual for each channel based on the state and conditional transition center of step k+1 corresponding to each channel. The expression is: ； ； ； ； ； in, Indicates the atomic type residual; Indicates lattice residual; This represents the original fractional coordinate residual between the (k+1)th step fractional coordinate state component and the one-step conditional transition center without periodic equivalence processing; This represents the smallest image integer translation matrix obtained by rounding the original fractional coordinate residuals to the nearest integer. Represents a set of integers. This represents all atomic slots after filling; This represents the effective residual of the fractional coordinates after periodic equivalence processing; Step 322: Sum the effective atom dimensions of each channel based on the residuals, random perturbation standard deviations, and effective atom masks, and calculate the logarithmic score for each channel, expressed as: ； in, Let represent the logarithmic score of channel c at step k in the i-th trajectory; The actual score residual of channel c at step k in the i-th trajectory is represented by the atomic type residual, lattice residual, or fractional coordinate effective residual, respectively. This represents the standard deviation of the random perturbation in channel c; This represents the set of effective scalar dimension indices for channel c, determined by the effective atomic mask; according to Determine the mask length and construct an effective atom mask based on the actual effective atom count of the i-th trajectory. ; Step 323: Sum the logarithmic scores of each channel to calculate the joint logarithmic score, expressed as: ； in, Let represent the joint logarithmic score at step k in the i-th trajectory; This represents the logarithmic score of the atom type at the k-th step in the i-th trajectory; The logarithmic score of the fractional coordinates at the k-th step in the i-th trajectory; Let represent the lattice logarithm score at the k-th step in the i-th trajectory; Step 324: For the k-th step of the i-th trajectory, combine the joint logarithmic score corresponding to the behavior policy with the joint state at the k-th step, the joint state at the (k+1)-th step, and the generation time. Discrete time step Standard deviation of random disturbances for each channel The effective atomic mask and the minimum image rule of component rounding are associated with the cache to obtain cached data.

6. The reinforcement learning method for a direct spatial crystal flow generation model as described in claim 4, characterized in that, The process of material evaluation in step 4 is as follows: Step 41: Determine the crystal structure of each candidate crystal. Check if the pre-inspection rules are met; if they are met, mark the candidate crystal structure as pre-inspection passed and proceed to step 42; if they are not met, mark the candidate crystal structure as evaluation failed, retain the corresponding trajectory and proceed to step 5. Step 42: Apply conditionally triggered two-stage relaxation to each candidate crystal structure Perform machine learning to relax the potential between atoms and obtain the convex hull energy difference; Step 43: Based on each candidate crystal structure The convex hull energy difference determines the corresponding final thermodynamic evaluation.

7. The reinforcement learning method for a direct spatial crystal flow generation model as described in claim 6, characterized in that, Step 42 specifically includes: Step 421: Utilize the structural relaxation operator For candidate crystal structures Perform the first depth relaxation to obtain the first depth convex hull energy difference. If no abnormality occurs during the calculation process, the candidate crystal structure is marked as successfully evaluated and proceeds to step 422; otherwise, it is marked as unsuccessful and proceeds to step 5. Step 422: If the first depth convex hull has a difference , To preset the second depth trigger threshold, then for candidate crystal structures Performing a second-depth relaxation yields the energy difference of the second-depth convex hull. If no abnormalities occur during the calculation process, the candidate crystal structure is marked as successfully evaluated, and the second depth convex hull energy difference is recorded. If an error occurs during the calculation process, the process is marked as an evaluation failure and proceeds to step 5. If the first depth convex hull energy is poor Then the energy difference of the first depth convex hull will be... Proceed to step 43.

8. The reinforcement learning method for a direct spatial crystal flow generation model as described in claim 7, characterized in that, The specific process of step 5 is as follows: Step 51: For successfully evaluated candidate crystal structures, calculate the reward component based on one or more of the following: final thermodynamic evaluation, stability or metastability threshold, compositional effectiveness, inventiveness, compositional diversity, and structural diversity, and construct the trajectory-level total reward; the expression is: ； in, Indicates the reward component index; This represents the j-th reward component of the i-th candidate crystal structure after standardization within the same sampling batch; Indicates the first The preset weight of each reward item; Represents the total trajectory-level reward for the i-th candidate crystal structure; the reward components include one or more of the following: continuous convex hull energy reward, stability threshold reward, metastability threshold reward, compositional effectiveness reward, creativity reward, compositional diversity reward, and structural diversity reward. For candidate crystal structures that fail the evaluation, their total trajectory-level reward is set to the minimum total trajectory-level reward of the candidate crystal structures that succeed in the same sampling batch minus a preset finite interval. Step 52: Generate multiple trajectories using a behavioral strategy, and divide the candidate crystal structures corresponding to each trajectory within the same sampling batch into the same comparison group. ,in The identifier of the comparison group to which the i-th candidate crystal structure belongs is used; the trajectory-level advantage is calculated based on the total trajectory-level reward of each candidate crystal structure within the same comparison group, and is expressed as: ； in, This represents the trajectory-level advantage of the i-th candidate crystal structure; and These represent the mean and standard deviation of the total trajectory-level reward within the same comparison group, respectively. This represents a preset positive number used for numerical stability; This represents the correspondence between the total trajectory-level reward and the comparison group, where Let represent the total trajectory-level reward for the m-th candidate crystal structure. This indicates the identifier of the comparison group to which the m-th candidate crystal structure belongs.

9. The reinforcement learning method for a direct spatial crystal flow generation model as described in claim 8, characterized in that, The specific process of step 6 is as follows: Step 61: Extract adjacent state transition pairs from the cached data of multi-channel randomly generated trajectories. Generation time Discrete time step and the joint logarithmic score corresponding to the caching behavior strategy. Using joint logarithmic scoring As a baseline, and using the current strategy For the k-th step joint state and corresponding generation time Perform forward computation to obtain the current velocity field of each channel, and then calculate based on the current velocity field and the discrete time step. Calculate the current conditional transfer center; expressed as: ； in, Indicates the current conditional transfer center of channel c; This represents the current velocity field of channel c; This indicates the joint state at step k based on the i-th trajectory. The state component of the k-th step of a given channel c; Step 62: Without resampling the state transition results, use the cached joint state from step k+1. As the scoring object, and using the standard deviation of random perturbation of each channel, the effective atomic mask, and the minimum image rule of rounding by components, the current logarithmic scores of the three channels are recalculated, and then summed to obtain the current joint logarithmic score. , is represented as: ； in, Indicates the current logarithmic score of the atom type; Represents the current logarithmic score on the fractional coordinate; This represents the current logarithmic score of the lattice. Step 63: Score based on joint logarithm and current joint logarithmic score The difference between them constructs a stepwise importance ratio. The expression is: ； Step 64: Construct the pruning strategy loss based on the stepwise importance ratio The pruning strategy loss serves as the strategy optimization loss for a single parameter update of this batch of trajectories. The expression is: ； in, This indicates the preset clipping width that limits a single policy change; This represents the expectation operator that averages the index pairs of the trajectories and generation steps involved in this parameter update; This indicates the trajectory-level advantage of the candidate crystal structure corresponding to the i-th trajectory; Step 65: Based on the joint state of the cache, generation time, discrete time step, and standard deviation of random perturbation for each channel, use the reference strategy. Forward computation is performed on the joint state and generation time of the same cache to obtain the reference velocity field and reference conditional transition center for each channel. The reference conditional transition centers are then mapped one-to-one with the current conditional transition centers according to the same trajectory, generation step, and channel. The stepwise reference constraints of the current policy relative to the reference policy are calculated. This is represented as: ； in, The stepwise reference constraint of channel c at step k of the i-th trajectory; Indicates the current conditional transfer center of channel c; Indicates the reference condition transfer center for channel c; This represents the standard deviation of the random perturbation in channel c; Step 66: Averaging the stepwise reference constraints for each trajectory, generation step, and channel yields the reference constraint term. The strategy optimization loss is combined with the reference constraint term to form the total loss. , is represented as: ； Step 67: Compare the reference constraint with the preset circuit breaker threshold; if it exceeds the preset circuit breaker threshold, skip this parameter update; if it does not exceed the preset circuit breaker threshold, perform backpropagation on the total loss, perform global gradient pruning on the gradient of the current policy parameters, and use the optimizer to update the current policy parameters once. Step 68: After the parameter update is completed, the reference policy remains fixed within the preset training interval; when the preset refresh interval is reached, the updated current policy parameters are copied to the reference policy; then the updated current policy is used as the behavior policy for the next round, and the process returns to step 3 to regenerate the trajectory of a sampling batch.

10. The reinforcement learning method for a direct spatial crystal flow generation model as described in claim 1, characterized in that, Using the updated direct spatial crystal flow generation model, multi-step integration is performed starting from the atom type, fractional coordinates, and lattice prior states to jointly generate the atom type, fractional coordinates, and lattice parameters of candidate crystals, resulting in ab initio crystal structures optimized by reinforcement learning.

Images