Polymer formula analysis method and device based on graph Q learning network

Abstract

The invention discloses a polymer formula analysis method and device based on a graph Q learning network, and the method comprises the steps: building a Markov decision process of formula generation through defining a state space, an action space, a state transition probability and a reward function; constructing a graph Q learning network model based on the Markov decision process; training the graph Q learning network model through a double-Q learning algorithm and a strategy function to obtain a trained graph Q learning network model; and performing state transition and action selection processing through the trained graph Q learning network model to generate a polymer formula. According to the method, the problems of low learning speed, high computing resource consumption, difficulty in accurately matching multi-target performance and the like in the prior art are solved.

Description

Technical Field

[0001] This invention relates to the field of polymer formulation optimization technology, specifically to a polymer formulation analysis method and apparatus based on graph Q learning networks. Background Technology

[0002] In the field of polymer material research and development and production, formulation optimization is a core step that determines material performance, production costs, and application scenarios. Precise formulation derivation technology to match target performance has become a key requirement for industry development. Currently, polymer formulation generation technologies are mainly divided into two categories: traditional experimental trial-and-error methods and machine learning-based intelligent generation methods. Traditional experimental trial-and-error methods rely on the experience accumulated by researchers, adjusting raw material combinations and ratios through numerous orthogonal experiments to gradually approach the target performance. Among machine learning-based methods, policy learning techniques, by constructing generative network structures and pre-training on massive datasets of formulation strings, achieve automated formulation generation, and have already been initially applied in the formulation development of some general-purpose polymer materials.

[0003] However, existing technologies still have significant limitations: traditional experimental trial-and-error methods are extremely inefficient, have long development cycles, and incur high costs in terms of manpower and materials, and struggle to simultaneously meet the precise matching of multi-dimensional target performance; while policy-based intelligent generation methods improve efficiency to some extent, they rely on complex generative network structures, have stringent requirements on the scale and quality of pre-training datasets, are slow to learn, and consume a large amount of computational resources, making them unsuitable for the development of small-batch, multi-batch personalized formulations. As for attempts based on value learning, a complete decision-making mechanism and network model architecture have not yet been established, and there is still considerable room for improvement in the targeting, accuracy, and efficiency of formulation generation, failing to fully meet the industry's urgent need for efficient, accurate, and low-cost polymer formulation derivation technology.

[0004] Therefore, there is an urgent need for a polymer formulation analysis method based on graph Q learning networks to solve the problems of slow learning speed, high computational resource consumption, and difficulty in accurately matching multi-objective performance in existing technologies. Summary of the Invention

[0005] To address these issues, this invention provides a polymer formulation analysis method and apparatus based on a graph Q learning network, which solves the problems of slow learning speed, high computational resource consumption, and difficulty in accurately matching multiple target performances in existing polymer formulation generation methods, and enables efficient back-calculation of the optimal formulation when multiple polymer target performances are given.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a polymer formulation analysis method based on graph-Q learning networks, characterized in that it includes:

[0007] By defining the state space, action space, state transition probability, and reward function, a Markov decision process for recipe generation is constructed.

[0008] Based on the Markov decision process, a graph Q-learning network model is constructed; the graph Q-learning network model is trained using a double Q-learning algorithm and a policy function to obtain a trained graph Q-learning network model; state transition and action selection are performed on the trained graph Q-learning network model to generate a polymer formulation.

[0009] As a preferred embodiment of the polymer formulation analysis method based on graph Q learning networks, in defining the state space, the action space, the state transition probability, and the reward function, the state space is defined based on the polymer formulation dataset; the action space is defined based on the formulation composition and adjustment requirements of the state space; the state transition probability is defined according to the deterministic execution rules of the action space; and the reward function is defined based on the formulation performance matching target.

[0010] As a preferred embodiment of the polymer formulation analysis method based on graph Q learning networks, when defining the state space, each state in the state space is a (f,t) tuple; initially, f is empty, t=0, and the final set of states is:

[0011]

[0012] In the formula, f is the current recipe; t is the current step number; T is the maximum number of time steps; and s is each state in the state space.

[0013] When defining the action space, the action space includes five actions: adding new raw materials, adjusting the raw material ratio upwards, adjusting the raw material ratio downwards, deleting raw materials, and making no changes.

[0014] As a preferred embodiment of the polymer formulation analysis method based on graph Q learning network, the state transition probability is specified as 1 when defining the state transition probability, so that the state transition is deterministic.

[0015] When defining the reward function, the reward is calculated based on the relative error between the predicted performance and the target performance of the generated recipe; the formula for calculating the reward is:

[0016]

[0017] In the formula, Here, T is the discount factor; T is the maximum time step; t is the current step number. This is a comprehensive reward for multi-objective performance.

[0018] As a preferred embodiment of the polymer formulation analysis method based on graph Q-learning networks, the graph Q-learning network model has a two-input head structure, including: an A input head, a B input head, a graph convolutional layer, a fully connected hidden layer, and an output layer; the A input head is a graph representation of the formulation raw materials; the B input head is the formulation vector and the remaining steps of the episode; the output of the graph Q-learning network model is the Q-value; during the training of the graph Q-learning network model, the policy function is... Greedy strategy.

[0019] This invention also provides a polymer formulation analysis device based on graph Q learning networks, employing the above-mentioned polymer formulation analysis method based on graph Q learning networks, including:

[0020] The Markov Decision Process Construction Module is used to construct a recipe-generated Markov Decision Process by defining the state space, action space, state transition probabilities, and reward function.

[0021] The Graph Q Learning Network Model Training and Processing Module is used to construct a Graph Q Learning Network Model based on the Markov Decision Process; train the Graph Q Learning Network Model using the Double Q Learning Algorithm and Policy Function to obtain the trained Graph Q Learning Network Model; and perform state transition and action selection processing on the trained Graph Q Learning Network Model to generate a polymer formulation.

[0022] As a preferred embodiment of a polymer formulation analysis device based on a graph Q learning network, the Markov decision process construction module defines the state space based on a polymer formulation dataset during the process of defining the state space, the action space, the state transition probability, and the reward function; defines the action space based on the formulation composition and adjustment requirements of the state space; defines the state transition probability according to the deterministic execution rules of the action space; and defines the reward function based on the formulation performance matching objective.

[0023] As a preferred embodiment of a polymer formulation analysis device based on a graph Q learning network, in the Markov decision process construction module, when defining the state space, each state in the state space is a (f,t) tuple; initially, f is empty, t=0, and the final set of states is:

[0024]

[0025] In the formula, f is the current recipe; t is the current step number; T is the maximum number of time steps; and s is each state in the state space.

[0026] When defining the action space, the action space includes five actions: adding new raw materials, adjusting the raw material ratio upwards, adjusting the raw material ratio downwards, deleting raw materials, and making no changes.

[0027] As a preferred embodiment of the polymer formulation analysis device based on graph Q learning network, in the Markov decision process construction module, when defining the state transition probability, the state transition probability is specified as 1, so that the state transition has determinism.

[0028] When defining the reward function, the reward is calculated based on the relative error between the predicted performance and the target performance of the generated recipe; the formula for calculating the reward is:

[0029]

[0030] In the formula, Here, T is the discount factor; T is the maximum time step; t is the current step number. This is a comprehensive reward for multi-objective performance.

[0031] As a preferred embodiment of a polymer formulation analysis device based on a graph Q-learning network, the graph Q-learning network model training and processing module comprises a two-input-head structure, including: an A-input head, a B-input head, a graph convolutional layer, a fully connected hidden layer, and an output layer; the A-input head is a graph representation of the formulation raw materials; the B-input head is the formulation vector and the remaining steps of the episode; the output of the graph Q-learning network model is the Q-value; and during the training of the graph Q-learning network model, the strategy function is... Greedy strategy.

[0032] The present invention has the following advantages:

[0033] First, precise matching of multi-objective performance: By using a reward function based on the relative error of performance and combining it with multi-objective weight design, the performance requirements of multiple polymer targets can be met simultaneously, resulting in a high degree of matching between formulation performance and target performance.

[0034] Second, it has high generation efficiency and low resource consumption: it does not require complex pre-trained datasets, and with the help of Markov decision processes with deterministic state transitions and graph Q learning network architecture, it can significantly improve the speed of recipe generation and reduce the consumption of computing resources.

[0035] Third, the model has strong representation capabilities: the dual input head design simultaneously captures the molecular structure features of raw materials, as well as the formula vector and progress information. The graph convolutional layer effectively extracts deep molecular features, improving the targeting and rationality of formula generation.

[0036] Fourth, the training is stable and converges well: the double Q learning algorithm avoids overestimation of Q value, the ε greedy strategy balances exploration and utilization, and the Adam optimizer and reasonable parameter settings ensure stable training and fast convergence of the model.

[0037] Fifth, it has a wide range of applications and high flexibility: it supports multiple actions such as adding raw materials, adjusting the ratio, and deleting raw materials, adapting to the formulation development needs of different types of polymer materials, and can flexibly cope with personalized, multi-batch formulation derivation scenarios. Attached Figure Description

[0038] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are merely exemplary, and those skilled in the art can derive other embodiments based on the provided drawings without creative effort.

[0039] The structures, proportions, sizes, etc. illustrated in this specification are only for the purpose of assisting those skilled in the art in understanding and reading the content disclosed herein, and are not intended to limit the conditions under which the present invention can be implemented. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportions, or adjustments to the size, without affecting the effects and objectives that the present invention can produce, should still fall within the scope of the technical content disclosed in the present invention.

[0040] Figure 1 This is a schematic flowchart of the polymer formulation analysis method based on graph Q learning network provided in Embodiment 1 of the present invention;

[0041] Figure 2 This is a schematic diagram of the graph Q learning network structure in the polymer formulation analysis method based on graph Q learning network provided in Embodiment 1 of the present invention;

[0042] Figure 3 This is a schematic diagram of the architecture of the polymer formulation analysis device based on graph Q learning network provided in Embodiment 2 of the present invention. Detailed Implementation

[0043] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0044] Example 1

[0045] See Figure 1 Embodiment 1 of the present invention provides a polymer formulation analysis method based on graph Q learning network, comprising the following steps:

[0046] S1. By defining the state space, action space, state transition probability, and reward function, a Markov decision process for recipe generation is constructed.

[0047] S2. Based on the Markov decision process, construct a graph Q learning network model; train the graph Q learning network model using the double Q learning algorithm and policy function to obtain the trained graph Q learning network model; perform state transition and action selection processing on the trained graph Q learning network model to generate a polymer formulation.

[0048] In this embodiment, in step S1, a Markov decision process for recipe generation is constructed by defining the state space, action space, state transition probability, and reward function.

[0049] Specifically, firstly, a polymer formulation dataset containing raw material names, raw material ratios, and corresponding performance test values ​​is collected. Based on this dataset, a state space S is defined: each state s in the state space is represented by a (f, t) tuple. Initially, f is empty, t=0, and the state reaches its endpoint when t=T. The set of endpoint states is:

[0050]

[0051] In the formula, f is the current recipe; t is the current step number; T is the maximum time step number; and s is each state in the state space.

[0052] Next, based on the iterative optimization requirements of the formula, the action space A is defined, which includes five core actions: adding new raw materials, adjusting the raw material ratio upwards, adjusting the raw material ratio downwards, deleting raw materials, and making no changes, comprehensively covering the core operations of formula optimization.

[0053] When adding new raw materials, the used raw materials are not added again, and the ratio is randomly selected between the maximum and minimum values ​​of the corresponding raw material column in the dataset.

[0054] Adjusting the raw material ratio upwards involves randomly selecting a value between the maximum value of the corresponding raw material column in the dataset and the current value.

[0055] Adjusting the raw material ratio downwards involves randomly selecting a value between the minimum value and the current value in the corresponding raw material column of the dataset.

[0056] Deleting an ingredient removes a specific ingredient from the current recipe;

[0057] Making no changes means keeping the current formula unchanged;

[0058] Then, based on the deterministic characteristics of action execution, the state transition probability {P} is defined. sa}, which is set to 1, means that after a certain action is performed, the state transition result is uniquely determined, thus avoiding the interference of uncertainty in the decision-making process.

[0059] Finally, a reward function R is defined based on the matching target of recipe performance: First, using the collected recipe dataset, a regression prediction model is trained for each performance using the random forest algorithm (learning rate 0.05, maximum tree depth 4, number of trees 500). Then, the single performance reward is calculated by the relative error between the predicted performance of the generated recipe and the target performance. The comprehensive reward is obtained by combining the weight vector and reward vector of the multi-target performance, ensuring that the reward can accurately reflect the fit between the recipe and the target performance, and providing effective feedback for subsequent model training.

[0060] The formula for calculating the reward is as follows:

[0061]

[0062] In the formula, Here, T is the discount factor; T is the maximum time step; t is the current step number. This is a comprehensive reward for multi-objective performance.

[0063]

[0064]

[0065] In the formula, This represents the weight vector for multi-objective performance. This represents the reward vector for multi-objective performance. This represents the output value of the i-th performance prediction model; It is the target value for the i-th performance.

[0066] By defining the above four key elements in an orderly manner, a complete Markov decision process for formula generation is constructed, clarifying the correspondence between "state-action-reward" and providing a clear decision logic for model learning.

[0067] In this embodiment, in step S2, a graph Q learning network model is constructed based on the Markov decision process; the graph Q learning network model is trained using the double Q learning algorithm and policy function to obtain the trained graph Q learning network model; state transition and action selection processing are performed on the trained graph Q learning network model to generate a polymer formulation.

[0068] Specifically, such as Figure 2 As shown, a graph Q learning network model is first constructed, employing a dual-input head structure to comprehensively capture formulation-related features: the model consists of an A input head, a B input head, a graph convolutional layer, a fully connected hidden layer, and an output layer. The A input head receives the graph representation encoding of the formulation's raw material molecules, including N-dimensional data. atom ×F atom The atomic eigenvector matrix with dimension N atom ×F bondThe key feature vector matrix has a dimension of N. atom ×N atom The adjacency matrix, where N atom F represents the total number of atoms. atom For atomic feature dimensions; F bond For key features, deep molecular structure features are extracted through three layers of graph convolutional layers and one layer of fully connected hidden layers. The B input head receives a 100-dimensional formula vector (filled with 0s if less than 100 dimensions, and the first 100 key raw materials if more than 100 dimensions) and a 1-dimensional episode remaining steps, and extracts the ratio and iteration progress features through two layers of fully connected hidden layers. After concatenating the hidden layer representations of the two input heads, the mixture is processed through two layers of fully connected hidden layers, and finally outputs the Q values ​​corresponding to the five actions, providing a quantitative basis for action selection.

[0069] The model was then trained using a double-Q learning algorithm with a parameter update weight of 0.8 to avoid overestimation of the Q-value and improve training stability. The training strategy employed an ε-greedy approach, where ε was reduced from 1 to 0.05 in a piecewise linear manner to balance the model's exploratory and exploitative aspects. The training parameters were set as follows: learning rate 0.003, optimizer Adam, number of episodes 2000, discount factor 0.9, and maximum time step T=15. Training proceeded according to the logic of "episode loop - state iteration - action selection - reward calculation - parameter update" until the model converged.

[0070] Finally, the formula is generated: the target performance is input into the trained model, the model starts from the initial state (empty formula, t=0), selects the optimal action based on the Q value of the current state, and updates the state according to the deterministic state transition rule after the action is executed. This process is repeated until the maximum time step T is reached. The final output formula is the optimal polymer formula that meets the multi-target performance requirements, realizing the accurate back-inference of "target performance-formula".

[0071] In one possible embodiment, a specific recipe generation example is provided as follows:

[0072] I. Basic Settings of the Implementation Example

[0073] Formula generation target performance and core parameters:

[0074] This embodiment targets a general-purpose polymer composite material, setting the target properties as follows: flexural strength ≥ 40 MPa, impact strength ≥ 5.0 kJ / m², and heat distortion temperature ≥ 120℃. The multi-target performance weight vector is ω = [0.4, 0.3, 0.3].

[0075] The core parameters are configured as follows: maximum time step T=15, ε in the greedy strategy is currently 0.08, discount factor γ=0.9, and a random forest prediction model (learning rate 0.05, maximum tree depth 4, number of trees 500) is used for performance prediction and reward calculation.

[0076] Initial state definition:

[0077] In a certain episode, the current state at time t=3 is s3=(Recipe: ["Raw Material 3":36,"Raw Material 12":12], Steps: 3). According to the random forest model, the recipe's performance is: flexural strength 35MPa, impact strength 4.2kJ / m², heat distortion temperature 105℃, and overall reward value R=0.65 (not meeting the target threshold R≥0.88), requiring further iterative optimization.

[0078] II. Formula Generation Iterative Process

[0079] M1, t=3: Initial proportioning adjustment decision

[0080] The current state s3 corresponds to five selectable state recipes in the action space output: {["Ingredient 3":36,"Ingredient 12":12,"Ingredient 16":6] (Add new ingredient), ["Ingredient 3":36,"Ingredient 12":15] (Adjust ratio upwards), ["Ingredient 3":36,"Ingredient 12":7] (Adjust ratio downwards), ["Ingredient 3":36] (Delete ingredient), ["Ingredient 3":36,"Ingredient 12":12] (Do not change)}

[0081] The model calculated the Q values ​​for each optional formulation as follows: 1.72, 1.58, 2.05, 1.31, and 1.16. Based on the ε-greedy strategy, the action with the highest Q value, "adjusting the raw material ratio downwards," was selected, determining the formulation at time t+1=4 as ["Raw Material 3":36,"Raw Material 12":7]. Prediction indicates that this formulation increases the overall reward value to 0.71, with slight improvements in impact strength and heat distortion temperature.

[0082] M2, t=4: Secondary proportioning optimization decision

[0083] At time t=4, state s4 = (Recipe: ["Ingredient 3":36,"Ingredient 12":7], Steps: 4), the action space output has 5 selectable recipe states: {["Ingredient 3":36,"Ingredient 12":7,"Ingredient 51":0.4] (Add new ingredient), ["Ingredient 3":36,"Ingredient 12":20] (Adjust ratio upwards), ["Ingredient 3":36,"Ingredient 12":4] (Adjust ratio downwards), ["Ingredient 3":36] (Delete ingredient), ["Ingredient 3":36,"Ingredient 12":7] (Do not change)}

[0084] The calculated Q values ​​for each optional formulation are: 1.93, 1.64, 2.18, 1.29, and 1.07. Continuing with the "adjust raw material ratio downwards" action with the highest Q value, the formulation at time t+2=5 is determined to be ["Raw Material 3":36,"Raw Material 12":4]. The predicted comprehensive reward value further increases to 0.78, and the bending strength is significantly improved.

[0085] M3, t=5 to t=14: Iterative optimization and raw material replenishment

[0086] At time t=5: The state recipe is ["Ingredient 3":36,"Ingredient 12":4]. The action with the highest Q value is "Add New Ingredient". Ingredient 28 (ratio 5) is selected, resulting in recipe ["Ingredient 3":36,"Ingredient 12":4,"Ingredient 28":5]. The overall reward value is 0.83.

[0087] At time t=6: The ratio of raw material 28 is adjusted upwards to 8, and the formula becomes ["Raw material 3":36,"Raw material 12":4,"Raw material 28":8], with a comprehensive reward value of 0.86;

[0088] At time t=7: Add ingredient 41 (ratio 3), update the recipe to ["Ingredient 3":36,"Ingredient 12":4,"Ingredient 28":8,"Ingredient 41":3], the overall reward value reaches 0.91 (meets the target threshold);

[0089] From t=8 to t=14: The model outputs the highest Q value for the "do no change" action multiple times. The exploration mechanism is only triggered at t=10. After briefly adding ingredient 19 (ratio 2), the ingredient is deleted due to the decrease in reward value (0.87). The recipe is restored to its optimal state and remains stable.

[0090] M4, Termination Status: Maximum time step reached.

[0091] The formulation generation process for this episode terminates at time t=15 (the maximum time step T=15 is reached). The final polymer formulation is: ["Raw Material 3":36,"Raw Material 12":4,"Raw Material 28":8,"Raw Material 41":3].

[0092] III. Verification of Results from Examples

[0093] The final formula was prepared and its performance was tested. The measured data were: flexural strength 42.3 MPa, impact strength 5.3 kJ / m², and heat distortion temperature 125℃. The overall performance fully met the target requirements, and the relative error between the measured and predicted values ​​was less than 4%, which fully verified the accuracy and effectiveness of the present invention in formula generation.

[0094] The application scenarios of this invention are as follows:

[0095] In the scenario of customizing general plastic formulations, this invention can efficiently back-calculate the optimal raw material combination and ratio based on the user-specified target performance such as tensile strength and weather resistance, thereby reducing trial and error costs.

[0096] In the development of high-end engineering plastic formulations, this invention can accurately match stringent performance requirements such as impact resistance, high temperature resistance, and corrosion resistance, generating high-performance formulations suitable for aerospace, automotive manufacturing, and other fields.

[0097] In the context of environmentally friendly polymer material formulation design, this invention can combine green indicators such as biodegradability and low VOC emissions to quickly iterate and develop formulation solutions that balance performance and environmental protection requirements.

[0098] In the scenario of small-batch, multi-batch production of polymer formulations, this invention does not require complex pre-training, flexibly responds to different performance requirements, and enables rapid switching and precise generation of formulations.

[0099] In the exploration of novel polymer material formulations, this invention utilizes an ε-greedy strategy to uncover potential raw material combinations and ratios, providing innovative directions for the research and development of novel materials.

[0100] In the context of optimizing the cost of polymer materials, this invention can prioritize the selection of low-cost raw materials and optimize their proportions to generate a formula with the best cost performance, while meeting the target performance requirements.

[0101] It should be noted that the method of this disclosure embodiment can be executed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the method of this disclosure embodiment, and the multiple devices will interact with each other to complete the method described.

[0102] It should be noted that the above description describes some embodiments of this disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the above embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0103] Example 2

[0104] See Figure 3 Embodiment 2 of the present invention also provides a polymer formulation analysis device based on a graph Q learning network, comprising:

[0105] Markov Decision Process Construction Module 001 is used to construct a Markov Decision Process for recipe generation by defining the state space, action space, state transition probability, and reward function.

[0106] The Graph Q Learning Network Model Training and Processing Module 002 is used to construct a Graph Q Learning Network Model based on the Markov Decision Process; train the Graph Q Learning Network Model using the Double Q Learning Algorithm and Policy Function to obtain the trained Graph Q Learning Network Model; and perform state transition and action selection processing on the trained Graph Q Learning Network Model to generate a polymer formulation.

[0107] In this embodiment, in the Markov decision process construction module 001, during the process of defining the state space, the action space, the state transition probability, and the reward function, the state space is defined based on the polymer formulation dataset; the action space is defined based on the formulation composition and adjustment requirements of the state space; the state transition probability is defined according to the deterministic execution rules of the action space; and the reward function is defined based on the formulation performance matching target.

[0108] In this embodiment, in the Markov decision process construction module 001, when defining the state space, each state in the state space is a (f,t) tuple; initially, f is empty, t=0, and the set of final states is:

[0109]

[0110] In the formula, f is the current recipe; t is the current step number; T is the maximum number of time steps; and s is each state in the state space.

[0111] When defining the action space, the action space includes five actions: adding new raw materials, adjusting the raw material ratio upwards, adjusting the raw material ratio downwards, deleting raw materials, and making no changes.

[0112] In this embodiment, in the Markov decision process construction module 001, when defining the state transition probability, the state transition probability is specified as 1, so that the state transition has determinism;

[0113] When defining the reward function, the reward is calculated based on the relative error between the predicted performance and the target performance of the generated recipe; the formula for calculating the reward is:

[0114]

[0115] In the formula, Here, T is the discount factor; T is the maximum time step; t is the current step number. This is a comprehensive reward for multi-objective performance.

[0116] In this embodiment, the graph Q-learning network model training and processing module 002 has a two-input-head structure, including: an A input head, a B input head, a graph convolutional layer, a fully connected hidden layer, and an output layer; the A input head is a molecular graph representation of the formulation raw materials; the B input head is the formulation vector and the remaining steps of the episode; the output of the graph Q-learning network model is the Q-value; during the training of the graph Q-learning network model, the policy function is... Greedy strategy.

[0117] It should be noted that the information interaction and execution process between the modules of the above system are based on the same concept as the method embodiment in Embodiment 1 of this application, and the resulting technical effects are the same as those in the method embodiment of this application. For details, please refer to the description in the method embodiment shown above in this application, and it will not be repeated here.

[0118] Example 3

[0119] Embodiment 3 of the present invention provides a non-transitory computer-readable storage medium storing program code for a polymer formulation analysis method based on a graph Q learning network. The program code includes instructions for executing the polymer formulation analysis method based on a graph Q learning network according to Embodiment 1 or any possible implementation thereof.

[0120] Computer-readable storage media can be any available medium that a computer can access, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

[0121] Example 4

[0122] Embodiment 4 of the present invention provides an electronic device, including: a memory and a processor;

[0123] The processor and the memory communicate with each other via a bus; the memory stores program instructions that can be executed by the processor, and the processor can execute the polymer formulation analysis method based on graph Q learning network according to Embodiment 1 or any possible implementation thereof by calling the program instructions.

[0124] Specifically, a processor can be implemented in hardware or software. When implemented in hardware, the processor can be a logic circuit, an integrated circuit, etc. When implemented in software, the processor can be a general-purpose processor that reads software code stored in memory. This memory can be integrated into the processor or located outside the processor and exist independently.

[0125] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable system. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0126] It is obvious to those skilled in the art that the modules or steps of the present invention described above can be implemented using general-purpose computing systems. They can be centralized on a single computing system or distributed across a network of multiple computing systems. Optionally, they can be implemented using program code executable by a computing system, thereby storing them in a storage system for execution by the computing system. In some cases, the steps shown or described can be performed in a different order than those presented herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0127] Although the present invention has been described in detail above with general descriptions and specific embodiments, modifications or improvements can be made to it, which will be obvious to those skilled in the art. Therefore, all such modifications or improvements made without departing from the spirit of the present invention fall within the scope of protection claimed by the present invention.

Claims

1. A polymer formulation analysis method based on graph Q learning networks, characterized in that, include: By defining the state space, action space, state transition probability, and reward function, a Markov decision process for recipe generation is constructed. Based on the Markov decision process, a graph Q-learning network model is constructed; the graph Q-learning network model is trained using a double Q-learning algorithm and a policy function to obtain a trained graph Q-learning network model; state transition and action selection are performed on the trained graph Q-learning network model to generate a polymer formulation.

2. The polymer formulation analysis method based on graph Q learning network according to claim 1, characterized in that, In defining the state space, the action space, the state transition probability, and the reward function, the state space is defined based on the polymer formulation dataset; The action space is defined based on the recipe composition and adjustment requirements of the state space; the state transition probability is defined according to the deterministic execution rules of the action space. The reward function is defined based on the goal of matching formula performance.

3. The polymer formulation analysis method based on graph Q learning network according to claim 2, characterized in that, When defining the state space, each state in the state space is a (f, t) tuple; initially, f is empty, t = 0, and the final set of states is: ； In the formula, f is the current recipe; t is the current step number; and T is the maximum number of time steps. s represents each state in the state space; When defining the action space, the action space includes five actions: adding new raw materials, adjusting the raw material ratio upwards, adjusting the raw material ratio downwards, deleting raw materials, and making no changes.

4. The polymer formulation analysis method based on graph Q learning network according to claim 3, characterized in that, When defining the state transition probability, the state transition probability is specified as 1 to make the state transition deterministic; When defining the reward function, the reward is calculated based on the relative error between the predicted performance and the target performance of the generated recipe; the formula for calculating the reward is: ； In the formula, Here, T is the discount factor; T is the maximum time step; t is the current step number. This is a comprehensive reward for multi-objective performance.

5. The polymer formulation analysis method based on graph Q learning network according to claim 4, characterized in that, The graph Q-learning network model has a two-input head structure, including: an A input head, a B input head, a graph convolutional layer, a fully connected hidden layer, and an output layer; the A input head is a molecular graph representation of the formulation ingredients; the B input head is the formulation vector and the remaining steps of the episode; the output of the graph Q-learning network model is the Q-value; during the training of the graph Q-learning network model, the policy function is... Greedy strategy.

6. A polymer formulation analysis device based on a graph Q learning network, employing the polymer formulation analysis method based on a graph Q learning network as described in any one of claims 1-5, characterized in that, include: The Markov Decision Process Construction Module is used to construct a recipe-generated Markov Decision Process by defining the state space, action space, state transition probabilities, and reward function. The Graph Q Learning Network Model Training and Processing Module is used to construct a Graph Q Learning Network Model based on the Markov Decision Process; train the Graph Q Learning Network Model using the Double Q Learning Algorithm and Policy Function to obtain the trained Graph Q Learning Network Model; and perform state transition and action selection processing on the trained Graph Q Learning Network Model to generate a polymer formulation.

7. The polymer formulation analysis device based on graph Q learning network according to claim 6, characterized in that, In the Markov decision process construction module, the state space is defined based on the polymer formulation dataset during the process of defining the state space, the action space, the state transition probability, and the reward function. The action space is defined based on the recipe composition and adjustment requirements of the state space; the state transition probability is defined according to the deterministic execution rules of the action space. The reward function is defined based on the goal of matching formula performance.

8. The polymer formulation analysis device based on graph Q learning network according to claim 7, characterized in that, In the Markov decision process construction module, when defining the state space, each state in the state space is a (f, t) tuple; initially, f is empty, t=0, and the final state set is: ； In the formula, f is the current recipe; t is the current step number; and T is the maximum number of time steps. s represents each state in the state space; When defining the action space, the action space includes five actions: adding new raw materials, adjusting the raw material ratio upwards, adjusting the raw material ratio downwards, deleting raw materials, and making no changes.

9. The polymer formulation analysis device based on graph Q learning network according to claim 8, characterized in that, In the Markov decision process construction module, when defining the state transition probability, the state transition probability is specified as 1, so that the state transition is deterministic. When defining the reward function, the reward is calculated based on the relative error between the predicted performance and the target performance of the generated recipe; the formula for calculating the reward is: ； In the formula, Here, T is the discount factor; T is the maximum time step; t is the current step number. This is a comprehensive reward for multi-objective performance.

10. The polymer formulation analysis device based on graph Q learning network according to claim 9, characterized in that, In the graph Q-learning network model training and processing module, the graph Q-learning network model has a two-input head structure, including: an A input head, a B input head, a graph convolutional layer, a fully connected hidden layer, and an output layer; the A input head is a molecular graph representation of the formulation raw materials; the B input head is the formulation vector and the remaining steps of the episode; the output of the graph Q-learning network model is the Q-value; during the training of the graph Q-learning network model, the policy function is... Greedy strategy.

Images