Molecular sieve adaptive synthesis method based on multi-modal embodied perception and process reward model
By combining multimodal sensing with a process reward model, molecular sieve synthesis parameters are adjusted in real time, solving the problems of blind operation and feedback lag in existing technologies, and achieving efficient, safe and high-quality control of molecular sieve synthesis.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- EAST CHINA NORMAL UNIV
- Filing Date
- 2026-03-19
- Publication Date
- 2026-06-19
AI Technical Summary
Existing automated molecular sieve synthesis systems lack process awareness, leading to blind operation and sparse feedback, resulting in high trial-and-error costs and safety hazards, and failing to effectively optimize synthesis parameters.
A multimodal embodied perception and process reward model is adopted, which uses visual and force sensors to capture the characteristics of the reaction process, and combines a reinforcement learning policy network for real-time adjustment to provide dense feedback signals and achieve closed-loop control.
This improved the success rate and repeatability of molecular sieve synthesis, shortened the R&D cycle of novel porous materials, and ensured the safety of the synthesis process and product quality.
Smart Images
Figure CN122245459A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of AI for Science and intelligent manufacturing, specifically to an adaptive synthesis method for molecular sieves based on a multimodal embodied perception and process reward model. Background Technology
[0002] Molecular sieves (such as zeolites, metal-organic frameworks (MOFs), and covalent organic frameworks (COFs)) are a class of porous materials with regular pore structures, widely used in catalysis, adsorption separation, and ion exchange. Traditionally, molecular sieve synthesis mainly employs hydrothermal or solvothermal methods, involving complex sol-gel transitions, precursor depolymerization and rearrangement, nucleation, and crystal growth stages. This is a typical high-dimensional, nonlinear, and strongly coupled physicochemical process. The crystallinity, purity, and morphology of the final product are highly sensitive to synthesis parameters (such as feed ratio, aging time, temperature profile, and shear stirring rate). For example, in the synthesis of metastable molecular sieves, even small fluctuations in the stirring rate can alter the fluid shear field, inducing competitive growth of non-target crystalline phases (impurities), leading to experimental failure.
[0003] With the advancement of the Materials Genome Initiative, automated synthesis technologies based on high-throughput experiments (HTEs) are becoming increasingly widespread. However, most existing automated synthesis systems operate under "open-loop control" or simple "script-execution" modes, exhibiting the following significant technical bottlenecks:
[0004] "Blind operation" lacking process awareness: Existing pipetting workstations or automated reactors can only perform preset mechanical actions (such as timed feeding and constant-speed stirring), lacking the ability to perceive the microscopic state of the reaction system in situ. Experienced chemists often rely on vision (observing opalescence, Tyndall effect) and touch (sensing viscosity feedback caused by changes in stirring resistance) to judge the degree of gelation or the induction period in experiments. Existing equipment cannot capture these key physical changes, resulting in the system being unable to intervene in a timely manner like a human when abnormalities occur in the reaction (such as local aggregation, phase separation, precipitation agglomeration), and mechanically continuing to execute the incorrect process.
[0005] The high trial-and-error costs caused by sparse feedback: Molecular sieve synthesis is characterized by significant long cycles and delayed feedback. A complete hydrothermal synthesis often takes several days or even weeks, and feedback on product quality can only be obtained after the reaction has finished and after tedious post-processing and offline characterization. This "sparse reward" mechanism leads to extremely low iterative efficiency of optimization algorithms. During the reaction period of several days, the system remains in a "black box" state, unable to predict the final result based on intermediate states, resulting in a significant waste of time and reagents on ineffective experimental pathways.
[0006] In summary, existing technologies suffer from the following problems: 1) They rely heavily on fixed scripts and cannot detect abnormal reactions; 2) They are "blind operations" lacking process awareness; 3) They suffer from "feedback lag" and "reward sparsity," requiring several days to complete the reaction and perform XRD characterization before feedback can be obtained, resulting in high trial-and-error costs due to sparse feedback; 4) Existing automated equipment is often helpless or causes safety hazards when facing complex coupled environments.
[0007] Given the aforementioned issues, there is an urgent need to introduce embodied AI technology, utilizing multimodal sensors (vision, force, etc.) to endow synthetic equipment with perceptual capabilities similar to those of human scientists. Simultaneously, to address the feedback lag problem, the concept of a reward model from reinforcement learning is introduced to establish a mapping relationship between reaction process characteristics and final product performance, providing dense process rewards, thereby achieving closed-loop adaptive control of the entire molecular sieve synthesis process. Summary of the Invention
[0008] The purpose of this invention is to address the shortcomings of existing technologies by proposing an adaptive synthesis method for molecular sieves based on a multimodal embodied perception and process reward model. This method employs a robotic arm equipped with visual and force sensors as the execution subject, constructing a multimodal state space to digitally characterize the chemical reaction state. The process reward model maps long-term experimental results back to real-time operational steps, providing dense feedback signals. Based on these signals, the system uses a reinforcement learning strategy network to adjust synthesis parameters (such as stirring and temperature control) in real time, and utilizes a safety mask to prevent dangerous operations. This closed-loop control method based on a reward model can capture and correct abnormal trends in the liquid / gel phase before crystallization, simulating and surpassing the "intuition" of human chemists. It effectively extracts high-order reaction characteristics, not only improving the success rate and reproducibility of molecular sieve synthesis but also accelerating the discovery of formulations for novel porous materials. The method is advanced, effective, and has promising application prospects.
[0009] The objective of this invention is achieved as follows: an adaptive synthesis method for molecular sieves based on a multimodal embodied perception and process reward model. The method is characterized by first utilizing the end-sensor of an embodied intelligent device to extract multimodal physical features during the synthesis process, and constructing a reaction state space through a time-series encoder. To address the feedback lag problem in molecular sieve synthesis, historical experimental trajectories and final characterization results are trained into a process reward model to simulate the long-term value assessment of the current microscopic operation from an "omniscient perspective," while incorporating chemical expert knowledge constraints. Next, the current state vector is input into a reinforcement learning policy network to generate an action probability distribution. Then, a safety constraint mask mechanism is used to filter out dangerous actions that violate chemical safety regulations. Finally, the actuator executes the preferred action and updates the environmental state, forming a closed-loop control until the synthesis is completed. Specifically, this invention includes the following steps:
[0010] 1) Data is collected using a multimodal sensing module and input into the encoder to extract the physical feature representation of the reaction process.
[0011] 1.1: The six-dimensional force / torque sensor at the end of the robotic arm is used to record the force feedback data stream during the stirring process at a high frequency sampling rate. At the same time, vision sensors (RGB-D camera and thermal imager) are used to collect images of the liquid surface and temperature field distribution inside the reactor.
[0012] 1.2: The collected torque data stream is input into a one-dimensional convolutional neural network (1D-CNN) or a long short-term memory network (LSTM) to extract rheological features, and the image data is input into a convolutional neural network (CNN) to extract macroscopic uniformity features. The two are concatenated to obtain the multimodal state vector of the current time step.
[0013] 2) Construct and utilize a process reward model to generate dense feedback signals.
[0014] 2.1: Based on historical experimental databases, construct a process reward prediction model based on the Transformer architecture;
[0015] 2.2: The process reward model is used to evaluate the process stability and predict the results of the current multimodal state vector generated in step 1. The process stability evaluation uses the process reward model to calculate the entropy values of the current rheological and visual texture features. If the entropy value is too high, it indicates that the system is chaotic or phase separation, and a negative process stability reward is output, simulating the expert's intuitive judgment of the gel quality. The result prediction evaluation uses the process reward model to analyze the similarity between the current state and the historical high success rate trajectory using an attention mechanism, predicts the contribution of the current operation to the crystallinity of the final product, and outputs a positive or negative prediction reward, thereby mapping the long-term experimental feedback into a real-time dense reward signal.
[0016] 3) By integrating the knowledge of chemical experts, a knowledge-enhanced state space and constraint boundary are obtained.
[0017] 4) Encode and generate adaptive operation instructions using a reinforcement learning policy network.
[0018] 4.1: Input the multimodal state vector with reward signal into the policy network of the Actor-Critic architecture;
[0019] 4.2: Construct a neural network capable of outputting a continuous action space.
[0020] 4.2.1: Utilize the backbone layer of the policy network; integrate current state information and reward feedback to calculate the probability distribution of the current optimal policy (such as the mean and variance of a Gaussian distribution).
[0021] 4.2.2: Motion Mapping Unit: Maps the abstract features output by the neural network to specific robotic arm control parameters, including stirring speed (rpm), feeding rate (ml / min), stirring depth (Z-axis position), and temperature change rate (°C / min).
[0022] 4.2.3: Critic: Estimates the value of the current policy in real time to guide the gradient update of network parameters and ensure that the policy converges in the direction of maximizing long-term cumulative reward;
[0023] 4.3: Utilize the policy network to output a preliminary set of action instructions.
[0024] 5) Use safety constraint masking to cover up actions that do not comply with chemical regulations.
[0025] 6) Utilize the actuator in conjunction with environmental feedback to perform high-confidence actions behind the mask and update the status.
[0026] 7) Apply linear transformation and threshold judgment to determine the reaction endpoint or abnormal termination.
[0027] Compared with the prior art, the present invention has the following beneficial technical effects and significant technical progress:
[0028] 1) This invention transforms the difficult-to-quantify chemical "feel" into a calculable high-dimensional state space, realizing the transparency of the "black box" state in the molecular sieve synthesis process.
[0029] 2) This invention uses a torque sensor at the end of a robotic arm to analyze rheological characteristics in real time, and a vision sensor to capture macroscopic uniformity. It simulates and digitizes the "intuition" of human chemists in observing turbidity and sensing stirring resistance. This multimodal perception capability enables the system to make precise interventions at the millisecond level at the critical induction period of gelation and nucleation, which significantly improves the robustness to nonlinear fluctuations of precursors and effectively avoids synthesis failures caused by batch differences in raw materials.
[0030] 3) The Process Reward Model (PRM) proposed in this invention effectively solves the fundamental problems of "feedback lag" and "reward sparsity" in the hydrothermal synthesis of molecular sieves. Utilizing a reward model trained on historical trajectories, a mapping relationship between "microscopic operations and final crystal quality" is established, providing real-time dense reward signals for current stirring, feeding, and other actions. This mechanism allows reinforcement learning algorithms to predict product trends and dynamically adjust strategies before the reaction is complete, thereby rapidly converging to the optimal synthesis path without requiring extensive exhaustive experiments, significantly shortening the formulation development cycle of novel porous materials.
[0031] 4) The present invention utilizes a reinforcement learning strategy network in conjunction with a security mask mechanism to achieve a leap from "open-loop automation" to "secure closed-loop intelligence".
[0032] 5) The strategy network of this invention can iteratively optimize the stirring rate and temperature control curve in real time based on the reward signal, while automatically filtering out dangerous actions that violate chemical regulations (such as operations with the risk of boiling over) using a safety mask. This not only ensures that the molecular sieve products have higher crystallinity, purity, and batch consistency, but also ensures operational safety in unmanned laboratory scenarios, and has extremely high industrial application value. Attached Figure Description
[0033] Figure 1 This is a flowchart of the present invention;
[0034] Figure 2 is a schematic diagram of the specific operation of Example 1. Detailed Implementation
[0035] To effectively capture the nonlinear physicochemical evolution during molecular sieve synthesis and address the feedback lag problem in traditional hydrothermal synthesis, this invention proposes a state-space construction method based on multimodal embodied perception. This method simulates the interactive process of a human chemist's "visual observation + tactile perception," and utilizes a Process Reward Model (PRM) based on historical experimental trajectories to enhance the system's ability to predict the synthesis endpoint. Then, an Actor-Critic-based reinforcement learning policy network is designed to process the multimodal state vectors, successively evaluating the process stability of the current reaction state and the expected product quality, performing millisecond-level operational strategy optimization. Finally, a safety constraint mask mechanism is used to filter high-risk actions, and an actuator performs physical intervention, updating the input state to achieve real-time closed-loop control of the molecular sieve crystallization path.
[0036] See Figure 1 The present invention specifically includes the following steps:
[0037] 1) Collect multi-source physical signals using a embodied intelligent terminal, and input them into a multimodal encoder to extract response state feature representations.
[0038] 1.1: The force / torque sensor (F / T Sensor) at the end of the robotic arm is used to collect force feedback data streams during the stirring process at high frequency (e.g., 100Hz). At the same time, the in-situ vision sensor (RGB-D camera) and thermal imager are used to obtain the liquid surface texture and temperature field distribution in the reactor.
[0039] 1.2: The collected torque time series is input into a Long Short-Term Memory (LSTM) network or a one-dimensional convolutional network (1D-CNN) to extract rheological features (such as viscosity abrupt change points and shear thinning rate). The image data is input into a convolutional neural network (CNN) to extract macroscopic uniformity features. The images are then concatenated to obtain the original physical feature representation of the current time step.
[0040] 2) Constructing an enhanced state space that integrates prior chemical knowledge
[0041] 2.1: The physical characteristics are verified and normalized using an externally pre-set chemical expert knowledge base (including the formulation boundaries, temperature limits, and pH thresholds of the target molecular sieve).
[0042] 2.2: State encoding is performed based on time series and causal dependencies. A trajectory vector is constructed from the reaction initiation point to the current time step to obtain the time-enhanced reaction state space, as detailed below:
[0043] 2.2.1: Construct transient eigenvectors to model the current microstate (such as the current stirring resistance torque and instantaneous temperature gradient) to simulate a chemist's judgment on the "current intensity of the reaction";
[0044] 2.2.2: Construct trend feature vectors to model the changes in the first and second derivatives of reaction parameters over time (such as viscosity growth rate and heating rate) to simulate chemists' predictions of "gelation trend" or "precursor to boiling".
[0045] 3) Utilize the process reward model to evaluate the state and generate dense feedback signals.
[0046] 4) Encode and generate adaptive operation instructions using a reinforcement learning policy network.
[0047] 4.1: By multiplying (or inputting) the multimodal state vector with reward signal with the backbone of the current policy network, the physical state is mapped to high-dimensional policy features;
[0048] 4.2: Construct a neural network (Actor-Critic architecture) capable of outputting the probability distribution of continuous actions, as detailed below:
[0049] 4.2.1: Using the value assessment network (Critic) to aggregate historical trajectory information based on the state characteristics of the current layer, predict the probability of obtaining high-quality crystals (high crystallinity, high specific surface area) in the future, and output the state value function V(s);
[0050] 4.2.2: Action Generation Unit (Actor): Utilizing a fully connected layer (MLP) structure, it outputs Gaussian distribution parameters (mean and variance) of the action space based on the current state characteristics and value evaluation results.
[0051] 4.2.3: Parameter Mapping Unit: Maps the generated abstract motion parameters to specific actuator control commands, including stirring speed (rpm), temperature change rate (°C / min), and feed pump flow rate (ml / min);
[0052] 4.2.4: Policy Optimization: Using the Proximal Policy Optimization (PPO) algorithm, the cumulative reward given by the process reward model is maximized, and the network parameters are iteratively optimized to make the output action tend to the optimal synthesis path;
[0053] 4.3: Generate a set of compound action instructions for fluid dynamics control (stirring) and thermodynamics control (heating) in sequence.
[0054] 5) Use safety constraint masking to cover up dangerous actions that violate chemical regulations.
[0055] 6) Utilize the actuator in conjunction with environmental feedback to perform high-confidence actions behind the mask, and use sensors to update the state space.
[0056] 7) Apply threshold judgment and linear transformation to determine the reaction endpoint or abnormal termination, and complete a single closed-loop control.
[0057] The present invention will be further described in detail below using a specific example of the hydrothermal synthesis process of ZSM-5 molecular sieve.
[0058] Example 1
[0059] See Figure 2 The system first controls a robotic arm to mix the silicon source (TEOS) and template agent (TPAOH). A torque sensor detects that the solution viscosity does not increase significantly over time (an anomaly is indicated by the rheological characteristics extracted by LSTM), and the Process Reward Model (PRM) outputs a negative reward signal. Upon receiving this signal, the policy network searches for an optimal strategy in the action space and outputs the instruction to "increase the stirring speed to 500 rpm and start auxiliary heating." This instruction is executed after being verified by a safety mask (confirming no overheating). Subsequently, a vision sensor detects that the solution gradually exhibits the Tyndall effect, and the torque curve shows the expected S-shaped growth. The PRM model then switches to outputting a positive reward, and the system determines that the gelation process has returned to normal, proceeding to the aging stage. Finally, a linear transformation is used to calculate the cumulative reward score. When the score exceeds a preset threshold, heating is automatically stopped and a cooling cycle is initiated, yielding a highly crystalline ZSM-5 product.
[0060] The above description is only a preferred embodiment of the present invention. Modifications may be made within the scope defined by the claims of the present invention, but all such modifications shall fall within the protection scope of the present invention.
Claims
1. An adaptive synthesis method for molecular sieves based on a multimodal embodied perception and process reward model, characterized in that, The method includes the following steps: 1) A robotic arm equipped with multimodal sensors is used to perform molecular sieve precursor mixing operations, and the physical characteristics of the reaction system are extracted in real time; 2) The visual image, torque feedback, and environmental parameters are encoded using a multimodal encoder to obtain the reaction state vector at the current time step; 3) By integrating a pre-defined chemical expert knowledge base, a knowledge-enhanced reaction state space is obtained; 4) Utilize a process reward model to evaluate the contribution of the current state to the quality of the final product and generate dense reward signals; 5) Generate motion correction instructions for the robotic arm based on the reward signal using a policy network; 6) Utilize a safety constraint masking mechanism to conceal the probability of actions that do not comply with chemical safety regulations; 7) Utilize the actuator to perform the preferred actions and update the environmental state to form a real-time closed-loop control until the synthesis reaction ends.
2. The adaptive synthesis method for molecular sieves based on a multimodal embodied perception and process reward model according to claim 1, characterized in that, Step 1) specifically includes: 1.1: The data stream from the six-dimensional force / torque sensor at the end of the robotic arm is filtered to extract the rheological characteristics of viscosity and shear resistance during the stirring process; 1.2: In-situ visual sensors and thermal imagers are used to acquire images of the liquid surface and temperature field distribution inside the reactor, which are then combined into a multi-channel image matrix to extract macroscopic uniformity features.
3. The adaptive synthesis method for molecular sieves based on a multimodal embodied perception and process reward model according to claim 1, characterized in that, Step 2) specifically includes: 2.1: Utilize temporal convolutional networks or Transformer structures to process time-series torque data and capture abrupt changes in fluid properties; 2.2: Concatenate the image feature vector and the mechanical feature vector, embed the current time step information, and construct a multimodal state vector.
4. The adaptive synthesis method for molecular sieves based on a multimodal embodied perception and process reward model as described in claim 1, characterized in that, Step 4) specifically includes: 4.1: Construct a process reward model based on the Transformer architecture, which is trained by the historical experimental trajectory and the final XRD representation results; 4.2: Input the current state vector into the process reward model and calculate the weighted sum of the process stability reward and the product quality prediction reward; 4.3: For the hysteresis feedback problem in molecular sieve synthesis, the reward model is used to predict the long-term impact of current micro-operations on future crystallinity.
5. The adaptive synthesis method for molecular sieves based on a multimodal embodied perception and process reward model as described in claim 1, characterized in that, Step 5) specifically includes: 5.1: The near-end policy optimization algorithm is used as the core of the policy network to receive reward values and update policy parameters; 5.2: Outputs include, but are not limited to: stirring speed adjustment, feeding rate adjustment, temperature curve correction, and motion space for adjusting the spatial position of the robotic arm end effector.
6. The adaptive synthesis method for molecular sieves based on a multimodal embodied perception and process reward model according to claim 4, characterized in that, Step 4.2 specifically includes: rheological stability reward calculation and visual entropy reward calculation. The rheological stability reward calculation calculates the second derivative of the torque data. If the absolute value of the derivative exceeds a preset threshold and the predetermined aging time has not been reached, a negative reward is given to punish premature aggregation or phase separation. The visual entropy reward calculation calculates the texture entropy of the liquid surface image. If the entropy value gradually decreases and tends to stabilize, a positive reward is given to encourage system homogenization.