Method for predicting yield of reaction for preparing methanol by adding hydrogen into CO2 based on FMGTransformer model

Abstract

The invention relates to an FMGTransform model-based reaction yield prediction method for preparing methanol by adding hydrogen into CO2, which comprises the following steps of: collecting reaction data of preparing methanol by adding hydrogen into CO2, and establishing a data set which comprises classification characteristics and numerical characteristics; the classification features are subjected to Embedding coding and embedded coding, numerical features are subjected to normalization processing and coded through a numerical feature embedded layer, then the features are fused together, and CLS Token is added to serve as input of a model; an FMGTransform model is built, the FMGTransform model comprises a Transform encoder, a multi-scale convolution module, a gating attention mechanism and a multi-layer perceptron, and complex relations among reactants, catalysts and reaction conditions are captured globally and locally. According to the method, nonlinear interaction among variables such as catalyst types, temperature and pressure can be more effectively captured, and the generalization and prediction stability of the model are improved.

Description

A method for predicting the yield of CO2 hydrogenation to methanol reaction based on the FMGTransformer model Technical Field

[0001] This invention relates to the field of chemical reaction modeling and catalytic reaction yield prediction technology, specifically a method for predicting the yield of CO2 hydrogenation to methanol reaction based on the FMG Transformer (Fusion-Guided Transformer) model. Background Technology

[0002] The CO2-to-hydrogen-to-methanol reaction, as a typical artificial carbon cycle pathway, not only helps reduce greenhouse gas emissions but is also a key reaction in green energy synthesis, making it an important direction for current catalyst design and reaction pathway optimization research. Many factors influence the yield of this reaction, including catalyst type, reaction temperature, reaction pressure, GHSV, and reaction conditions such as support agents and auxiliaries. Traditional methods rely on experimental measurements of yields under different conditions, which are not only time-consuming and costly but also unsuitable for rapid evaluation of catalyst performance in the initial screening process. Existing catalytic reaction modeling mainly employs physical modeling or data-driven methods. Physical modeling is often based on microscopic methods such as density functional theory (DFT) or reaction kinetic models, but solving complex systems suffers from high computational costs and difficulty in scaling to real industrial conditions. Furthermore, due to the diverse types of variables in catalytic reaction systems (including categorical and continuous variables), high data dimensionality, and strong nonlinear relationships between variables, traditional modeling methods (such as linear regression and response surface analysis) are limited in prediction accuracy and adaptability when dealing with complex reaction systems with diverse characteristics.

[0003] In recent years, with the rapid development of machine learning and computational catalysis, we have begun to explore the use of data-driven methods to predict the yield of CO2 hydrogenation to methanol reaction, in order to quickly screen catalytic systems and guide experimental design before the experiment. Vanjari et al. [1] constructed a gradient boosting tree (GBT) model for methanol yield based on Cu-based catalysts, using reaction conditions such as temperature, pressure, and gas space-time velocity (GHSV) as input features. This study proved the feasibility of machine learning methods in predicting the yield of catalytic reactions, and the model's coefficient of determination R 2 The accuracy is approximately 0.75, which is an improvement over traditional regression models. However, this method mainly relies on continuous variable inputs and does not model discrete variables such as catalyst type and support type, thus making it difficult to comprehensively reflect the structural effects of different catalyst systems. In addition, the model uses fixed feature weights, which cannot characterize the nonlinear coupling relationship between reaction conditions such as temperature and pressure, limiting its generalization ability in diverse reaction systems.

[0004] To address the aforementioned issues, this invention proposes an FMGTransformer model based on the Transformer architecture, combining multi-scale convolution (Multi-Scale CNN) and gated attention. The multi-scale convolution module captures local and global feature correlations at different scales, reflecting the hierarchical dependencies between the microscopic (catalyst structure) and macroscopic (reaction conditions) aspects of the catalytic reaction. The gated attention mechanism dynamically weights feature importance, enabling the model to adaptively adjust feature contributions under different reaction conditions, thereby significantly improving the accuracy and generalization performance of CO2 hydrogenation to methanol yield prediction. Summary of the Invention

[0005] To address the shortcomings of traditional methods in accurately modeling and generalizing the synergistic effects of multiple variables in the CO2 hydrogenation to methanol reaction, the present invention aims to provide a method for predicting the yield of the CO2 hydrogenation to methanol reaction based on the FMGTransformer model.

[0006] This invention is achieved through the following technical solution: a method for predicting the yield of CO2 hydrogenation to methanol reaction based on the FMGTransformer model. The prediction method includes the following steps: collecting reaction data of CO2 hydrogenation to methanol under different reaction conditions and preprocessing it to establish a dataset. Each data set in the dataset is saved as: [Yield—Classification Feature—Numerical Feature], where each row represents the data of a reaction experiment, containing corresponding classification features and numerical features; embedding and encoding the classification features through a classification feature embedding layer, embedding and encoding the numerical features through a numerical feature embedding layer, and concatenating the embedded classification features and numerical features to obtain a fused feature sequence [CN]; constructing an FMGTransformer model, which includes a Transformer encoder, a multi-scale convolutional module, a gated attention mechanism, and a multilayer perceptron (MLP); inserting a learnable classification tag vector (CLS Token) at the beginning of the fused feature sequence to construct the Transformer input sequence [TCN], and feeding this input sequence into the Transformer encoder for feature modeling; and then... The feature sequence output by the encoder is fed into the first multi-scale convolution module as input to output a global feature vector; at the same time, the fused feature sequence [CN] is input into the second multi-scale convolution module to output a local feature vector; the global feature vector and the local feature vector are concatenated and input into the gating attention mechanism to dynamically adjust the feature importance and obtain the final fused feature vector g; the final fused feature vector g is input into the multilayer perceptron (MLP) and the output is the predicted methanol yield of the reaction.

[0007] Furthermore, the classification characteristics include catalyst type, support 1, support 2, and accelerator type; the numerical characteristics include metal content, covalent radius, molecular weight of support 1, molecular weight of support 2, accelerator content, SBET, temperature, pressure, GHSV, and H2 / CO2 feed ratio.

[0008] Further, the embedding encoding process is as follows: A `categorical_dict` dictionary is created to store the names and corresponding data of categorical features, forming a categorical feature list; a `numerical_dict` dictionary is created to store the names and corresponding data of numerical features, forming a numerical feature list; Categorical feature encoding: four categorical features are read from the categorical feature list; based on the feature names stored in the `categorical_dict` dictionary, each categorical feature is processed using Embedding encoding, and then mapped to a low-dimensional dense vector representation by the categorical feature embedding layer, so that the network can more effectively capture the potential relationships between categories, obtaining the embedded categorical features; Numerical feature encoding: ten numerical features are read from the numerical feature list; firstly, the numerical features are normalized, and then, according to the storage method in `numerical_dict`, the normalized features are subjected to feature-by-feature, dimension-by-dimensional affine mapping through the numerical feature embedding layer to obtain the embedded numerical features; the embedded categorical features and numerical features are concatenated and used as the input to the FMGTransformer model.

[0009] Furthermore, the Transformer encoder contains N Transformer encoding layers. Each Transformer encoding layer captures long-range dependencies between input features through a self-attention mechanism, and the output feature vector is further transformed nonlinearly through a feedforward neural network (FFN) module.

[0010] The present invention also protects a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, enables the implementation of the method described thereon.

[0011] Compared with existing technologies, the advantages of this invention are: it effectively combines the Transformer deep learning structure, multi-scale convolutional structure, and gated attention mechanism, and introduces them into the modeling and prediction task of CO2 hydrogenation to methanol reaction, solving the challenge of modeling mixed tabular data in catalytic reaction systems. By effectively handling categorical and continuous variables, this model supports data-driven optimization of reaction conditions and catalyst selection.

[0012] This invention addresses the challenge of jointly modeling both categorical features (catalyst type, support, promoter, etc.) and numerical features (temperature, pressure, GHSV, etc.) in the CO2 hydrogenation to methanol reaction. By introducing Embedding encoding and numerical feature embedding encoding, categorical and continuous variables are mapped to a unified embedding space, achieving an integrated expression of heterogeneous features and overcoming the limitations of traditional machine learning models (such as random forests and SVR) in mixed data processing.

[0013] This invention extracts local correlation patterns (such as the coupling effect of temperature and pressure, and the synergistic effect of the carrier and the promoter) and global relationships between reaction conditions through multi-scale convolution modules. It can learn the local nonlinear dependencies between features at different receptive field scales, making the model more flexible and accurate in characterizing the hierarchical relationships of complex catalytic systems. This overcomes the shortcomings of the traditional TabTransformer model, which only captures feature relationships through the global self-attention mechanism.

[0014] To address the significant differences in the importance of variables under varying experimental conditions, this invention employs a gated attention mechanism. During the Transformer encoding process, attention weights are calculated using the Softmax function. This gated attention mechanism further adjusts the weighted features based on a Sigmoid gate, enabling the model to adaptively select features under different reaction conditions, significantly improving prediction stability and interpretability. It can dynamically adjust the importance weights of each input feature and automatically identify the synergistic relationships between temperature, pressure, and catalyst structure, thus achieving higher interpretability. This mechanism is particularly suitable for catalytic reaction systems with significant inter-variable coupling, providing theoretical reference for reaction mechanism research and experimental optimization. Therefore, its enhanced feature interpretability makes it more adaptable to complex reaction systems.

[0015] The proposed FMGTransformer model maintained relatively stable prediction performance across multiple experiments, with significantly smaller performance fluctuations than comparative models, indicating better stability and generalization ability. This stability stems from the model's shared embedding and adaptive normalization in both categorical feature embedding and numerical feature normalization encoding, avoiding training imbalance caused by different feature scales. Under the same experimental dataset and training conditions, the proposed FMGTransformer model achieved a higher coefficient of determination R0. 2 It outperforms existing methods in three metrics: root mean square error (RMSE), mean absolute error (MAE), and mean square error (RMSE). Specifically, R... 2 The improvements are approximately 6%–23%, while RMSE and MAE decrease by approximately 25%–40%, indicating significant improvements in both fitting accuracy and error control. Therefore, the proposed "multi-scale convolution + gated attention" structure can more effectively capture the nonlinear interactions between variables such as catalyst type, temperature, and pressure, thereby improving the model's generalization ability and predictive stability. Attached Figure Description

[0016] Figure 1. Schematic diagram of a process for predicting the yield of CO2 hydrogenation to methanol reaction based on the FMGTransformer model of the present invention.

[0017] Figure 2. Example of a dataset.

[0018] Figure 3 shows the overall architecture of the FMGTransformer model. FMGTransformer stands for Feature-Multi-scale-Gated Transformer. Detailed Implementation

[0019] The steps of the present invention described below are intended to be described as embodiments of the present invention and are not the only forms that can be made or utilized. Other embodiments that can achieve the same function should also be included within the scope of the present invention.

[0020] The Transformer neural network is a deep learning architecture based on an attention mechanism. Initially used for natural language processing tasks, it has been widely applied in the modeling and prediction of structured data in recent years. This model can establish global dependencies between input features through a multi-head self-attention mechanism, possessing the ability to efficiently capture nonlinear interactions between variables. The TabTransformer, however, uses a uniform global attention mechanism for all features, failing to adaptively capture local nonlinear interactions (such as the coupling effect between temperature and pressure) in the CO2 hydrogenation reaction system. Furthermore, it lacks a dynamic feature weighting mechanism, making it difficult to adjust feature importance according to different reaction conditions, thus limiting model interpretability and predictive stability. For CO2 hydrogenation to methanol reaction systems involving multiple catalyst types, propellant combinations, and complex reaction conditions, traditional neural networks or statistical models struggle to simultaneously handle heterogeneous variables and accurately model their potential correlations. In contrast, compared to traditional modeling methods such as molecular dynamics simulations and response surface methodology, the FMGTransformer model of this invention does not rely on reaction mechanism assumptions, requires no high computational resources, and can accurately predict the yield of the catalytic reaction system, significantly improving modeling efficiency and reducing experimental costs.

[0021] This invention uses categorical features (catalyst type, support type, and promoter type) and numerical features (reaction temperature, pressure, GHSV, metal content, covalent radius, molecular weight of support 1, molecular weight of support 2, promoter content, SBET, H2 / CO2 ratio, etc.) as known conditions, and takes the yield of methanol in the CO2 hydrogenation reaction as the object to be predicted, constructing input features and performing modeling. The categorical and numerical features (categorical features are discrete, numerical features are continuous) are embedded and encoded separately, then concatenated to obtain a fused feature sequence [CN]. A learnable classification token vector (CLS Token) is inserted at the beginning of the fused feature sequence [CN] to construct the Transformer input sequence [TCN]. This TCN is then processed by an FMGTransformer model containing a Transformer encoder with multi-head attention, a multi-scale convolutional neural network (CNN), and a gated attention module for global and local feature extraction and fusion, finally outputting the predicted yield. This invention uses an end-to-end deep learning approach to model the nonlinear relationship between reaction conditions and yield, improving the modeling capability and prediction accuracy for complex reaction systems.

[0022] Example 1: This example describes a method for predicting the yield of CO2-to-methanol reaction based on the FMGTransformer model, including the following steps: Step 1: Collect reaction data for CO2-to-methanol reaction and establish a dataset. For the prediction target, collect data on CO2-to-methanol reaction under different reaction conditions, standardize the units of all features to ensure consistency. Store all reaction data in array format, saving each data set as: [Yield—Categorical Feature—Numerical Feature], where each row represents data from one reaction experiment, including the corresponding categorical and numerical features. This ensures efficient and convenient data processing.

[0023] Step Two: Categorical Feature Embedding Encoding and Numerical Feature Normalization Encoding. Categorical features (such as catalyst type, carrier type, and promoter type) are discrete variables. Directly using one-hot encoding would cause high-dimensional sparsity, increasing the computational cost of the model and affecting convergence. Therefore, this invention employs a label encoding combined with an embedding table. Each category is first mapped to a unique integer index, and then a low-dimensional dense vector representation is obtained through a shared embedding matrix (the categorical feature embedding layer). This method significantly reduces feature dimensionality and memory overhead while maintaining learnable relationships between categories. It enables the model to automatically learn the semantic distance and potential synergistic effects between catalyst types, carriers, and promoters, thereby improving the model's expressive power and computational efficiency under mixed features.

[0024] Numerical features (such as temperature, pressure, GHSV, catalyst content, etc.) are continuous variables with significant differences in value range and dimensions. To eliminate the impact of these magnitude differences on training stability, this invention first performs normalization, ensuring that all numerical features are distributed within a uniform scale range. Subsequently, through a numerical feature embedding layer, trainable scaling factors and bias terms are set for each numerical feature. A dimensional affine mapping is used to map the normalized scalar values ​​into vector representations of a preset embedding dimension, thereby enabling continuous features to obtain a more flexible and stable expression in a unified embedding space, while maintaining dimensionality consistency with the classification feature embedding results. This achieves "integrated embedding of classification and numerical features," allowing both types of features to be modeled uniformly in the same vector space, while preserving the semantic relationships between categories and the numerical distribution characteristics. This is an improvement tailored to the data features of catalytic reaction systems, rather than a general standard encoding.

[0025] The feature encoding process is as follows: Create a categorical_dict dictionary to store the names and corresponding data of categorical features, forming a list of categorical features; create a numerical_dict dictionary to store the names and corresponding data of numerical features, forming a list of numerical features.

[0026] Categorical Feature Embedding Encoding: Four categorical features (catalyst type, carrier type, promoter type, etc.) are read from the categorical feature list. Based on the feature names stored in the dictionary `categorical_dict`, each categorical feature is processed using the Embedding encoding method, and each categorical feature is mapped to a low-dimensional dense vector representation. This allows the network to more effectively capture the potential relationships between categories, improving the model's computational efficiency and expressive power.

[0027] Numerical feature normalization encoding: 10 numerical features (such as temperature, pressure, GHSV, catalyst content, etc.) are read from the numerical feature dictionary; firstly, the numerical features are normalized; then, according to the storage method in numerical_dict, the normalized features are subjected to feature-by-feature and dimension-by-dimensional affine mapping through the numerical feature embedding layer to obtain the embedded numerical features.

[0028] For categorical features, a trainable categorical feature embedding layer is used to map each discrete value to a dimension of . The dense vector representation is used to obtain the embedding sequence of category features. Where B is the batch size, N c Let be the number of categorical features and d be the embedding dimension. This representation can characterize the latent semantic distance between different categories in a low-dimensional continuous space, providing a foundation for subsequent feature interaction modeling.

[0029] For numerical features, after normalization to eliminate dimensional differences, a numerical embedding layer is introduced to perform a feature-wise, dimension-wise affine mapping on each continuous variable. This involves configuring trainable scaling coefficients and bias terms for each numerical feature, mapping the scalar input to a d-dimensional vector representation, and obtaining the numerical feature embedding sequence. , where N n The number of numerical features is denoted by . This strategy avoids the limitation in expression caused by simple linear projection sharing parameters among all numerical variables, enabling the model to learn more suitable scales and translations for the distribution characteristics of different physical quantities, thereby improving the separability of the embedding space and training stability.

[0030] Step 3: Feature Fusion: After obtaining the vectorized representations of the two types of features, the category embedding sequence C and the numerical embedding sequence N are concatenated along the sequence dimension to form the fused feature sequence [CN]. To enable the Transformer to explicitly obtain a globally converged representation, a trainable classification token vector (CLS Token) is inserted at the beginning of the fused sequence. And concatenate it with the fusion sequence to form the final input [TCN], .

[0031] The CLS Token, as a learnable global convergence node, interacts with each feature location during the self-attention calculation process, enabling the encoder output to obtain a comprehensive representation of the overall input features. The input sequence [TCN] is then fed into the Transformer encoder to model the higher-order relationships between catalyst type, adjuvant combination, and reaction condition variables, and to provide a unified representation input for subsequent convolutional feature extraction and gated weighted fusion.

[0032] Step 4: Building the FMGTransformer Model. The input sequence [TCN] with the added classification label vector is fed into the Transformer encoder. The Transformer encoder consists of a multi-layer, multi-head self-attention structure. It models the global dependency between categorical and numerical features through a self-attention mechanism, and outputs an encoded result containing both a global convergent representation and a feature-level representation. The classification label vector at the first position of the sequence represents the global information of the overall input features, while the outputs at other positions retain the representation of each feature after global modeling.

[0033] Building upon this foundation, a multi-scale convolution module is further introduced to model local features of the Transformer-encoded feature sequences. By setting different convolution kernel scales, the combination relationships of the feature sequences within different neighborhood ranges are modeled, thereby extracting the global correlation features between reaction condition variables and catalyst properties. Simultaneously, the fused feature sequence [CN] without the introduction of classification label vectors is input as a parallel branch to another multi-scale convolution module to preserve the structural information in the original feature combinations and extract local detail features.

[0034] Subsequently, the feature representations from the global and local modeling branches are concatenated and fused, and the contributions of different features are adaptively weighted using a gated attention mechanism to form a final fused feature vector g that takes into account both global correlations and local details. The fused feature vector g is then passed through the fully connected prediction layer of a multilayer perceptron (MLP) to output the yield prediction results for the CO2 hydrogenation to methanol reaction.

[0035] The FMGTransformer model has now been constructed and is used to model the effects of reaction conditions, catalysts and their combinations on reaction yield.

[0036] Step 5: Divide the dataset into training and testing sets, train the FMGTransformer model, and use the trained FMGTransformer model to predict the yield of the CO2-to-hydrogen methanol reaction.

[0037] Example 2 This example is implemented in Python 3.9 environment. The code is written in Python and the frameworks used include Python, PyTorch, NumPy, Pandas, etc. The implementation flowchart is shown in Figure 1. The specific steps include: Step 1: Collect the reaction data of CO2 plus hydrogen to produce methanol and perform data preprocessing (including cleaning (deleting the feature with many missing features) and standardization (unifying the units) to build the dataset, as shown in Figure 2.

[0038] In the implementation, the dataset compiled by Manu Suvarna et al. was used as the primary data source. This dataset was constructed by searching the Web of Science and Scopus databases, with search keywords including "CO2 to methanol" and "Cu / Pd / In2O3 solid solution catalysts," covering publicly published literature from 1996 to 2021. Based on initial screening, a total of 131 relevant articles were collected, and reaction conditions and performance data were extracted from experimental sections or figures. The dataset covers multiple input variables, including catalyst type, metal content (wt%) (referring to metal or metal oxide), support type, promoter type and content (wt%), and catalyst specific surface area (SBET, m²). 2 The data included the yield per gram (g), and reaction conditions (temperature T, pressure P, gas space-time velocity GHSV, and H2 / CO2 feed ratio, etc.). After data cleaning and standardization, 1140 valid samples were obtained, containing 4 categorical features and 10 numerical features. The categorical feature list (cat_list) and the numerical feature list (num_list) were compiled as the basis for subsequent modeling. Methanol STY (space-time yield) was set as the target variable for prediction, and all other variables were used as input features for the model, serving as the basis for subsequent modeling.

[0039] The original Excel data was read, and each row was treated as a reaction sample. All numerical features were standardized in units, as shown in Table 1. The data was stored in an array format as a file named data.csv, containing the following for each sample: [Yield, Categorical Feature 1, ..., Categorical Feature 4, Numerical Feature 1, ..., Numerical Feature 10]. Yield was the target variable, and the remaining features were input features. Categorical features included catalyst type, support 1, support 2, and promoter type; numerical features included metal content, covalent radius, molecular weight of support 1, molecular weight of support 2, promoter content, SBET, temperature, pressure, GHSV, and H2 / CO2 feed ratio. The format of some sample data is shown below: ['Cu-based', 'ZrO2', 'SiO2', 'K', 8.5, 1.35, 92.4, 2.3, 140.5, 260,3.0, 6000, 3.0, 0.65]['Pd-based', 'In2O3', 'Al2O3', 'Na', 5.0, 1.28, 86.2, 1.5, 122.1, 250,2.5, 5500, 3.5, 0.72]['Cu-based', 'CeO2', 'ZrO2', 'Cs', 6.2, 1.40, 90.0, 1.8, 130.3, 270,3.2, 5800, 3.2, 0.78]...['Cu-based', 'ZnO', 'Al2O3', 'Na', 4.5, 1.31, 85.1, 2.0, 115.2, 240,2.0, 5000, 2.8, 0.68]['Pd-based', 'In2O3', 'ZrO2', 'K', 7.0, 1.25, 91.5, 2.1, 145.0, 265,3.1, 6100, 3.3, 0.74]['Cu-based', 'ZrO2', 'Al2O3', 'Cs', 5.5, 1.33, 89.0, 2.2, 135.4, 255,2.7, Table 1. Characteristic Units [5700, 3.1, 0.70]

[0040] Step Two: Categorical Feature Embedding Encoding and Numerical Feature Normalization Encoding. Categorical Feature Embedding Encoding: Read the prepared categorical feature list (cat_list) and numerical feature list (num_list). There are 4 categorical features: catalyst type, support 1, support 2, and promoter type; and 10 numerical features, including metal content, covalent radius, molecular weight of support 1, molecular weight of support 2, promoter content, specific surface area (SBET), temperature, pressure, gas space-time velocity (GHSV), and H2 / CO2 feed ratio. Categorical features are encoded using the embedding method: Let the i-th categorical feature be X. i The encoding process is as follows: Where c represents the number of categories for the feature. Classification features encoded with Embedding. The input to the classification feature embedding layer maps each classification feature to a dense vector of length d, forming a classification feature embedding sequence, i.e., the embedded classification features.

[0041] Numerical Feature Normalization Encoding: Numerical features are normalized to standardize their value range to the same scale. Then, a numerical feature embedding layer is introduced. For each numerical feature, a trainable scaling vector and bias vector are assigned. Through a dimensional affine mapping, scalar values ​​are mapped to d-dimensional embedding vectors, resulting in a sequence of embedded numerical features, which are the embedded numerical features. Their representation is as follows: , in, The trainable parameters corresponding to the j-th numerical feature; the j-th numerical feature X j Numerical features after numerical embedding Thus, each numerical feature is mapped to a vector representation consistent with the embedding dimension of the categorical feature, thereby realizing the expression of continuous features in a unified embedding space.

[0042] By embedding and encoding categorical features through a categorical feature embedding layer and numerical features through a numerical feature embedding layer, and then concatenating the embedded categorical and numerical features, a fused feature sequence [CN] is obtained, represented as: , where n is the number of categorical features, m is the number of numerical features, and d is the embedding dimension. Examples are as follows: Classification Feature 1 (Embedding) → [0.12, 0.07, 0.56, 0.23, 0.48, 0.74,…,0.44, 0.53, 0.55] Classification Feature 2 (Embedding) → [0.19, 0.22, 0.43, 0.38, 0.50, 0.56, …,0.62, 0.52, 0.41]… Numerical Feature 1 (Linear + Norm) → [0.45, 0.36, 0.49, 0.33, 0.52, …,0.48,0.50, 0.56, 0.60] Numerical Feature 2 (Linear + Norm) → [0.51, 0.44, 0.59, 0.61, 0.47, …,0.54,0.58, 0.61, 0.48]…A learnable classification tag vector CLS Token is inserted at the beginning position of the fused feature sequence to construct the input sequence [TCN] of the Transformer, and the input sequence is fed into the Transformer encoder for feature modeling; Step 3: Construct the FMGTransformer model. The FMGTransformer model combines global dependency modeling with local feature extraction to achieve accurate prediction of the yield of CO2 hydrogenation to methanol reaction. The FMGTransformer model comprises a Transformer encoder, a multi-scale convolutional module, a gated attention mechanism, and a multilayer perceptron (MLP). The Transformer encoder takes a sequence [TCN] as input. The output feature sequence is divided into an encoded feature sequence containing a classification label vector and encoded feature sequences corresponding to the remaining feature positions. This TCN is then used as the first branch input to the first multi-scale convolutional module, outputting the first branch feature vector (i.e., the global feature vector). Simultaneously, the fused feature sequence [CN] is input to the second multi-scale convolutional module, outputting the second branch feature vector (i.e., the local feature vector). The first and second branch feature vectors are concatenated and input into the gated attention mechanism to dynamically adjust feature importance, obtaining the final fused feature vector g. This final fused feature vector g is then input into the MLP, outputting the predicted methanol yield.

[0043] In this embodiment, after encoding the categorical and numerical features, the two types of features are fused under a unified embedding dimension d, and the input sequence of the Transformer encoder is constructed. Specifically, the categorical features are first input into the categorical feature embedding layer to obtain the category embedding sequence. The normalized numerical features are input into the numerical embedding layer. For each numerical feature, a trainable scaling vector and bias vector are set, and a numerical embedding sequence is obtained through a one-dimensional affine mapping. Subsequently, X cat With X num By concatenating along the sequence dimension, a fused feature sequence is obtained. , where B is the batch size, n and m represent the number of categorical features and numerical features, respectively, and d is the embedding dimension.

[0044] Building upon this, a global convergence path is constructed: a trainable classification token vector (CLS Token) is inserted at the beginning of the fused feature sequence [CN]. This vector serves as a model parameter, automatically learning the global convergence representation through training. The CLS Token is concatenated with the fused feature sequence to obtain the input to the Transformer encoder, represented as: Where Concat represents vector concatenation; X cls This represents a classification label vector, as shown in the example below, X (input) = [[0.29, 0.36, 0.47, ..., 0.34, 0.53, 0.44, 0.55], CLS Token[0.12, 0.07, 0.56, ..., 0.84, 0.44, 0.53, 0.55], Classification Feature 1 (Cu-based) [0.23, 0.11, 0.62, ..., 0.75, 0.62, 0.52, 0.41], Classification Feature 2 (ZrO2) [0.56, 0.35, 0.22, ..., 0.61, 0.53, 0.48, 0.60], Numerical Feature 1 (8.5) [0.44, 0.28, 0.50, ..., 0.72, 0.51, [0.49, 0.53], Numerical Feature 2 (1.35)...] This input sequence is then fed into a Transformer encoder to model the effects of reaction condition parameters, catalyst properties, and their combinations on the yield.

[0045] Through this feature fusion and input construction process, the present invention realizes the structured expression of multi-source heterogeneous features, solves the problem that traditional models have difficulty in handling classification and numerical information at the same time, and provides a structural basis for the efficient prediction of CO2 hydrogenation reaction.

[0046] The input categorical and numerical features are first processed through categorical feature embedding and numerical feature embedding, respectively. This layer maps all input features (including categorical and numerical features) to a unified 64-dimensional space, resulting in a fused feature sequence [CN]. Subsequently, a trainable classification token vector (CLS Token) is inserted at the beginning of the fused feature sequence to construct the input sequence X of the Transformer encoder. (input) (i.e., [TCN]). X (input) The input is fed into the Transformer encoder for feature extraction. The Transformer encoder (which consists of multiple Transformer encoding layers) utilizes a multi-head attention mechanism to capture the dependencies between features. The formula is as follows: Where Q, K, and V represent query, key, and value, respectively; d k The dimension of the key; This represents a multi-head attention mechanism. The output is a global interaction representation for each input feature location, still with a dimension of (n+m)×64, but the dependencies between features have been modeled and have contextual information.

[0047] The feature vector output from the final Transformer encoding layer is further transformed nonlinearly using the FFN module. This yields a non-linear enhanced version of each feature vector, which helps to further improve the model's expressive power.

[0048] Where X represents the input of the FFN module, ReLU represents the activation function, and W1, W2, b1, and b2 are parameters.

[0049] Based on this, the present invention constructs two parallel multi-scale convolutional feature extraction paths to enhance the model's ability to model the relationships between local feature combinations. The first branch takes the feature sequence encoded by the Transformer as input, which is then fed into the first multi-scale convolutional module, and its computational form is expressed as follows: (Conv1D represents 1D convolution operation, and k represents the size of the convolution kernel) Through multi-scale convolution, the model can extract feature combination information from different local ranges. The convolution results of each scale are concatenated in the channel dimension and obtained by pooling operation to obtain the first branch feature vector.

[0050] The second branch takes the fused feature sequence [CN] without introducing CLS tokens and without being re-encoded by the Transformer as input and feeds it directly into the second multi-scale convolution module for processing. This branch is used to preserve the local structural information of the original feature combination before attention modeling, thereby enhancing the model's ability to perceive local correlation patterns in the original feature space, and obtaining the feature vector of the second branch.

[0051] Then, the feature vectors of the two branches are concatenated and fed into the introductory attention mechanism to dynamically adjust the importance of the features: This yields the final fused feature vector g, which integrates local and global modeling information and serves as input before final prediction.

[0052] in, This represents the result of concatenating the feature vectors of the two branches. This represents the gating function, with W3, W4, b3, and b4 as parameters.

[0053] The final fused feature vector g is processed by a multilayer perceptron (MLP) to generate a prediction result, which is the predicted methanol yield of the reaction. The MLP is set to a three-layer structure with a 512-dimensional hidden layer and a 1-dimensional output, generating a prediction result for the yield of the CO2 hydrogenation to methanol reaction.

[0054] Step 5: Divide the dataset into training and testing sets and train the neural network. During training, the dataset is divided into training and testing sets in a ratio of 0.85 / 0.15. To ensure training stability and generalization ability, the Adam optimizer is used with a batch size of 128 and a dropout rate of 0.1 to enhance generalization. The initial learning rate is 0.001, and the learning rate decay is performed using the StepLR scheduler, reducing the learning rate to 0.3 times its original value every 40 epochs, for a total of 250 training optimizations.

[0055] To enable the model to learn better, the labels (i.e., methanol yield) of the training set were normalized using Min-Max. The label data was normalized using MinMaxScaler, which linearly compressed it to the range [0, 1].

[0056] During training, data is input into the FMGTransformer model, and training is performed using the hyperparameters configured above. The final output is the predicted methanol yield. The mean squared error (MSE) is used as the loss function to measure the average squared difference between the predicted and actual values. This loss function is defined as follows: , where y i It's a real label. Here, n is the predicted value, and n is the number of samples. The mean squared error loss function is very effective for regression tasks, as it can highlight large errors and encourage the model to reduce outliers, thus better fitting the data.

[0057] The model is trained to minimize the loss function, resulting in the trained FMGTransformer model.

[0058] Step Six: Test the Model and Predict Methanol Yields Under Different Conditions. After model training, predictions are made on the test set to obtain the model's prediction results. After training, the model has learned to capture the relationship between input features and methanol yield, thus providing accurate predictions for new data in the test set. The model's prediction results are destandardized and then compared with the actual labels.

[0059] To further highlight the advantages of the model in this application, after the model training is completed and the prediction results are obtained, the mean absolute error (MAE), root mean square error (RMSE), and coefficient of determination (R²) are used. 2 The prediction results of different models were evaluated, as shown in Table 2.

[0060] Table 2 Experimental Results

[0061] As can be seen from the table, the FMGTransformer model in R... 2 The model achieved a value of 0.9143, indicating that it can explain approximately 91.43% of the data variation, demonstrating a very high goodness of fit. The RMSE was 3.32%, and the MAE was 1.84%, both showing that the model has low prediction error and high prediction accuracy.

[0062] These results demonstrate that the FMGTransformer model exhibits strong generalization ability and superior performance in predicting methanol yield in CO2 hydrogenation reactions. Furthermore, compared to traditional methods, the FMGTransformer effectively captures complex nonlinear interactions, providing more accurate predictions.

[0063] Any aspects not covered in this invention are applicable to existing technologies.

Claims

1. A method for predicting the yield of CO2 hydrogenation to methanol reaction based on the FMGTransformer model, characterized in that, The prediction method includes the following steps: Collecting reaction data for CO2-to-methanol production under different reaction conditions and preprocessing it to create a dataset. Each data set in the dataset is saved as: [Yield—Categorical Features—Numerical Features], where each row represents data from a reaction experiment, containing corresponding categorical and numerical features; embedding and encoding the categorical features through a categorical feature embedding layer, and embedding and encoding the numerical features through a numerical feature embedding layer; concatenating the embedded categorical and numerical features to obtain a fused feature sequence [CN]; constructing an FMGTransformer model, which includes a Transformer encoder, a multi-scale convolutional module, a gated attention mechanism, and a multilayer perceptron (MLP); inserting a learnable classification tag vector (CLS Token) at the beginning of the fused feature sequence to construct the Transformer input sequence [TCN], and feeding this input sequence into the Transformer encoder for feature modeling; and then... The feature sequence output by the encoder is fed into the first multi-scale convolution module as input to output a global feature vector; at the same time, the fused feature sequence [CN] is input into the second multi-scale convolution module to output a local feature vector; the global feature vector and the local feature vector are concatenated and input into the gating attention mechanism to dynamically adjust the feature importance and obtain the final fused feature vector g; the final fused feature vector g is input into the multilayer perceptron (MLP) and the output is the predicted methanol yield of the reaction.

2. The prediction method according to claim 1, characterized in that, The classification characteristics include catalyst type, support 1, support 2 and accelerator type; the numerical characteristics include metal content, covalent radius, molecular weight of support 1, molecular weight of support 2, accelerator content, SBET, temperature, pressure, GHSV and H2 / CO2 feed ratio.

3. The prediction method according to claim 1, characterized in that, The embedding encoding process is as follows: a categorical_dict dictionary is created to store the names and corresponding data of categorical features, forming a list of categorical features; Create a dictionary named numerical_dict to store the names and corresponding data of numerical features, forming a list of numerical features; Categorical Feature Encoding: Four categorical features are read from the categorical feature list. Based on the feature names stored in the dictionary `categorical_dict`, each categorical feature is processed using Embedding encoding. Then, each categorical feature is mapped to a low-dimensional dense vector representation via a categorical feature embedding layer, allowing the network to more effectively capture the potential relationships between categories and obtain the embedded categorical features. Numerical Feature Encoding: Ten numerical features are read from the numerical feature list. First, the numerical features are normalized. Then, based on the storage method in `numerical_dict`, the normalized features are subjected to feature-by-feature, dimension-by-dimensional affine mapping via a numerical feature embedding layer to obtain the embedded numerical features. The embedded categorical features and numerical features are concatenated and used as the input to the FMGTransformer model.

4. The prediction method according to claim 1, characterized in that, The Transformer encoder contains N Transformer encoding layers. Each Transformer encoding layer captures long-range dependencies between input features through a self-attention mechanism. The output feature vector is further transformed nonlinearly through a feedforward neural network (FFN) module.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program can implement the method described in any one of claims 1-4.

Images