Multi-modal characterization molecular property prediction method based on layered bidirectional cross attention

Abstract

The invention provides a multi-modal characterization molecular property prediction method based on hierarchical bidirectional cross attention, and relates to the technical field of machine learning assisted organic chemistry, and the method comprises the following steps: S10, generating same-molecule multiple sequences for data enhancement; s20, coding the sequence features through a pre-trained molecular language model MolBERT; s30, performing multi-modal feature fusion through a layered bidirectional cross attention mechanism; s40, establishing a prediction head; s50, in the reasoning stage, only the feature extraction and fusion steps are executed, and a molecular property prediction result is output through the trained prediction head. According to the method, the molecular sequence, the topological graph structure and the fingerprint features are effectively integrated, so that the prediction precision of the model on a plurality of MoleculeNet (molecular network benchmark) public data sets is superior to that of an existing method.

Description

Technical Field

[0001] This invention relates to the field of machine learning-assisted organic chemistry, and more specifically to a multimodal characterization method for predicting molecular properties based on hierarchical bidirectional cross-attention. Background Technology

[0002] With the development of deep learning, machine learning has become an important tool in the fields of molecular property prediction and drug discovery. Traditional quantitative structure-activity relationship methods rely on manually selected molecular descriptors, which are difficult to capture complex three-dimensional structures and electronic effects, resulting in limited generalization ability. In recent years, end-to-end representation learning based on deep learning has significantly improved the accuracy of molecular property prediction, but the mainstream model approaches each have their own limitations.

[0003] In sequence modeling, *ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Prediction*, inspired by natural language processing, effectively captures sequence semantics and long-range dependencies, but struggles to explicitly utilize molecular topological information, exhibiting insufficient characterization capabilities for fused rings or highly branched molecules. Graph neural networks directly treat atoms as nodes and chemical bonds as edges, learning local and global topology through message passing mechanisms. *Neural Message Passing for Quantum Chemistry* laid the foundation for graph models in quantum property regression. Subsequently, *Directional Message Passing for Molecular Graphs* introduced angular information into the propagation process, significantly improving the prediction accuracy of energy and force constants. These methods can explicitly capture bonding and spatial constraints, but often neglect the chemical grammar rules inherent in the simplified linear input canonical sequence order of molecules, and struggle to directly incorporate the experiential knowledge of expert fingerprints. To alleviate overfitting caused by small sample sizes, pre-training and contrastive learning are widely adopted. The paper "Molecular Contrastive Learning of Representations via Graph Neural Networks" proposes the MolCLR framework, further demonstrating the advantages of self-supervised contrastive learning in molecular representation learning and property prediction. However, existing contrastive frameworks mostly focus on consistency within a single modality, lacking systematic solutions for cross-modal semantic alignment and information complementarity.

[0004] Therefore, the challenge facing this invention is how to synergistically utilize three types of information—sequence semantics, molecular topology, and molecular fingerprint—while ensuring computational efficiency, and further improve the generalization robustness of the model under small sample conditions. Summary of the Invention

[0005] The purpose of this invention is to provide a multimodal molecular property prediction method based on hierarchical bidirectional cross-attention. The method first extracts four features in parallel: MolBERT is used to extract sequence semantics from the simplified linear input canonical sequence of the molecule; Graphermer is used to extract global features of the molecular topology; NNConv is used to extract local molecular graph features; and a multilayer perceptron is used to extract molecular fingerprint features. These four features are aligned to a unified embedding dimension through independent linear mappings. Then, two complementary bidirectional cross-attention pathways are set up: the local graph and the simplified linear input canonical sequence of the molecule mutually query each other to establish the correspondence between atomic environment and semantic fragments; the global topology and fingerprint mutually query each other to associate macroscopic structure with chemical empirical knowledge. After obtaining the four interaction results, they are concatenated in the feature dimension and fed into a fusion perceptron for nonlinear compression to form a comprehensive molecular representation, and the prediction result is output through a prediction head. During the inference stage, feature extraction, hierarchical bidirectional cross-attention, forward computation of the fusion perceptron and the prediction head are performed, and the prediction result is directly output.

[0006] To achieve the above-mentioned objectives, the present invention employs the following technical solution: a multimodal method for predicting molecular properties based on hierarchical bidirectional cross-attention, comprising the following steps:

[0007] S10. First, the simplified linear input sequence set of molecules is validated and randomly enumerated. Then, a cheminformatics toolkit is used to construct the molecular graph. Extracting the node feature matrix With edge feature matrix The three types of molecular fingerprints were calculated and concatenated to obtain the fingerprint vector. The molecular map, fingerprint vector, and original label are combined. Unified caching prepares for subsequent feature extraction;

[0008] S20, will Each simplified linear input canonical sequence of molecules is fed into a 6-layer molecular bidirectional encoding representation transformer MolBERT model, and the word vectors and position vectors are superimposed to form an embedding tensor. Then, mean pooling is used to obtain the sequence representation. .by As input, the global topology vector is first obtained through the graph transformer Graphormer. Then, the local atomic-bond environment variables are calculated through neural network convolution (NNConv). Fingerprint vector The data is fed into a perceptron for nonlinear transformation, and the output features are then normalized using LayerNorm. All four vectors are subsequently mapped to a unified dimension. ;

[0009] S30. Design a hierarchical bidirectional cross-attention fusion mechanism: The first layer focuses on local graph vectors. With sequence characterization The second layer performs bidirectional cross-attention interaction on the global graph vector. With fingerprint vector A two-way cross-attention interaction is performed, resulting in four updated features after two layers of interaction. These features are then concatenated along the feature dimension to form... The fused vector of dimension is nonlinearly compressed through a multilayer perceptron to obtain a comprehensive molecular representation. ;

[0010] S40. Establish a prediction head Define a loss function, using binary cross-entropy loss with log odds (BCEWithLogitsLoss) for classification tasks. Mean squared error is used for regression tasks. , uniformly recorded as The AdamW weight decay adaptive moment estimation optimizer is used in conjunction with linear learning rate preheating and cosine annealing scheduling strategies, and the model parameters are updated through backpropagation algorithm.

[0011] In the S50 inference phase, only the S20 feature extraction and S30 feature fusion steps are performed, using the trained prediction head. Output molecular property prediction results .

[0012] As a multimodal molecular property prediction method based on hierarchical bidirectional cross-attention provided by the present invention, the specific steps of step S10 are as follows:

[0013] S11. Simplified Molecular Linear Input Canonical Sequence Validation and Enhancement

[0014] First, for the set of original simplified molecular linear input canonical sequences read from the dataset... For each string The cheminformatics toolkit (RDKit) is used to verify the validity of chemical structures, filtering out invalid molecules. After successful verification, a random enumeration algorithm is used to generate additional structures. Chemically equivalent but distinct sequences are denoted as... These new sequences are different from the original sequences. Together they constitute a collection Enhanced set of sequences :

[0015]

[0016] S12. Molecular Graph Construction and Feature Extraction

[0017] For each original molecule, use RDKit to extract its string. Constructing Molecular Diagram Objects At the node level, for each atomic node... Extract atomic properties to form a 9-dimensional node feature vector. , All node features together constitute the node feature matrix. At the edge level, for each chemical bond edge Extract key attributes to form a 3D edge feature vector. All edge features together constitute the edge feature matrix. Simultaneously, construct the edge index matrix. This represents the connectivity relationships in the graph; ultimately, each molecular graph is converted into a PyTorch Geometric (PYG) data object, containing... , as well as ;

[0018] S13, Molecular fingerprint generation and splicing

[0019] For each original molecule Calculate three types of binary molecular fingerprints: MACCS, PubChem, and Morgan, with dimensions of 167, 881, and 2048 respectively. Concatenate these fingerprint vectors into a 3096-dimensional hybrid molecular fingerprint vector. :

[0020]

[0021] S14. Simplified Tokenization of Linear Input Canonical Sequences for Molecular Applications

[0022] The augmented simplified molecular linear input canonical sequences were encoded using a MolBERT segmenter pre-trained on the PubChem-10M dataset (using byte-pair encoding). For the augmented set... Each string in Perform the following tokenization operations:

[0023]

[0024] in, A sequence of integers representing the Token ID. For sequence length, It is a binary mask sequence.

[0025] As a multimodal molecular property prediction method based on hierarchical bidirectional cross-attention provided by the present invention, the specific steps of step S20 are as follows:

[0026] S21. Simplified feature extraction of linear input canonical sequences in molecules

[0027] The tokenized simplified molecule is linearly input into the normalized sequence. The input is fed into the MolBERT model. This model first maps the word ID sequence to word embeddings and adds them to the corresponding positional embeddings to generate an initial embedding tensor. :

[0028]

[0029] Then With attention mask The context hidden state is fed into the MolBERT encoder and outputs the MolBERT encoder. :

[0030]

[0031] Finally, according to right The effective token portion is subjected to mean pooling to obtain a fixed-length feature vector representing the entire simplified molecule linear input canonical sequence. :

[0032]

[0033] in, Let j be the value of the attention mask vector at position j. It is the hidden state vector of the j-th token in the sequence;

[0034] S22, Global Graph Topological Feature Extraction

[0035] The molecular graph is fed into the Graphormer model. This model adds dummy nodes and introduces structural bias terms in the self-attention computation. To effectively capture the global structural information of the graph:

[0036]

[0037] Model output includes Hidden representation of each node The 0th node is a virtual node that is added, and the output of the virtual node is taken as the topological feature vector of the global graph. :

[0038]

[0039] S23, Local Image Environment Feature Extraction

[0040] Molecular graphs processed using NNConv and its node feature matrix Sum of edge feature matrices This model aggregates local neighborhood information through multi-layer message passing. In each layer, each node... The features are updated according to the following rules:

[0041]

[0042] in, It is a learnable linear transformation. Let v represent the set of neighbors of node v. It is a connection node and The feature vectors of the edges are obtained; subsequently, a global average pooling operation is performed on these node representations to obtain feature vectors representing the local environment information of the molecule. :

[0043]

[0044] in, , It is a node go through The updated feature vector after layer;

[0045] S24. Molecular fingerprint feature extraction

[0046] 3096-dimensional hybrid molecular fingerprint vector Enter one Multilayer perceptron (MLP) performs nonlinear transformation:

[0047]

[0048] in, It is the first The output feature vector of the layer perceptron represents the processed molecular fingerprint. and They are The weight matrix and bias vector of the layer.

[0049] when When it is 1, with As the initial input, after After layer transformation, the following is obtained Then, LayerNorm is applied to obtain the final fingerprint feature vector. , means as follows:

[0050]

[0051] S25, Multimodal Feature Unified Projection

[0052] The four feature vectors obtained above , , , Each element is transformed to a unified dimension through its own linear projection layer. The aligned feature vectors are obtained. , , , The projection formula is uniformly expressed as follows:

[0053]

[0054]

[0055] in, and It consists of the learnable parameter matrix and bias vector for the corresponding modality, and finally the aligned feature vector is used as the input of the subsequent cross-attention module.

[0056] As a multimodal molecular property prediction method based on hierarchical bidirectional cross-attention provided by the present invention, the specific steps of step S30 are as follows:

[0057] S31, Layered Bidirectional Cross-Attention

[0058] For the first fusion branch, the aligned local map feature vectors The aligned sequence feature vector serves as the input for the query. As inputs of key and value, calculate the interaction result between them. Its formula is expressed as follows:

[0059]

[0060] in, It is a learnable output projection matrix. The number of attention heads, and the calculation for each attention head:

[0061]

[0062]

[0063] At the same time, the aligned sequence feature vector The aligned local graph feature vector serves as the query input. Calculate the interaction result using key and value inputs. Its formula is expressed as follows:

[0064]

[0065] For the second fusion branch, the aligned global graph feature vectors are processed in parallel. The aligned fingerprint feature vector serves as the query input. Calculate the interaction result using key and value inputs. Its formula is expressed as follows:

[0066]

[0067] Aligned fingerprint feature vectors The aligned global graph feature vector serves as the query input. Calculate the interaction result using key and value inputs. Its formula is expressed as follows:

[0068]

[0069] S32, Feature Concatenation and Nonlinear Compression

[0070] The four interaction results are concatenated along the feature dimension to obtain a... The dimensionality-wise fusion vector is then fed into a fusion perceptron for nonlinear transformation and dimensionality reduction compression to obtain the final comprehensive molecular representation of the i-th molecule. This MLP integrates a linear transformation layer, LayerNorm, Gaussian error linear unit activation function (GELU), and random deactivation (Dropout). It uses the MLP for feature transformation to achieve feature compression.

[0071]

[0072] As a multimodal molecular property prediction method based on hierarchical bidirectional cross-attention provided by the present invention, the specific steps of step S40 are as follows:

[0073] S41. Establish a prediction head The MLP structure will synthesize molecular representation Map the output dimensions to the task space and design the corresponding output dimensions according to the task type;

[0074] S42, Model Task Loss The calculation uses different loss functions depending on the task type:

[0075] For classification tasks, Logits are used. The input binary cross-entropy loss The loss function introduces class weights. :

[0076]

[0077] in, These are the model's predicted values, i.e., the raw scores that have not yet been activated. This is the true label, i.e., the actual category of the sample. In classification tasks, it is usually 1 or 0 (corresponding to the positive and negative classes);

[0078] For regression tasks, mean squared error loss is used. :

[0079]

[0080] The model uses the AdamW optimizer for parameter updates and is trained using a learning rate scheduling strategy of linear learning rate warm-up-cosine annealing.

[0081] As a multimodal molecular property prediction method based on hierarchical bidirectional cross-attention provided by the present invention, the specific steps of step S50 are as follows:

[0082] S51. In the model inference stage, only the multimodal feature extraction step S20 and the feature fusion step S30 are executed to generate a comprehensive molecular representation. By predicting the head Directly output molecular property prediction results :

[0083]

[0084] in, T represents the number of tasks, and each dimension of the vector corresponds to a predicted value for a specific property.

[0085] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0086] 1. This invention provides more comprehensive multimodal feature extraction: Existing methods typically utilize only a single perspective of sequence or graph, resulting in information loss. This invention extracts three types of features in parallel: simplified linear input sequence semantics, molecular graph, and molecular fingerprint, obtaining a more three-dimensional and comprehensive initial characterization of the molecule.

[0087] 2. Deeper Feature Fusion Mechanism of this Invention: Addressing the redundancy and insufficient information coupling issues in existing fusion methods, this invention proposes a hierarchical bidirectional cross-attention fusion framework. Instead of simple concatenation or weighting, this framework employs a complementary strategy of "local graph query sequence" and "global graph query fingerprint," forcing the model to actively learn the correlations and dependencies between different modalities. This strengthens complementary information and yields a comprehensive molecular representation with higher information density.

[0088] 3. Enhanced generalization and robustness of this invention: To address the overfitting problem on small sample datasets, this invention employs enumeration for data augmentation. By constructing multiple equivalent representations of the same molecule with different strings, the model can learn a wider range of molecular representation variants. Attached Figure Description

[0089] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with examples of the invention to explain the invention and do not constitute a limitation thereof.

[0090] Figure 1 This is an overall flowchart of the multimodal molecular property prediction method in this application example.

[0091] Figure 2 This is a model framework diagram of the multimodal molecular property prediction method in the examples of this application. Detailed Implementation

[0092] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. Of course, the specific examples described herein are merely illustrative and not intended to limit the invention.

[0093] Example 1

[0094] See Figure 1 As shown, the technical solution provided in this example is a multimodal characterization method for predicting molecular properties based on hierarchical bidirectional cross-attention, which includes the following steps:

[0095] S10. First, the set of simplified molecular linear input canonical sequences is validated and randomly enumerated. Then, a cheminformatics toolkit is used to construct the molecular graph. Extracting the node feature matrix With edge feature matrix The three types of molecular fingerprints were calculated and concatenated to obtain the fingerprint vector. The molecular map, fingerprint vector, and original label are combined. Unified caching prepares for subsequent feature extraction;

[0096] S20, will Each simplified linear input canonical sequence of molecules is fed into a 6-layer molecular bidirectional encoding representation transformer MolBERT model, and the word vectors and position vectors are superimposed to form an embedding tensor. Then, mean pooling is used to obtain the sequence representation. .by As input, the global topology vector is first obtained through the graph transformer Graphormer. Then, the local atomic-bond environment variables are calculated through neural network convolution (NNConv). Fingerprint vector The data is fed into a perceptron for nonlinear transformation, and the output features are then normalized using LayerNorm. All four vectors are subsequently mapped to a unified dimension. ;

[0097] S30. Design a hierarchical bidirectional cross-attention fusion mechanism: The first layer focuses on local graph vectors. With sequence characterization The second layer performs bidirectional cross-attention interaction on the global graph vector. With fingerprint vector A two-way cross-attention interaction is performed, resulting in four updated features after two layers of interaction. These features are then concatenated along the feature dimension to form... The fused vector of dimension is nonlinearly compressed through a multilayer perceptron to obtain a comprehensive molecular representation. ;

[0098] S40. Establish a prediction head Define a loss function, using binary cross-entropy loss with log odds (BCEWithLogitsLoss) for classification tasks. Mean squared error is used for regression tasks. , uniformly recorded as The model parameters are updated using the AdamW algorithm in combination with linear learning rate preheating and cosine annealing scheduling strategy, and the backpropagation algorithm is used.

[0099] In the S50 inference phase, only the S20 feature extraction and S30 feature fusion steps are performed, using the trained prediction head. Output molecular property prediction results .

[0100] The specific steps of step S10 are as follows:

[0101] S11. Simplified Molecular Linear Input Canonical Sequence Validation and Enhancement

[0102] First, taking the ClinTox dataset as an example, this dataset is a benchmark task from MoleculeNet designed to evaluate the toxicity of drugs in clinical trials. This dataset contains 1478 compounds, including drugs that failed in clinical trials and those approved by the U.S. Food and Drug Administration (FDA), and is labeled with two toxicity endpoints: clinical trial toxicity and FDA approval status. When processing this dataset, for the original simplified linear input canonical sequence set read from the dataset, the chemical structure validity of each string is first verified using the cheminformatics toolkit (RDKit). After successful verification, a random enumeration algorithm is used to set... This generates two additional chemically equivalent but distinct sequences, denoted as... These new sequences, together with the original sequences, constitute an enhanced set containing three sequences:

[0103]

[0104] The table below shows when The string variants generated from the same aspirin molecule, as well as the original representation.

[0105] Table 1. Data enhancement of aspirin molecules

[0106]

[0107] S12. Molecular Graph Construction and Feature Extraction

[0108] For each original molecule, use RDKit to extract its string. Constructing Molecular Diagram Objects At the node level, for each atomic node... Extract atomic properties to form a 9-dimensional node feature vector. , All node features together constitute the node feature matrix. At the edge level, for each chemical bond edge Extract key attributes to form a 3D edge feature vector. All edge features together constitute the edge feature matrix. Simultaneously, construct the edge index matrix. This represents the connectivity relationships in the graph; ultimately, each molecular graph is converted into a PyTorch Geometric (PYG) data object, containing... , as well as ;

[0109] S13, Molecular fingerprint generation and splicing

[0110] In order to incorporate classical chemical structure information, for each original molecule Calculate three types of binary molecular fingerprints: MACCS, PubChem, and Morgan, with dimensions of 167, 881, and 2048 respectively. Concatenate these fingerprint vectors into a 3096-dimensional hybrid molecular fingerprint vector. :

[0111]

[0112] S14. Simplified Tokenization of Linear Input Canonical Sequences for Molecular Applications

[0113] The augmented simplified molecular linear input canonical sequences were encoded using a MolBERT segmenter pre-trained on the PubChem-10M dataset (using byte-pair encoding). For the augmented set... Each string in Perform the following tokenization operations:

[0114]

[0115] in, A sequence of integers representing the Token ID. The sequence length is given. It is a binary mask sequence.

[0116] The specific steps of step S20 are as follows:

[0117] S21. Simplified feature extraction of linear input canonical sequences in molecules

[0118] The tokenized simplified molecule is linearly input into the normalized sequence. The input is fed into the MolBERT model. This model first maps the word ID sequence to word embeddings and adds them to the corresponding positional embeddings to generate an initial embedding tensor. :

[0119]

[0120] Then With attention mask The 6-layer MolBERT encoder is fed along with the data. Based on the configuration (freeze_layers: "0-2"), the parameters of the first 3 layers are frozen to preserve the basic chemical semantics learned during pre-training on PubChem-10M, while the remaining layers are fine-tuned in downstream tasks. Output context hidden state. :

[0121]

[0122] Finally, according to right The effective token portion is subjected to mean pooling to obtain a fixed-length feature vector representing the entire simplified molecule linear input canonical sequence. :

[0123]

[0124] in, Let j be the value of the attention mask vector at position j. It is the hidden state vector of the j-th token in the sequence.

[0125] S22, Global Graph Topological Feature Extraction

[0126] The molecular graph is fed into the Graphormer model. This model adds dummy nodes and introduces structural bias terms in the self-attention computation. To effectively capture the global structural information of the graph:

[0127]

[0128] Model output includes Hidden representation of each node The 0th node is a virtual node that was added. The output representation of the virtual node is taken as the topological feature vector of the global graph. :

[0129]

[0130] S23, Local Image Environment Feature Extraction

[0131] Molecular graphs processed using NNConv and its node feature matrix Sum of edge feature matrices The network consists of four NNConv layers with a hidden layer dimension of 192, aggregating local neighborhood information through multi-layer message passing. In each layer, each node... The features are updated according to the following rules:

[0132]

[0133] in It is a learnable linear transformation. Let v represent the set of neighbors of node v. It is a connection node and The feature vectors of the edges are obtained; subsequently, a global average pooling operation is performed on these node representations to obtain feature vectors representing the local environment information of the molecule. :

[0134]

[0135] in, , It is a node go through The updated feature vectors of the layer.

[0136] S24. Molecular fingerprint feature extraction

[0137] 3096-dimensional hybrid molecular fingerprint vector Enter one Multilayer perceptron (MLP) performs nonlinear transformation:

[0138]

[0139] in, It is the first The output feature vector of the layer perceptron represents the processed molecular fingerprint. and They are The weight matrix and bias vector of the layer.

[0140] when When it is 1, with As the initial input, after three layers of transformation, we obtain Then, LayerNorm was applied to obtain the final 256-dimensional fingerprint vector. , means as follows:

[0141]

[0142] S25, Multimodal Feature Unified Projection

[0143] The four feature vectors obtained above , , , Each feature vector is transformed to a uniform dimension of 256 through its own linear projection layer, resulting in aligned feature vectors. , , , As the input for cross-attention, its projection formula is uniformly expressed as follows:

[0144]

[0145]

[0146] in, and These are the learnable parameter matrix and bias vector for the corresponding mode.

[0147] The specific steps of step S30 are as follows:

[0148] S31, Layered Bidirectional Cross-Attention

[0149] For the first fusion branch, the aligned local map feature vectors The aligned sequence feature vector serves as the input for the query. As inputs of key and value, calculate the interaction result between them. Its formula is expressed as follows:

[0150]

[0151] in, It is a learnable output projection matrix. The number of attention heads, and the calculation for each attention head:

[0152]

[0153]

[0154] At the same time, the aligned sequence feature vector The aligned local graph feature vector serves as the query input. Calculate the interaction result using key and value inputs. Its formula is expressed as follows:

[0155]

[0156] For the second fusion branch, the aligned global graph feature vectors are processed in parallel. The aligned fingerprint feature vector serves as the query input. Calculate the interaction result using key and value inputs. The formula is expressed as follows:

[0157]

[0158] At the same time, the aligned fingerprint feature vector The aligned global graph feature vector serves as the query input. Calculate the interaction result using key and value inputs. Its formula is expressed as follows:

[0159]

[0160] S32, Feature Concatenation and Nonlinear Compression

[0161] The four interaction results are concatenated along their feature dimensions to obtain a 1024-dimensional fusion vector. This vector is then fed into a fusion perceptron for nonlinear transformation and dimensionality reduction compression to obtain the final comprehensive molecular representation of the i-th molecule. The perceptron employs a two-layer fully connected structure, with Dropout set to 0.5 and a uniform embedding dimension of 256.

[0162]

[0163] The specific steps of step S40 are as follows:

[0164] S41. Establish a prediction head The MLP structure represents a 256-dimensional integrated molecular structure. Mapped to the task space. This MLP employs a two-layer hidden layer structure. Using the GELU activation function and Dropout, the output 2D vector corresponds to the two tasks of ClinTox (Clinical Toxicity CT_TOX and Clinical Trial Phase CT_USED).

[0165] S42. Calculate the loss based on the task type.

[0166] For the ClinTox classification task, a binary cross-entropy loss based on Logits is used. To address the class imbalance problem in the dataset, class weights pos_weight=[0.0646,12.8023] are introduced into this loss. The first value in this vector corresponds to the first task, assigning higher weights to negative samples; the second value corresponds to the second task, assigning higher weights to positive samples.

[0167]

[0168] The model uses the AdamW optimizer for parameter updates, with a base learning rate (lr) of 1.0e-04 and a weight decay of 0.018. It also employs a cosine annealing learning rate scheduling strategy with restart, with an initial epoch of 10 epochs, to achieve stable and efficient training.

[0169] The specific steps of step S50 are as follows:

[0170] S51. During the model inference phase, only steps S20 and S30 are executed to generate the comprehensive molecular representation. The loss calculation and parameter update process in step S42 are not performed. Instead, the prediction head... Directly output molecular property prediction results :

[0171]

[0172] in, This corresponds to the two classification tasks in ClinTox, where each dimension of the vector corresponds to a predicted value for a specific property.

[0173] To verify the effectiveness of this model framework, the performance of the method was tested using publicly available benchmark datasets, the information of which is shown in Table 2. The datasets are from MoleculeNet and include 5 classification task datasets and 3 regression task datasets.

[0174] Table 2 shows the dataset information.

[0175]

[0176] To ensure the consistency and reliability of the experimental results, all datasets were split according to the skeleton partitioning method and divided into training, validation, and test sets in an 8:1:1 ratio. Furthermore, three random seeds were used for each dataset, and the final evaluation was performed on the test set. Table 3 below compares the proposed model with several baseline models, including those based on simplified molecular linear input canonical sequences, molecular graphs, and multimodal representations. All results are taken from published literature. ROC-AUC area was used to evaluate the classification task, and RMSE was used to evaluate the regression task. The results in the table are the mean and standard deviation of all evaluation metrics across three experiments. Bold text indicates the best performance on the corresponding dataset, and underlined text indicates the second best. The results show that the method of this invention significantly outperforms several existing baseline models and achieves better performance in molecular property prediction.

[0177] Table 3 shows the performance comparison.

[0178]

[0179] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multi-modal representation molecular property prediction method based on hierarchical bidirectional cross-attention, characterized in that, The step S10 comprises the following steps: S10, first, the legality of the input of the simplified molecular linear input specification sequence set is checked and randomly enumerated, and a chemical information toolkit is called to construct a molecular graph , a node feature matrix is extracted and an edge feature matrix , and three types of molecular fingerprints are calculated, and a fingerprint vector is spliced , the molecular graph, the fingerprint vector and the original label are unified and cached, and preparation is made for subsequent feature extraction; S20, will Each simplified linear input canonical sequence of molecules is fed into a 6-layer molecular bidirectional encoding representation transformer MolBERT model, and the word vectors and position vectors are superimposed to form an embedding tensor. Then, mean pooling is used to obtain the sequence representation. ,by As input, the global topology vector is first obtained through the graph transformer Graphormer. Then, the local atomic-bond environment variables are calculated through neural network convolution NNConv. fingerprint vector The data is fed into a perceptron for nonlinear transformation, and the output features are then normalized using LayerNorm. All four vectors are subsequently mapped to a unified dimension. ; S30, design layered bidirectional cross attention fusion mechanism: the first layer to local graph vector with sequence representation Bidirectional cross attention interaction, the second layer to global graph vector with fingerprint vector Bidirectional cross attention interaction, two layers of interaction get four updated features, which are spliced in feature dimension to form Dimension fusion vector, nonlinear compression through multilayer perceptron, get comprehensive molecular representation ; S40、establishing a prediction head , defining a loss function, using binary cross-entropy loss with logits BCEWithLogitsLoss for classification tasks , using mean squared error for regression tasks , uniformly denoted as , using the weight decay adaptive moment estimation optimizer AdamW with linear learning rate warm-up and cosine annealing scheduling strategies to update the model parameters through the back propagation algorithm; S50, the inference stage only performs the feature extraction step S20 and the feature fusion step S30, through the trained prediction head outputting the prediction result of the molecular property .

2. The hierarchical bidirectional cross-attention based multi-modal representation molecular property prediction method according to claim 1, characterized in that, The step S10 comprises the following steps: S11, Simplified molecular linear input canonical sequence verification and enhancement First, for the original set of simplified molecular-input line-entry system sequences read from the dataset , call the chemistry information toolkit RDKit for chemical structure validity checking for each string , filter out invalid molecules, and after passing the check, use a random enumeration algorithm to additionally generate chemically equivalent but different sequences, denoted as , which, together with the original sequences , form an enhanced set containing sequences : ； S12, Molecular graph construction and feature extraction For each original molecule, use RDKit to extract its string Construct a molecule graph object At the node level, for each atomic node Extract atomic properties to form a 9-dimensional node feature vector , All node features together form a node feature matrix At the edge level, for each chemical bond edge Extract bond properties to form a 3-dimensional edge feature vector All edge features together form an edge feature matrix At the same time, construct an edge index matrix To represent the connection relationship in the graph; finally, each molecule graph is converted into a Pytorch geometry library data object, containing , And ; S13, Molecular fingerprint generation and splicing For each original molecule Three types of binary molecular fingerprints: MACCS, PubChem and Morgan, with dimensions 167, 881 and 2048, respectively, were computed and concatenated into a hybrid molecular fingerprint vector of dimension 3096 : ； S14, Simplified molecular linear input canonical sequence tokenization The enhanced SMILES canonical sequence is encoded using the MolBERT tokenizer pre-trained on the PubChem-10M dataset with byte pair encoding for each string in the enhanced set The following tokenization is performed for each string in the enhanced set ； wherein, is a sequence of integers representing Token IDs, is the sequence length, is a sequence of binary masks.

3. The hierarchical bidirectional cross-attention based multi-modal representation molecular property prediction method according to claim 1, characterized in that, The step S20 comprises the following steps: S21, Simplified molecular linear input canonical sequence feature extraction Tokenized simplified molecular linear input specification sequence is input to the MolBERT model, which first maps the token ID sequence into word embeddings and adds the corresponding position embeddings to generate an initial embedding tensor : ； Subsequently, the output of the MolBERT encoder is passed through a linear layer to obtain the context hidden state h along with the attention mask and is output as the context hidden state h : ； Finally, according to perform mean-pooling on the valid Token part of , obtaining a fixed-length feature vector representing the entire simplified molecular linear input canonical sequence : ； wherein, is the value of the attention mask vector at the j-th position, is the hidden state vector of the j-th Token in the sequence; S22, Global graph topology feature extraction The molecular graph is fed into the Graphormer model, which adds virtual nodes and introduces a structural bias term in the self-attention computation to capture the global structural information of the graph . ； The model output contains the hidden representation of the 0th node is a dummy node added, taking the output representation of the dummy node as the global graph topology feature vector : ； S23, Local graph environment feature extraction Processing a molecular graph by a NNConv and its node feature matrix and edge feature matrix The model aggregates local neighborhood information by multi-layer message passing, in each layer, the features of each node are updated according to the following rule: ； wherein, is a learnable linear transformation, denotes the set of neighbors of node v, is the feature vector of the edge connecting nodes and ; performing a global average pooling operation on these node representations yields a feature vector representative of the local environmental information of the molecule ； wherein , is a node after layer updated feature vector S24, Molecular fingerprint feature extraction a 3096-dimensional mixed molecular fingerprint vector inputting one a multi-layer perceptron (MLP) of layers performs a non-linear transformation: ； wherein, is the first layer perceptron output feature vector representing the processed molecular fingerprint representation, and are the weight matrix and bias vector of the layer, respectively. When is 1, the following is applied: As the initial input, the following is applied: After layer transformation, the following is obtained: The final fingerprint feature vector is obtained by applying LayerNorm: is represented as follows: ； S25, Multi-modal feature unified projection The four feature vectors obtained above are respectively transformed to a unified dimension by respective linear projection layers , , , to obtain aligned feature vectors , , , , and the projection formula is uniformly represented as follows:​ ； ； wherein, and are learnable parameter matrices and bias vectors for the corresponding modalities, and the aligned feature vectors are finally obtained as the input of the subsequent cross-attention module.

4. The hierarchical bidirectional cross-attention multi-modal representation based molecular property prediction method according to claim 1, characterized in that, The step S30 comprises the following steps: S31, Hierarchical bidirectional cross-attention For the first fusion branch, the aligned local graph feature vector As query input Query, the aligned sequence feature vector As key Key and value Value input, the interaction result of the two is calculated The formula is as follows: ； wherein, is a learnable output projection matrix, is the number of attention heads, each of which is computed as: ； ； the aligned sequence feature vector the aligned local graph feature vector as query input the interaction result as key and value input which is expressed in the formula as follows: ； For the second fusion branch, the aligned global graph feature vectors As query input, the aligned fingerprint feature vectors As key and value input, the interaction results are computed which is expressed as follows: ； the aligned fingerprint feature vector the aligned global graph feature vector as query input the interaction result as key and value input which is expressed in the formula as follows: ； S32, Feature splicing and nonlinear compression The four interaction results are spliced in the feature dimension to obtain a dimensional fusion vector, and the vector is sent to a fusion perception machine for nonlinear transformation and dimension reduction compression to obtain a final comprehensive molecular representation of the i th molecule The MLP integrates a linear transformation layer, LayerNorm, a Gaussian error linear unit activation function GELU, and random inactivation Dropout inside, and adopts the MLP to perform feature transformation to realize feature compression. 。 5. The hierarchical bidirectional cross-attention based multi-modal representation molecular property prediction method according to claim 1, characterized in that, The step S40 comprises the following steps: S41, establishing a prediction head MLP structure, the integrated molecular representation is mapped to the task space, and a corresponding output dimension is designed according to the task type; S42, model task loss The calculation of the model task loss S42 is different according to the type of the task. For classification tasks, a binary cross-entropy loss with Logits as input is used , the loss function incorporates class weights : ； wherein, is the predicted value of the model, i.e. the raw score that has not yet been activated, is the true label, i.e. the actual class of the sample, typically 1 or 0 in classification tasks, corresponding to the positive and negative class; For regression tasks, the mean squared error loss is adopted : ； The model uses AdamW to update parameters, and cooperates with the learning rate scheduling strategy of linear learning rate warm-up-cosine annealing to train.

6. The hierarchical bidirectional cross-attention based multi-modal representation molecular property prediction method according to claim 1, characterized in that, The step S50 comprises the following steps: S51, in the model inference stage, only the multi-modal feature extraction of S20 and the feature fusion step of S30 are performed to generate a comprehensive molecular representation , the prediction head directly outputs the molecular property prediction result : ； wherein, T is the number of tasks, each dimension of the vector corresponds to a predicted value for a particular property.

Images