Molecular multi-modal characterization method based on attention fusion

Abstract

The invention discloses a molecular multi-mode characterization method based on attention fusion, and the method comprises the following steps: firstly, carrying out the independent feature extraction of the one-dimensional, two-dimensional and three-dimensional modes of a molecule through a molecular sequence encoder, a molecular map encoder and a molecular conformation encoder, and carrying out the feature extraction of the one-dimensional, two-dimensional and three-dimensional features of the molecule; based on natural language text description, multi-level contrast learning is adopted, and gradual alignment of each mode and the text is achieved; secondly, multi-modal consistency constraint is introduced, the characterization distance of the same molecule in different modals is shortened, and semantic consistency is ensured; thirdly, adaptively integrating multi-modal features through a modal attention fusion module to obtain uniform molecular semantic representation, and further performing final alignment with a text embedding space; and finally, completing tasks such as molecule-text bidirectional retrieval and molecule attribute prediction by utilizing fusion representation. According to the method, under the condition that the model does not need to be retrained for different tasks, natural language-driven molecular retrieval, editing and property prediction can be realized, and the method has relatively high robustness, expansibility and universality.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of molecular representation learning and molecule generation, and specifically relates to a molecular multi-modal representation method based on attention fusion, which can be widely applied to drug molecule design, property prediction, structure retrieval and intelligent processing of multi-modal chemical information. BACKGROUND

[0002] In the research fields of drug design and material discovery, how to effectively represent and understand molecular structure has always been a core problem. Traditional molecular modeling methods usually rely on artificial experience or rule-based modeling methods, such as using molecular SMILES sequences or molecular graph structures for feature extraction and generation. Although this method can capture some structural features of the molecule, it is difficult to express and interact with semantics through natural language, resulting in poor controllability and insufficient explanation in the molecule generation process, and cannot meet the needs of human-computer collaborative molecule design.

[0003] In recent years, the introduction of multi-modal learning and contrast learning has provided a new technical approach for molecule-text joint modeling. Researchers have attempted to map molecular structures and text descriptions to a shared embedding space to achieve cross-modal retrieval and semantic matching. However, existing molecule-text modeling methods still have the following main shortcomings:

[0004] (1) Single modality, insufficient semantic coverage: existing methods are mostly based on single-modal representation (such as SMILES or molecular graph), ignoring the spatial geometry and chemical semantic information contained in the three-dimensional conformation of the molecule, resulting in limited generalization ability of the model in complex property prediction and structure generation tasks;

[0005] (2) Rough alignment, difficult to maintain consistency: most methods only perform one-time molecule-text alignment, lacking a multi-stage gradual alignment mechanism, which can easily lead to semantic space collapse or inconsistency between different modalities;

[0006] (3) Lack of adaptive mechanism for modality fusion: existing multi-modal fusion mostly uses simple concatenation or weighted averaging, failing to adaptively adjust the importance of different modalities according to task semantics;

[0007] (4) Insufficient task generality: most models need to be retrained or fine-tuned for specific tasks, making it difficult to support multiple types of downstream tasks such as retrieval, editing and property prediction.

[0008] Therefore, there is an urgent need for a multi-modal joint learning framework that can simultaneously align one-dimensional molecular sequences, two-dimensional molecular graphs and three-dimensional conformations, through gradual alignment, multi-modal consistency constraints and modality attention fusion mechanisms, to construct a robust shared semantic embedding space, thereby realizing natural language-driven integrated modeling of molecule understanding, retrieval and attribute editing. SUMMARY

[0009] In view of the problems of single mode, insufficient semantic alignment, static fusion mode, and insufficient task universality of existing molecular-text cross-modal modeling methods, the application provides an end-to-end three-modal molecular-text representation learning method based on step-by-step alignment and modal attention fusion.

[0010] The method can simultaneously model one-dimensional sequence (SMILES), two-dimensional topological graph, and three-dimensional conformation information of a molecule, realize high-precision semantic alignment of the molecule and natural language text through stage-by-stage alignment, multi-modal consistency constraint, and modal attention fusion mechanism, support cross-modal retrieval, molecular attribute prediction, and natural language driven molecular generation tasks, and thus significantly improve semantic controllability, generalization, and interactive experience of the molecular design process.

[0011] The technical scheme adopted by the application to achieve the above object is:

[0012] A molecular multi-modal representation method based on attention fusion comprises the following steps:

[0013] 1) Obtain a SMILES sequence of a molecule and a corresponding natural language text description thereof, convert the SMILES sequence into a molecular graph structure and a molecular three-dimensional conformation respectively, and form a three-modal-text paired data set;

[0014] 2) Use different encoders for encoding to obtain multi-modal molecular embedding vectors and text embedding vectors;

[0015] 3) Align each kind of molecular mode and the text mode respectively through single-modal contrastive learning, obtain a modal-level shared embedding space, and impose multi-modal consistency constraints to minimize the distance between the 1D, 2D, and 3D three-modal embedding of the same molecule;

[0016] 4) Through a multi-head attention mechanism, weight and fuse the features of the three kinds of molecular modes to obtain a unified fused molecular shared embedding space, and perform final alignment with the text embedding vector;

[0017] 5) Based on the fused shared embedding space, perform downstream molecular-text bidirectional retrieval and molecular attribute prediction tasks to realize cross-modal semantic matching and reasoning.

[0018] The step 1) comprises the following steps:

[0019] 1.1) Convert the SMILES sequence of the molecule into a molecular graph structure using a chemical parsing tool, wherein the nodes represent atoms and the edges represent chemical bonds;

[0020] 1.2) Convert the SMILES sequence of the molecule into three-dimensional atoms, and calculate the three-dimensional atomic coordinate information using the molecular conformation generation algorithm, to obtain the three-dimensional molecular modality.

[0021] The step 2) comprises the following steps:

[0022] 2.1) The SMILES sequence of the molecule, the molecular graph structure and the three-dimensional conformation of the molecule are respectively represented by the pre-trained one-dimensional molecular encoder, the two-dimensional molecular encoder and the three-dimensional molecular encoder, to obtain the multi-modal molecular embedding vector;

[0023] 2.2) The molecular text description is encoded by the pre-trained PubMedBERT model to obtain the text embedding vector, and the text embedding vector is consistent in dimension with the multi-modal molecular embedding vector.

[0024] The step 3) comprises the following steps:

[0025] 3.1) The molecular embedding vectors of each modality and the text embedding vectors are compared and learned by using the symmetric InfoNCE loss function, while using the hard negative sample mining mechanism to maximize the similarity of the matching pairs and minimize the similarity of the non-matching pairs, to obtain the shared embedding space of different modalities.

[0026] 3.2) The consistency loss of the three molecular modalities is calculated, and the sum is minimized to impose consistency constraints between the three molecular modalities.

[0027] For different modalities of the same molecular sample and the corresponding text description, define the positive sample pair as the matching pair, and the negative sample pair randomly combined between different samples is defined as the non-matching pair. The cosine similarity of the molecular embedding vectors and the text embedding vectors of different modalities is calculated as the matching similarity, and the symmetric InfoNCE loss function is used for contrastive learning, and the symmetric InfoNCE loss function is specifically:

[0028]

[0029] Wherein, represents the average of all positive sample pairs , that is, the sample distribution takes the mathematical expectation, and represent the structure-text pair of each molecule, and represent the negative samples randomly sampled from the noise distribution, represents the energy function with flexibility with and as variables, , denotes a function composition, and denote two linear transformations.

[0030] The calculation of the three molecular modal consistency loss is specifically:

[0031]

[0032] wherein, denotes the sum of the three molecular modal consistency losses, , and denote three molecular embedding space vectors.

[0033] The three molecular modal features are weighted and fused through the multi-head attention mechanism, and the specific process is as follows:

[0034]

[0035]

[0036] wherein, denotes a molecular vector of the kth modal, denotes a molecular vector obtained after attention fusion, denotes an attention weight of each modal, denotes a trainable parameter.

[0037] The loss function is used for contrastive learning to align the fused molecular shared embedding space and the text embedding vector, wherein is a text vector and is an InfoNCE contrastive loss of the three molecular fused vectors , that is:

[0038] .

[0039] A molecular multi-modal representation system based on attention fusion comprises:

[0040] A multi-modal molecular data processing module is configured to obtain SMILES sequences of molecules and corresponding natural language text descriptions thereof, convert the SMILES sequences into molecular graph structures and molecular three-dimensional conformations respectively, and form a three-modal-text paired data set.

[0041] A natural language encoding module is configured to use different encoders to encode and obtain multi-modal molecular embedding vectors and text embedding vectors.

[0042] The step-by-step alignment and multi-modal consistency learning module is used to realize the alignment of each molecular modality and the text modality respectively through single-modality contrastive learning, obtain a modality-level shared embedding space, and impose multi-modal consistency constraints to minimize the distance between the 1D, 2D and 3D modality embeddings of the same molecule;

[0043] The modality attention fusion module is used to fuse the features of the three molecular modalities through a multi-head attention mechanism, obtain a unified fusion molecular shared embedding space, and finally align with the text embedding vector;

[0044] The downstream task module is used to perform downstream molecular-textual bidirectional retrieval and molecular attribute prediction tasks based on the fused shared embedding space, and realize cross-modal semantic matching and reasoning.

[0045] The present application has the following advantages and benefits:

[0046] 1. Multi-modal collaborative representation is realized: 1D, 2D and 3D molecular features are integrated at the same time to obtain comprehensive semantic representation;

[0047] 2. Cross-modal alignment accuracy is improved: step-by-step alignment and consistency constraints effectively prevent embedding space collapse;

[0048] 3. Adaptive modality fusion: attention mechanism dynamically adjusts modality weights to enhance task adaptability;

[0049] 4. Support natural language driven molecular optimization: no need to retrain the generator to perform semantic editing;

[0050] 5. Strong universality: can be applied to molecular retrieval, drug discovery, property prediction and molecular generation tasks. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 The overall architecture diagram of the model;

[0052] Figure 2 The overview diagram of the molecular three-branch contrastive learning module of the model;

[0053] Figure 3 The multi-modal consistency module schematic diagram;

[0054] Figure 4 The multi-modal molecular representation fusion flowchart based on the attention mechanism in TriFusion-Mol. DETAILED DESCRIPTION

[0055] The present application will be further described in detail below in conjunction with the drawings and examples.

[0056] An end-to-end three-modal molecular text representation learning method based on step-by-step alignment and modal attention fusion, comprising the following steps:

[0057] 1) Data acquisition and preprocessing

[0058] Obtain the SMILES string, molecular graph structure and molecular three-dimensional conformation data of the molecule, and collect the corresponding natural language text description to form a three-modal-text paired data set.

[0059] 2) Multi-modal encoding

[0060] Respectively use pre-trained one-dimensional molecular encoder (ChemBERTa), two-dimensional molecular encoder (MolGT) and three-dimensional molecular encoder (Uni-Mol2) to represent the different modal features of the molecule; use the pre-trained PubMedBERT model to encode the molecular text description to obtain the text embedding vector.

[0061] 3) Single-modal alignment stage

[0062] For each molecular modality (1D, 2D, 3D), independently perform symmetric contrastive learning with the text modality, calculate the similarity between the molecular embedding and the text embedding, and maximize the similarity between the matching samples and minimize the similarity between the non-matching samples through the InfoNCE loss, to establish the preliminary cross-modal alignment.

[0063] 4) Multi-modal consistency constraint stage

[0064] On the basis of obtaining single-modal-text alignment, introduce a consistency constraint loss to impose Euclidean distance or cosine similarity constraints on the three modal embeddings of the same molecule, so that different modalities maintain semantic consistency in the shared embedding space, avoiding representation drift or space collapse.

[0065] 5) Modal attention fusion and final alignment stage

[0066] Input the three molecular modal features after consistency constraint into the modal attention fusion module, adaptively assign weights to different modalities through the multi-head attention mechanism, generate the fused molecular representation, and perform contrastive learning with the text embedding again to realize unified semantic alignment.

[0067] 6) Dynamic hard negative sample mining mechanism

[0068] In the model training process, a step-by-step enhanced negative sample strategy is adopted, including random sampling, batch-hardmining, memory queue and adversarial negative sample generation, to improve the discriminability and convergence stability of the representation space.

[0069] The dynamic hard negative sample mining mechanism is adopted in the training process, including the following stages:

[0070] (1) Random negative sample sampling stage;

[0071] (2) Batch hard mining stage;

[0072] (3) Negative sample enhancement stage based on memory queue;

[0073] (4) Adversarial negative sample generation stage.

[0074] 7) Downstream task stage

[0075] Two types of tasks are implemented in the shared embedding space: (1) Molecule-text bidirectional retrieval task: according to the vector cosine similarity, the bidirectional matching of "finding text description given molecule" and "finding molecule structure given text" is realized; (2) Molecule attribute prediction task: under the condition of freezing the pre-training encoder, a lightweight prediction head is used for molecule classification or regression to realize property prediction and molecule optimization.

[0076] 8) System implementation module

[0077] The method can be implemented as the following modules at the system level:

[0078] Data processing module: responsible for the preprocessing and feature analysis of multi-modal molecule and text data;

[0079] Encoder module: generates multi-modal molecule embedding and text embedding;

[0080] Alignment learning module: performs step-by-step alignment, consistency constraint and modal attention fusion;

[0081] Task module: complete cross-modal retrieval, editing and property prediction.

[0082] The present application proposes an end-to-end three-modal molecule-text representation learning method based on step-by-step alignment and modal attention fusion, as shown in Figure 1 The core idea is to model one-dimensional sequence (SMILES), two-dimensional molecular graph and three-dimensional molecular conformation information of molecules at the same time, through step-by-step alignment, multi-modal consistency constraint and modal attention fusion, to build a unified molecule-text shared semantic space, to realize natural language driven molecule understanding, retrieval and attribute editing, Figure 1The framework in the figure integrates one-dimensional SMILES sequences, two-dimensional molecular graphs, and three-dimensional conformations with textual descriptions. The framework includes three stages: (1) modality alignment pre-training - aligning each molecular modality with the text; (2) multi-modal consistency learning - ensuring semantic consistency between one-dimensional / two-dimensional / three-dimensional representations; (3) attention-based fusion - obtaining unified molecular embeddings. The entire training process uses a dynamic difficult example mining contrast learning strategy, thereby achieving robust and generalizable molecular-text alignment capabilities suitable for retrieval and prediction tasks.

[0083] Step 1: Multi-modal molecular data processing module

[0084] As shown in Figure 2 , the input molecular data includes SMILES strings, two-dimensional molecular graph structures, and three-dimensional molecular conformations.

[0085] (1) Use chemical analysis tools (such as RDKit) to convert SMILES strings to molecular graph structures, where nodes represent atoms (including atomic number, charge, hybridization state, etc.) and edges represent chemical bonds (including bond type, aromaticity, bond order, etc.).

[0086] (2) Further use molecular conformation generation algorithms (such as UFF or ETKDG) to calculate three-dimensional atomic coordinate information to obtain three-dimensional molecular modality input.

[0087] (3) Embedding representation of the above 1D, 2D, and 3D molecular modalities is performed by corresponding encoders, laying the foundation for subsequent cross-modal alignment.

[0088] Step 2: Natural language encoding module

[0089] As shown in Figure 2 , the natural language description corresponding to the molecule (such as "with high solubility, low toxicity" or "contains a beta-lactam ring") is input into the PubMedBERT text encoder, and after tokenization, embedding, and Transformer encoding, the text embedding vector is output.

[0090] The text embedding vector and the molecular multi-modal embedding vector have the same dimension, which is used for subsequent contrast learning and shared space construction.

[0091] Step 3: Gradual alignment and multi-modal consistency learning module

[0092] (1) Single-modal alignment stage

[0093] As shown in Figure 2As shown, each molecular modality (1D, 2D, 3D) is compared with the text embedding using a symmetric InfoNCE loss function for contrastive learning, while using a hard negative mining mechanism to maximize the similarity of matching pairs and minimize the similarity of non-matching pairs.

[0094] The loss function is in the form of:

[0095]

[0096] wherein represents the average of all positive sample pairs , that is, the mathematical expectation of the sample distribution . and represent the structure-text pair of each molecule, and are negative samples randomly sampled from the noise distribution, represents an energy function with flexibility with and as variables, , is the function composition, and represent two linear transformations.

[0097] (2) Multi-modal consistency constraint phase

[0098] As shown in Figure 3 , to avoid semantic drift between different modalities, the invention imposes consistency constraints between the three molecular modalities:

[0099]

[0100] wherein is the sum of the consistency loss of the three molecular modalities, , and represent the molecular embedding space vectors obtained after the processing of step 2), and the loss is reduced to ensure the semantic consistency of different modalities embedded in the shared space.

[0101] Step 4: Modal attention fusion module

[0102] As shown in Figure 4 , after completing the single-modality alignment and consistency constraints, the model adaptively integrates the features of the three modalities through the modal attention fusion module.

[0103] This module uses a multi-head attention mechanism to calculate the weight coefficients of different modalities:

[0104]

[0105]

[0106] where, denote the molecular vectors of various modalities, denote the molecular vectors obtained after attention fusion, denote the attention weights of each modality, are trainable parameters.

[0107]

[0108] The loss function of the final step is the text vector and the vector after fusion of the three molecules InfoNCE contrastive loss . The fused molecular representation can dynamically adapt to the task requirements and perform contrastive learning with the text embedding again to form a unified molecular-text shared semantic space.

[0109] The model described in the application is a multi-modal molecular representation learning model, which is used to simultaneously learn the correlation between molecules in different modalities (including one-dimensional SMILES representation, two-dimensional molecular graph structure and three-dimensional conformation information) and text description.

[0110] The model as a whole is composed of three main sub-modules:

[0111] (1) Modality encoding module: 1D, 2D and 3D molecular encoders are used to extract multi-modal features, and a text encoder is used to extract molecular description text features;

[0112] (2) Modality alignment module: through symmetric InfoNCE contrastive learning loss, the features of different modalities are aligned in a common embedding space;

[0113] (3) Modality fusion module: a modality attention mechanism is used to fuse the features of each modality into a unified molecular representation, and cross-modal contrastive learning and attribute prediction are performed in the fusion space.

[0114] In the model training process, a plurality of loss functions are introduced for the above different modules to constrain the learning goals of the model in the modal alignment, consistency and fusion stages. The overall loss function of the model is:

[0115]

[0116] where, and Adjustable hyperparameters. Throughout the training process, the ultimate goal is to minimize the weighted sum of three-part loss functions. Each loss term corresponds to the key stages of modal alignment, modal consistency, fusion contrastive learning, and hard negative sample optimization, respectively. By optimizing these loss terms, the model can consider various training objectives simultaneously, thereby achieving efficient learning and optimization across modalities. In these loss functions, the weight hyperparameters and control factors can be adjusted to meet different task requirements.

[0117] Step 5: Downstream task module

[0118] (1) Text-driven molecular editing task

[0119] The present application supports the implementation of molecular attribute editing through natural language instructions. Specifically, the system first receives a source molecular structure and its corresponding text instructions, such as "reduce toxicity" or "increase polarity." The model inputs the source molecular structure and text description into the corresponding encoder to obtain molecular embedding vectors and text embedding vectors. Subsequently, fusion operations are performed on these two vectors in a shared latent space to generate a new target molecular vector. The fusion process controls the modification amplitude by adjusting a weight parameter: when the weight is small, the generated molecule is closer to the original molecular structure; when the weight is large, the generated molecule is closer to the target attribute described by the text. Finally, the target molecular vector is input into a pre-trained molecular generator or performs vector similarity retrieval in an existing molecular database to obtain candidate molecular structures that meet the text semantic requirements. Through this process, the system can achieve semantic-driven modification of molecular attributes according to natural language descriptions without retraining the generation model.

[0120] (2) Molecular-text bidirectional retrieval task

[0121] The model of the present application also supports bidirectional retrieval tasks from structure to text and from text to structure. When the input is a molecular structure, the system first calculates the embedding representation of the molecule and performs similarity calculation with the embedding vectors of multiple candidate text descriptions, and then outputs several text descriptions that are most semantically close to the molecule. Conversely, when the input is a natural language text description, the system calculates the embedding representation of the text and performs similarity comparison with the embedding of each molecule in the candidate molecule set, and outputs the molecular structure that best meets the text semantics. This bidirectional retrieval mechanism can be widely applied to drug screening, molecule discovery, pharmacodynamic analysis, and drug reutilization tasks, and can achieve efficient matching and intercommunication of molecular structure and natural language knowledge at the semantic level.

[0122] Embodiment 1: Three-modal contrastive learning

[0123] In one specific embodiment, the present application is pre-trained on about 370k molecule-text pairs, including SMILES sequences, molecular graphs, molecular conformations and text descriptions. The three molecular encoders are ChemBERTa, MolGT and Uni-Mol2, and the text encoder is PubMedBERT. During training, the AdamW optimizer (learning rate 1x10 -4 ), batch size 512, and 100 rounds of training.

[0124] The experimental results show that compared with the same type of model method, the present application improves the Top-10 accuracy rate by about 5-8% in the molecule-text bidirectional retrieval task, and the AUC is improved by an average of 2.3% in a plurality of molecular property prediction tasks.

[0125] Embodiment 2: Natural language driven molecular editing

[0126] In another embodiment, the user inputs the editing instruction "increase solubility and reduce toxicity", and the system encodes the text into an embedding vector and performs a latent operation with the source molecule embedding to obtain a new target embedding. The generation module generates a new molecular structure based on the diffusion model, which predicts that the solubility is improved by about 15% and the toxicity index is reduced by about 10%, verifying the effectiveness of the present application in the natural language guided molecular optimization task.

[0127] Embodiment 3: Cross-modal molecular retrieval

[0128] Verification is performed on the DrugBank database, and the model can retrieve structurally similar penicillin molecules according to the descriptive text "a β-lactam antibiotic effective against Gram-positive bacteria". Similarly, when a new molecular structure is input, the system can return text results describing its pharmacological action, target information and treatment category, realizing a semantic level retrieval closed loop.

[0129] The above only describes the preferred embodiments of the present application and is not used to limit the present application. Those skilled in the art can make various modifications, equivalent replacements or improvements to the present application without departing from the principles of the present application, and all such modifications, equivalent replacements or improvements shall fall within the protection scope of the present application.

Claims

1. An attention fusion-based molecular multi-modal representation method, characterized in that, The method comprises the following steps: 1) obtaining the SMILES sequence of a molecule and its corresponding natural language text description, converting the SMILES sequence into a molecular graph structure and a molecular three-dimensional conformation respectively, and constructing a three-modal-text paired data set; 2) using different encoders for encoding to obtain multi-modal molecular embedding vectors and text embedding vectors; 3) aligning each molecular modality with the text modality by single-modal contrastive learning to obtain a shared embedding space at the modality level, and imposing multi-modal consistency constraints to minimize the distance between the 1D, 2D and 3D modality embeddings of the same molecule; 4) weighting and fusing the features of the three molecular modalities through a multi-head attention mechanism to obtain a unified fused molecular shared embedding space, and finally aligning the text embedding vector; 5) based on the fused shared embedding space, performing downstream molecular-text bidirectional retrieval and molecular attribute prediction tasks to realize cross-modal semantic matching and reasoning.

2. The method of claim 1, wherein, The step 1) comprises the following steps: 1.1) converting the SMILES sequence of a molecule into a molecular graph structure using a chemical analysis tool, wherein the nodes represent atoms and the edges represent chemical bonds; 1.2) converting the SMILES sequence of a molecule into three-dimensional atoms, and calculating the three-dimensional atomic coordinate information using a molecular conformation generation algorithm to obtain a three-dimensional molecular modality.

3. The method of claim 1, wherein, The step 2) comprises the following steps: 2.1) representing the SMILES sequence of a molecule, the molecular graph structure and the molecular three-dimensional conformation through a pre-trained one-dimensional molecular encoder, a two-dimensional molecular encoder and a three-dimensional molecular encoder respectively to obtain multi-modal molecular embedding vectors; 2.2) encoding the molecular text description through a pre-trained PubMedBERT model to obtain a text embedding vector, and the text embedding vector has the same dimension as the multi-modal molecular embedding vector.

4. The method of claim 1, wherein, The step 3) comprises the following steps: 3.1) using a symmetric InfoNCE loss function to perform contrastive learning between the molecular embedding vectors of each modality and the text embedding vector, while using a difficult negative sample mining mechanism to maximize the similarity of matching pairs and minimize the similarity of non-matching pairs to obtain shared embedding spaces of different modalities; 3.2) calculating the consistency loss of the three molecular modalities and minimizing the sum to impose consistency constraints between the three molecular modalities.

5. The method of claim 1, wherein, For different modalities of the same molecule sample and their corresponding text description, define the positive sample pair as the matching pair, and the negative sample pair composed of random combinations of different samples is defined as the non-matching pair. The cosine similarity of the molecular embedding vectors of different modalities and the text embedding vector is calculated as the matching similarity, and a symmetric InfoNCE loss function is used for contrastive learning. The symmetric InfoNCE loss function is specifically: where denotes the average over all positive sample pairs denotes the average over the sample distribution takes the mathematical expectation, and denotes the structure-text pair of each molecule, and denotes a negative sample randomly sampled from the noise distribution, denotes the energy function with flexibility in and denotes the energy function with flexibility in , denotes the function composition, and denotes the two linear transformations.

6. The method of claim 1, wherein, The calculation of the consistency loss of the three molecular modalities is specifically: wherein, denotes the sum of three modal consistency losses, , and denote three molecular embedding space vectors.

7. The method of claim 1, wherein the method is based on attention fusion. The weighting and fusion of the features of the three molecular modalities through the multi-head attention mechanism are specifically: wherein, represents the molecular vector of the kth modality, represents the molecular vector obtained after attention fusion, represents the attention weight of each modality, represents the trainable parameters.

8. The method of claim 1, wherein, By loss function Contrastive learning is performed to realize alignment of the fusion molecule shared embedding space and the text embedding vector. The text vector The vector after fusion with the three molecules InfoNCE contrastive loss That is: 。 9. An attention fusion-based molecular multi-modal representation system, characterized in that, It comprises: A multi-modal molecular data processing module is configured to obtain a SMILES sequence of a molecule and a corresponding natural language text description of the molecule, convert the SMILES sequence into a molecular graph structure and a molecular three-dimensional conformation respectively, and form a three-modal-text paired data set; A natural language encoding module is configured to use different encoders to obtain multi-modal molecular embedding vectors and text embedding vectors; A step-by-step alignment and multi-modal consistency learning module is configured to realize alignment of each molecular modality and the text modality by single-modal contrastive learning, obtain a modality-level shared embedding space, and impose a multi-modal consistency constraint to minimize the distance between 1D, 2D and 3D modality embeddings of the same molecule; A modality attention fusion module is configured to perform weighted fusion of features of the three molecular modalities by a multi-head attention mechanism, obtain a unified fused molecular shared embedding space, and perform final alignment with the text embedding vector; A downstream task module is configured to perform downstream molecular-text bidirectional retrieval and molecular attribute prediction tasks based on the fused shared embedding space, and realize cross-modal semantic matching and reasoning.

Images