Three-mode comparison semantic alignment method and data processing method for chemical structure object

Abstract

The invention provides a three-mode comparison semantic alignment method and data processing method for a chemical structure object, and the method comprises the steps: obtaining an input data set which comprises a plurality of chemical structure image data, a plurality of text data and a plurality of SMILES character string data; carrying out feature extraction on the input data set, and respectively determining an image embedding vector, a text embedding vector and a molecule embedding vector; inputting an embedding vector combination formed by the image embedding vector, the text embedding vector and the molecule embedding vector into a three-mode comparison semantic alignment model, and determining whether the image embedding vector, the text embedding vector and the molecule embedding vector of the embedding vector combination correspond to the same chemical structure object or not; and finally obtaining a three-mode comparison semantic alignment result corresponding to the input data set. According to the scheme, deep alignment of chemical structure image data, text data and SMILES character string data of a chemical structure object in a unified semantic space is achieved through novel semantic alignment of three-mode comparison loss.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of chemical information, in particular, to a three-modal contrastive semantic alignment method of a chemical structure object and a data processing method. BACKGROUND

[0002] In the field of chemical informatics, there has been a long-standing challenge: how to make machines deeply understand that when a chemical entity appears in literature in different forms (visual image, natural language description, chemical symbol), it essentially points to the same concept.

[0003] In recent years, cross-modal contrastive learning frameworks have achieved great success. Representative works such as CLIP (Contrastive Language-Image Pre-training) learn a shared embedding space by training on a large number of image-text pairs, so that the vector representations of images and their text descriptions are close to each other in space. This idea has been successfully transferred to the professional field, giving birth to models such as DrugCLIP (aligning protein pocket structure and molecular structure) and MoleculeSTM (aligning molecular graph structure and text description). The core of these methods is to use a contrastive loss function to pull the distance of "positive sample pairs" (i.e. matched representations of two modalities) closer, while pushing the distance of "negative sample pairs" (i.e. unmatched representations) further apart in a batch of data.

[0004] However, these advanced existing technologies have a fundamental limitation when applied to chemical literature understanding: they are essentially bi-modal. In the actual scenario of chemical literature, a core chemical entity (for example, the compound "ellipticine") usually exists in at least three indispensable and complementary modalities: 1. Visual modality (Image, I): its two-dimensional or three-dimensional chemical structure diagram.

[0005] 2. Text modality (Text, T): natural language paragraphs describing its properties, synthesis methods, and biological activities.

[0006] 3. Symbolic / graph modality (Symbolic / Graph, G): its standardized, machine-readable SMILES string (such as Cc1c2ccc3c(c2n(C)c1)c(C)c1ccnc13), which not only represents a string, but also directly represents its molecular connection graph.

[0007] If the dual-modal contrastive learning method is simply combined two by two, for example, one model is trained to align (I, T), and another model is trained to align (G, T), it cannot be guaranteed to achieve true three-modal alignment. In high-dimensional vector space, there is a kind of "geometric ambiguity": vectors z_I and z_G can maintain high cosine similarity with vector z_T (i.e. close in angle), but z_I and z_G themselves can be far apart. This "tearing" or "inconsistency" within the semantic space will lead to a one-sided and unreliable understanding of chemical entities by the system, thereby seriously affecting the quality of subsequent knowledge graph construction and reasoning ability. SUMMARY

[0008] The purpose of the embodiments of the present application is to provide a three-modal contrastive semantic alignment method and data processing method for chemical structure objects, so as to achieve deep alignment of chemical structure image data, text data and SMILES string data of chemical structure objects in a unified semantic space through a novel three-modal contrastive loss semantic alignment.

[0009] To achieve the above purpose, the embodiments of the present application are implemented in the following way: In a first aspect, the embodiments of the present application provide a three-modal contrastive semantic alignment method for chemical structure objects, comprising: obtaining an input data set, wherein the input data set includes a plurality of chemical structure image data, a plurality of text data and a plurality of SMILES string data; performing feature extraction on the chemical structure image data, the text data and the SMILES string data in the input data set to determine an image embedding vector corresponding to each chemical structure image data, a text embedding vector corresponding to each text data and a molecular embedding vector corresponding to each SMILES string data; inputting the image embedding vector, the text embedding vector and the molecular embedding vector formed by the embedding vector combination into a three-modal contrastive semantic alignment model to determine whether the image embedding vector, the text embedding vector and the molecular embedding vector of the embedding vector combination correspond to the same chemical structure object, and finally obtain a three-modal contrastive semantic alignment result corresponding to the input data set.

[0010] In a first possible implementation manner of the first aspect, the feature extraction is performed on the chemical structure image data, the text data and the SMILES string data in the input data set to determine an image embedding vector corresponding to each chemical structure image data, a text embedding vector corresponding to each text data and a molecule embedding vector corresponding to each SMILES string data, respectively, and the feature extraction includes: for each chemical structure image data, converting the chemical structure image data into the image embedding vector by using a chemical vision encoder; for each text data, converting the text data into the text embedding vector by using a text encoder special for the biochemistry field; and for each SMILES string data, converting the SMILES string data into a molecule graph, and then converting the molecule graph into the molecule embedding vector by using a molecule graph encoder.

[0011] In a second possible implementation manner of the first aspect, the converting the chemical structure image data into the image embedding vector by using the chemical vision encoder includes: implementing, by the chemical vision encoder, pre-processing on the chemical structure image data, where the pre-processing includes standardization, chemical structure centering and contrast enhancement; performing adaptive Patch segmentation on the pre-processed chemical structure image data to segment the chemical structure image data into image blocks and embed the image blocks into a high-dimensional vector space, where the adaptive Patch segmentation adjusts the Patch size in a bond region and an atom region; adding position information and ring system identification to each image block to perform hybrid position encoding, where the hybrid position encoding includes relative position encoding and bond direction encoding, and the ring system identification reveals whether the image block belongs to a ring system; performing Transformer encoding on all the processed image blocks to extract multi-level features, where the multi-level features include atom types, bond patterns, functional groups and global structures; and performing pooling on the multi-level features to generate the image embedding vector of a target length.

[0012] In a third possible implementation manner of the first aspect, the converting the text data into the text embedding vector by using the text encoder special for the biochemistry field includes: performing text segmentation on the text data; and performing encoding processing on the text data after the text segmentation by using a text encoder special for the biochemistry field based on a Transformer-based language model architecture, and taking a [CLS] token representation as the text embedding vector, where the text embedding vector is of a target length, and the text encoder special for the biochemistry field is Sci-BERT or BERT further fine-tuned in the biochemistry field corpus.

[0013] In a fourth possible implementation manner of the first aspect, in the first possible implementation manner of the first aspect, the SMILES string data is converted into a molecular graph, and the molecular graph is converted into a molecular embedding vector by using a molecular graph encoder, and the method comprises the following steps: the SMILES string data is converted into a molecular graph by using RDKit, wherein the node label in the molecular graph is an atomic type, and the edge label is a bond type; the molecular graph is initialized by using the molecular graph encoder; the initialized molecular graph is aggregated by using a GNN, and then global pooling processing is performed, and the node embedding average value or maximum value is read out to obtain a molecular embedding vector with a target length, wherein the output layer of the GNN is set to the target length.

[0014] In a fifth possible implementation manner of the first aspect, the image embedding vector, the text embedding vector and the molecular embedding vector are combined to form an embedding vector combination, the three-modal contrastive semantic alignment model is inputted with the embedding vector combination, it is determined whether the image embedding vector, the text embedding vector and the molecular embedding vector of the embedding vector combination correspond to the same chemical structure object, and finally a three-modal contrastive semantic alignment result corresponding to the input data set is obtained, and the method comprises the following steps: based on the vector data set corresponding to the input data set, an embedding vector combination containing an image embedding vector, a text embedding vector and a molecular embedding vector is generated, wherein the vector data set contains an image embedding vector corresponding to each chemical structure image data, a text embedding vector corresponding to each text data and a molecular embedding vector corresponding to each SMILES string data in the input data set; the embedding vector combination is inputted into the three-modal contrastive semantic alignment model, the similarity S(v, t) between the image embedding vector and the text embedding vector, the similarity S(v, g) between the image embedding vector and the molecular embedding vector, and the similarity S(t, g) between the text embedding vector and the molecular embedding vector are calculated, and based on the similarity S(v, t), the similarity S(v, g) and the similarity S(t, g), it is determined whether the embedding vector combination corresponds to the same chemical structure object; if the embedding vector combination corresponds to the same chemical structure object, a mutual mapping relationship among the image embedding vector, the text embedding vector and the molecular embedding vector in the embedding vector combination is established, three-modal contrastive semantic alignment is realized, and the image embedding vector, the text embedding vector and the molecular embedding vector of the embedding vector combination are marked as processed in the vector data set, and the processed image embedding vector, text embedding vector and molecular embedding vector no longer participate in other embedding vector combinations; if the embedding vector combination does not correspond to the same chemical structure object, the next embedding vector combination is determined based on the vector data set and is inputted into the three-modal contrastive semantic alignment model for three-modal contrastive semantic alignment discrimination, until all image embedding vectors, text embedding vectors corresponding to each text data are subjected to three-modal contrastive semantic alignment or all embedding vector combinations are traversed, and finally a three-modal contrastive semantic alignment result corresponding to the input data set is obtained.

[0015] Combining the fifth possible implementation of the first aspect, in the sixth possible implementation of the first aspect, the loss function of the three-modal contrastive semantic alignment model is designed as follows: , in, For trimodal contrastive semantic fusion loss, For three-modal contrast loss, These are the weighting coefficients. This represents the three-modal cohesion loss.

[0016] Combining the sixth possible implementation of the first aspect, in the seventh possible implementation of the first aspect, the three-modal contrast loss... for: , , , , , , , in, This represents the InfoNCE loss with the image embedding vector as the anchor vector and the text embedding vector as the contrast vector. For positive sample vectors, For the first One negative sample vector, The total number of negative sample vectors. Anchor vector With positive sample vector similarity, Anchor vector With the negative sample vectors similarity, Hyperparameters are used to train the loss function and control its gradient. This represents the InfoNCE loss with text embedding vectors as anchor vectors and image embedding vectors as contrast vectors. For positive sample vectors, For the first There are n negative sample vectors, where N is the total number of negative samples. Anchor vector With positive sample vector similarity, Anchor vector With the negative sample vectors similarity, Hyperparameters are used to train the loss function and control its gradient. This represents the InfoNCE loss with the image embedding vector as the anchor vector and the molecular embedding vector as the contrast vector. For positive sample vectors, For the first One negative sample vector, The total number of negative sample vectors. Anchor vector With positive sample vector similarity, Anchor vector With the negative sample vectors similarity, Hyperparameters are used to train the loss function and control its gradient. This represents the InfoNCE loss with the molecular embedding vector as the anchor vector and the image embedding vector as the contrast vector. For positive sample vectors, For the first One negative sample vector, The total number of negative sample vectors. Anchor vector With positive sample vector similarity, Anchor vector With the negative sample vectors similarity, Hyperparameters are used to train the loss function and control its gradient. This represents the InfoNCE loss with the text embedding vector as the anchor vector and the molecular embedding vector as the contrast vector. For positive sample vectors, For the first One negative sample vector, The total number of negative sample vectors. Anchor vector With positive sample vector similarity, Anchor vector With the negative sample vectors similarity, Hyperparameters are used to train the loss function and control its gradient. This represents the InfoNCE loss with the molecular embedding vector as the anchor vector and the text embedding vector as the contrast vector. For positive sample vectors, For the first One negative sample vector, The total number of negative sample vectors. Anchor vector With positive sample vector similarity, Anchor vector With the negative sample vectors similarity, These are used to train hyperparameters, which control the gradient of the loss function.

[0017] Combining the sixth possible implementation of the first aspect, in the eighth possible implementation of the first aspect, the three-modal cohesion loss for: , in, The total number of samples in the batch. Indicates the first One sample, Indicates sample The image embedding vector, Indicates sample The text embedding vector, Indicates sample molecular embedding vector, Indicates the same sample The distance between the image embedding vector and the text embedding vector. Indicates the same sample The distance between the image embedding vector and the molecular embedding vector. Indicates the same sample The distance between the text embedding vector and the molecular embedding vector.

[0018] In a second aspect, the embodiments of the present application provide a data processing method for a multi-modal task of a chemical structure object. After the three-modal contrastive semantic alignment of the chemical structure object is performed by the three-modal contrastive semantic alignment method of the chemical structure object in the first aspect or any one of the possible implementation manners of the first aspect, the method comprises: obtaining an image embedding vector, a text embedding vector and a molecule embedding vector of each chemical structure object after the three-modal contrastive semantic alignment; for each chemical structure object: based on the image embedding vector, the text embedding vector and the molecule embedding vector of the chemical structure object, a bidirectional attention interaction is realized by an attention module to enhance the image embedding vector, the text embedding vector and the molecule embedding vector, and then the enhanced image embedding vector, the enhanced text embedding vector and the enhanced molecule embedding vector are spliced to obtain a spliced vector of the chemical structure object; the spliced vector is compressed to a target length by a fully connected layer to obtain a fusion vector of the chemical structure object, which is used for a multi-modal task of the chemical structure object, wherein the multi-modal task comprises a cross-modal retrieval task and an attribute prediction task of the chemical structure object.

[0019] Beneficial effects: The scheme designs a three-modal contrast semantic alignment model, respectively realizes adaptive Patch segmentation of the preprocessed chemical structure image data through a chemical vision encoder, divides and embeds into a high-dimensional vector space, adds position information and ring system identification to each image block, performs hybrid position coding, performs Transformer coding on all image blocks, extracts multi-level features (including atomic type, chemical bond pattern, functional group and global structure), and performs pooling to generate image embedding vectors of a target length (such as 768 dimensions); the text data segmented from the text is encoded by using a text encoder (Sci-BERT or BERT further fine-tuned in the biochemical field) based on the Transformer-based language model architecture in the biochemical field, and the [CLS] token representation is taken as the text embedding vector, and the text embedding vector is of a target length (such as 768 dimensions); the SMILES string data is converted into a molecular graph by using RDKit, the molecular graph is initialized by using a molecular graph encoder, the initialized molecular graph is aggregated by using GNN, and then global pooling processing is performed, the average value or maximum value of the node embedding is read out, and a molecular embedding vector of a target length (such as 768 dimensions) is obtained. Such embedding vector conversion can consider the characteristics of the chemical structure image data, text data and SMILES string data of the chemical structure object, and effectively convert the embedding vectors. In order to avoid the disadvantages of double-modal alignment or double-modal alignment (if the double-modal contrast learning method is simply combined in pairs, it cannot guarantee the realization of true three-modal alignment, and is easy to cause the "tearing" or "inconsistency" in the semantic space, which will cause the system to understand the chemical entity one-sidedly and unreliably), the scheme designs a three-modal contrast semantic alignment model, introduces a bidirectional contrast loss and a cohesion loss (which directly punishes the geometric dispersion degree of the three-modal embedding vectors of the same chemical entity in the semantic space, forces them to be not only close in angle but also as close as possible in distance in space, forms a tight cluster, and effectively solves the geometric ambiguity problem in high-dimensional space), and cooperates to realize the contrast learning between the sample pairs of the three modalities, so that the model can effectively shorten the "distance" between the positive sample pairs of the three modalities and lengthen the "distance" between the negative sample pairs of the three modalities, effectively avoid the "tearing" or "inconsistency" problem in the semantic space, and truly realize the three-modal semantic alignment of the chemical structure object. The three-modal contrast learning process is also the parameter learning process of the chemical vision encoder, the biochemical field special text encoder and the molecular graph encoder, the distance between the sample pairs is measured by using cosine similarity, the loss function is fed back to each encoder, and the training of the parameters of each encoder is realized.

[0020] When the three-modal contrastive semantic alignment result is applied to a downstream task, for each chemical structure object, based on the image embedding vector, the text embedding vector and the molecular embedding vector of the chemical structure object, bidirectional attention interaction can be realized through an attention module to enhance the image embedding vector, the text embedding vector and the molecular embedding vector, and then the enhanced image embedding vector, the text embedding vector and the molecular embedding vector are spliced to obtain a splicing vector of the chemical structure object. The splicing vector is compressed back to the target length through a full connection layer to obtain a fusion vector of the chemical structure object, which is used for a multi-modal task of the chemical structure object, such as a cross-modal retrieval task and an attribute prediction task of the chemical structure object. In this way, multi-modal information can be fused through splicing + projection to generate a compact unified representation, which is more suitable for application in downstream tasks.

[0021] In order to make the above objectives, characteristics and advantages of the present application more apparent, the following will describe a preferred embodiment in detail, and the accompanying drawings will be described as follows. BRIEF DESCRIPTION OF DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments of the present application. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0023] Figure 1 The flowchart of the three-modal contrastive semantic alignment method for the chemical structure object provided by the embodiments of the present application.

[0024] Figure 2 The flowchart of the data processing method for the multi-modal task of the chemical structure object provided by the embodiments of the present application. DETAILED DESCRIPTION

[0025] The technical solutions in the embodiments of the present application will be described below in combination with the drawings in the embodiments of the present application.

[0026] Please refer to Figure 1 , Figure 1 The flowchart of the three-modal contrastive semantic alignment method for the chemical structure object provided by the embodiments of the present application. In this embodiment, the three-modal contrastive semantic alignment method for the chemical structure object can include steps S11, S12 and S13.

[0027] Firstly, the electronic device can run step S11.

[0028] Step S11: obtaining an input data set, wherein the input data set includes a plurality of chemical structure image data, a plurality of text data and a plurality of SMILES string data.

[0029] In this embodiment, the electronic device can obtain an input data set, which can include a plurality of chemical structure image data, a plurality of text data and a plurality of SMILES string data. The chemical structure image data, text data and SMILES string data and the like information of the input data set can be extracted by analyzing scientific literature (such as papers, journals, patents and other literature in the field of biochemistry and pharmacy), which generally contains three modal data (chemical structure image data, text data and SMILES string data) of each chemical structure. In order to facilitate the description, this embodiment takes the three modal data of each chemical structure as an example, that is, the number of chemical structure image data, text data and SMILES string data in the input data set is not zero.

[0030] After obtaining the input data set, the electronic device can run step S12.

[0031] Step S12: Feature extraction is performed on the chemical structure image data, text data and SMILES string data in the input data set, and the image embedding vector corresponding to each chemical structure image data, the text embedding vector corresponding to each text data and the molecular embedding vector corresponding to each SMILES string data are determined.

[0032] In this embodiment, the electronic device can perform feature extraction on the chemical structure image data, text data and SMILES string data in the input data set, and determine the image embedding vector corresponding to each chemical structure image data, the text embedding vector corresponding to each text data and the molecular embedding vector corresponding to each SMILES string data.

[0033] For each chemical structure image data: The electronic device can convert the chemical structure image data into an image embedding vector using a chemical vision encoder.

[0034] For example, the electronic device can use a chemical vision encoder (the chemical vision encoder of this embodiment is ChemViT, which is a chemical vision encoder in a self-developed system of the unit) to preprocess the chemical structure image data, which includes standardization, chemical structure centralization and contrast enhancement. Then the chemical vision encoder is used to perform adaptive patch segmentation on the preprocessed chemical structure image data, which is segmented into image blocks and embedded into a high-dimensional vector space. The adaptive patch segmentation adjusts the patch size in the bond region and the atom region.

[0035] Next, position information and ring system identifiers are added to each image patch, and hybrid position encoding is performed. Hybrid position encoding includes relative position encoding and chemical bond direction encoding. The ring system identifier reveals whether the image patch belongs to a ring system (and which ring system it belongs to, for example, by using a ring system number with a set number of bits, and if it does not belong to a ring system, it is represented by a conventional all-zero number).

[0036] The processed image patches are then encoded using Transformer to extract multi-level features, including atom type, chemical bond pattern, functional groups, and global structure. Finally, the multi-level features are pooled to generate an image embedding vector of the target length (768 dimensions in this example).

[0037] The pseudocode is as follows: def chemical_visual_encoding_pipeline(raw_image): """ Chemical Image Encoding Process Input: Original chemical structure image (H, W, 3) Output: Visual embedding vector (768,) """ # Image Preprocessing image = preprocess_chemical_image(raw_image) # - Size standardized to 448x448 # - Chemical structure centralization # - Contrast Enhancement # Patch Splitting patches = create_chemical_patches(image) # - 16x16 patches, but adaptively adjusted at bonds and atoms. # - Output: (28, 28, 768) patch embeddings # Location-coded injection patches_with_pos = add_chemical_position_encoding(patches) # - Relative position encoding # - Chemical bond direction coding # - Ring system identifier # Transformer encoding for layer in transformer_layers: # Multi-head attention (chemical-aware) attn_output = chemical_multi_head_attention(patches_with_pos) # - 8 attention heads, each focusing on different chemical features # - heads 1-2: atom type identification # - heads 3-4: bond pattern # - heads 5-6: functional group detection # - heads 7-8: global structure # Feed-forward network patches_with_pos = feed_forward(attn_output) # Pooling to generate final embedding visual_embedding = chemical_aware_pooling(patches_with_pos) # - gives higher weight to regions of chemical active centers # - output: 768-dimensional vector return visual_embedding According to this, the embedding vector conversion of each chemical structure image data can be completed, and the image embedding vector of the target length (such as 768 dimensions) is obtained.

[0038] For each text data: the electronic device can convert the text data into a text embedding vector by using a biochemistry field special text encoder.

[0039] In this embodiment, Sci-BERT or a BERT variant further fine-tuned on a biochemistry field corpus is selected as the biochemistry field special text encoder (in this embodiment, the biochemistry field special text encoder constructed is ChemicalTextEncoder), which is responsible for processing the natural language description text T related to the chemical entity, capturing its deep semantic information, and outputting the text embedding vector t.

[0040] For example, an electronic device can perform text segmentation on text data, such as splitting "molecular weight 342.5" into `[molecular weight, 342.5]`. The segmented text data can be encoded using a biochemistry-specific text encoder based on a Transformer language model architecture (such as Sci-BERT, or BERT further fine-tuned on biochemistry corpora). The [CLS] token is taken as the text embedding vector, where the text embedding vector is also of the target length (such as 768 dimensions).

[0041] Based on this, the embedding vector transformation of each text data can be completed to obtain the text embedding vector of the target length (e.g., 768 dimensions).

[0042] For each SMILES string data: the electronic device can convert the SMILES string data into a molecular graph, and then use a molecular graph encoder to convert the molecular graph into a molecular embedding vector.

[0043] For example, an electronic device can use RDKit to convert SMILES string data into a molecular graph, where the node labels are of type atom and the edge labels are of type bond.

[0044] Then, the electronic device can use a molecular graph encoder (in this embodiment, the molecular graph encoder is a MolecularGraphEncoder) to initialize the molecular graph, then use a GNN (the output layer of the GNN is set to the target length) to aggregate the initialized molecular graph, then perform global pooling, and read out the average or maximum value of the node embeddings (in this embodiment, the average value is used as an example) to obtain the molecular embedding vector of the target length.

[0045] Based on this, the embedding vector transformation of each SMILES string data can be completed to obtain the molecular embedding vector of the target length (e.g., 768 dimensions).

[0046] After completing the embedding vector conversion of each chemical structure image data, text data, and SMILES string data, the electronic device can run step S13.

[0047] Step S13: Input the combination of image embedding vector, text embedding vector and molecular embedding vector into the trimodal contrast semantic alignment model to determine whether the image embedding vector, text embedding vector and molecular embedding vector of the embedding vector combination correspond to the same chemical structure object, and finally obtain the trimodal contrast semantic alignment result corresponding to the input dataset.

[0048] In this embodiment, step S13 mainly relies on the trimodal contrastive semantic alignment model. For ease of understanding, the architecture design and training method of the trimodal contrastive semantic alignment model will be introduced here first.

[0049] To overcome the drawbacks of traditional bimodal semantic alignment methods applied to trimodal semantic alignment in the field of biochemistry, this embodiment provides a trimodal contrastive semantic alignment model. By constructing a pairwise bidirectional contrastive learning method, it ensures the alignment of image embedding vectors, text embedding vectors, and molecular embedding vectors of the same chemical structure object in a high-dimensional semantic space. At the same time, it introduces cohesion constraints to ensure that the image embedding vectors, text embedding vectors, and molecular embedding vectors are close in the feature space.

[0050] During the training process, multiple training processes are constructed. Each training process contains training data for B samples (i.e., chemical structure objects). The training data for each sample includes positive sample pairs (i.e., different modal data of the same sample) and N negative sample pairs.

[0051] The loss function of the three-modal contrastive semantic alignment model is designed as follows: , (1) in, For trimodal contrastive semantic fusion loss, For three-modal contrast loss, These are the weighting coefficients. This represents the three-modal cohesion loss.

[0052] Three-modal contrast loss for: (2) , (3) in, This represents the InfoNCE loss with the image embedding vector as the anchor vector and the text embedding vector as the contrast vector. For positive sample vectors, For the first One negative sample vector, The total number of negative sample vectors. Anchor vector With positive sample vector similarity, Anchor vector With the negative sample vectors similarity, These are used to train hyperparameters, which control the gradient of the loss function.

[0053] , (4) in, This represents the InfoNCE loss with text embedding vectors as anchor vectors and image embedding vectors as contrast vectors. For positive sample vectors, For the first There are n negative sample vectors, where N is the total number of negative samples. Anchor vector With positive sample vector similarity, Anchor vector With the negative sample vectors similarity, These are used to train hyperparameters, which control the gradient of the loss function.

[0054] , (5) in, This represents the InfoNCE loss with the image embedding vector as the anchor vector and the molecular embedding vector as the contrast vector. For positive sample vectors, For the first One negative sample vector, The total number of negative sample vectors. Anchor vector With positive sample vector similarity, Anchor vector With the negative sample vectors similarity, These are used to train hyperparameters, which control the gradient of the loss function.

[0055] , (6) in, This represents the InfoNCE loss with the molecular embedding vector as the anchor vector and the image embedding vector as the contrast vector. For positive sample vectors, For the first One negative sample vector, The total number of negative sample vectors. Anchor vector With positive sample vector similarity, Anchor vector With the negative sample vectors similarity, These are used to train hyperparameters, which control the gradient of the loss function.

[0056] , (7) in, This represents the InfoNCE loss with the text embedding vector as the anchor vector and the molecular embedding vector as the contrast vector. For positive sample vectors, For the first One negative sample vector, The total number of negative sample vectors. Anchor vector With positive sample vector similarity, Anchor vector With the negative sample vectors similarity, These are used to train hyperparameters, which control the gradient of the loss function.

[0057] , (8) in, This represents the InfoNCE loss with the molecular embedding vector as the anchor vector and the text embedding vector as the contrast vector. For positive sample vectors, For the first One negative sample vector, The total number of negative sample vectors. Anchor vector With positive sample vector similarity, Anchor vector With the negative sample vectors similarity, These are used to train hyperparameters, which control the gradient of the loss function.

[0058] It should be noted that the similarity involved in the above loss function is cosine similarity. When different types of embedding vectors are used as anchor vectors, the number of corresponding negative sample pairs N, and the training hyperparameters... "Etc" can be different.

[0059] Trimodal cohesion loss for: , (9) in, The total number of samples in the batch. Indicates the first One sample, Indicates sample The image embedding vector, Indicates sample The text embedding vector, Indicates sample molecular embedding vector, Indicates the same sample The distance between the image embedding vector and the text embedding vector. Indicates the same sample The distance between the image embedding vector and the molecular embedding vector. Indicates the same sample The distance between the text embedding vector and the molecular embedding vector.

[0060] To avoid the drawbacks of bimodal alignment or bimodal pairwise alignment (simply combining bimodal contrastive learning methods pairwise does not guarantee true trimodal alignment, easily leading to "tears" or "inconsistencies" within the semantic space, resulting in a one-sided and unreliable understanding of chemical entities by the system), this scheme designs a trimodal contrastive semantic alignment model. It introduces bidirectional contrastive loss and cohesion loss (the cohesion loss term directly penalizes the geometric dispersion of the three modal embedding vectors of the same chemical entity in the semantic space, forcing them to not only have similar angles but also minimize their distance, forming a compact cluster, effectively solving the geometric ambiguity problem in high-dimensional space). This collaboratively achieves contrastive learning between trimodal sample pairs, enabling the model to effectively shorten the "distance" between positive trimodal sample pairs and widen the "distance" between negative trimodal sample pairs, effectively avoiding "tears" or "inconsistencies" within the semantic space. This truly achieves trimodal semantic alignment of chemical structure objects and can be well applied to the multimodal representation of chemical structure objects extracted from scientific literature, thereby improving the system's accuracy in understanding scientific literature.

[0061] The trimodal contrastive learning process is also the parameter learning process for chemical visual encoders, biochemical text encoders, and molecular graph encoders. Cosine similarity can be used to measure the distance between sample pairs. The training objective is to minimize our loss function, which is then fed back to each encoder to train its parameters. Ideally, when calculating cosine similarity, the distance between negative sample pairs is -1, while the distance between positive sample pairs is 1. However, in reality, we can only approach the ideal state as closely as possible, not completely achieve it. Therefore, when using the trimodal contrastive semantic alignment model for trimodal semantic alignment, multiple validations are performed using a validation set (preferably split into at least 10 validation sets) to determine the cosine similarity threshold that satisfies the alignment accuracy (e.g., accuracy exceeding 85%). After removing the highest and lowest values, the mean is calculated to obtain the cosine similarity judgment threshold.

[0062] After the three-modal contrastive semantic alignment model is trained, the electronic device can input the embedding vector combination formed by the image embedding vector, the text embedding vector and the molecule embedding vector into the three-modal contrastive semantic alignment model, determine whether the image embedding vector, the text embedding vector and the molecule embedding vector of the embedding vector combination correspond to the same chemical structure object, and finally obtain the three-modal contrastive semantic alignment result corresponding to the input data set.

[0063] For example, the electronic device can generate an embedding vector combination containing one image embedding vector, one text embedding vector and one molecule embedding vector based on the vector data set corresponding to the input data set, wherein the vector data set contains the image embedding vector corresponding to each chemical structure image data, the text embedding vector corresponding to each text data and the molecule embedding vector corresponding to each SMILES string data in the input data set.

[0064] The embedding vector combination is input into the three-modal contrastive semantic alignment model, and the cosine similarity S(v, t) between the image embedding vector and the text embedding vector, the cosine similarity S(v, g) between the image embedding vector and the molecule embedding vector, and the cosine similarity S(t, g) between the text embedding vector and the molecule embedding vector are calculated. Based on the similarities S(v, t), S(v, g) and S(t, g), it is determined whether the embedding vector combination corresponds to the same chemical structure object. For example, the similarity threshold condition is satisfied, such as S(v, t)>0.75, S(v, g)>0.85 and S(t, g)>0.80 (these thresholds are for illustration only, and the actual threshold is based on the mean value calculated by the verification set). Generally speaking, using threshold judgment will not cause multiple-to-one situation, but it cannot be completely avoided. In order to further improve the alignment accuracy, all embedding vector combinations can be calculated, and then the embedding vectors of other modalities aligned with the embedding vectors of a certain modality are screened through the threshold. If there are multiple embedding vectors that meet the condition, they are sorted, and the embedding vector with the highest similarity in each modality is selected as the embedding vector that is semantically aligned with the embedding vector of that modality.

[0065] The embedding vector combination (containing one image embedding vector, one text embedding vector and one molecule embedding vector) that is semantically aligned with each other, i.e., corresponds to the same chemical structure object. Accordingly, the electronic device can establish the mutual mapping relationship of the image embedding vector, the text embedding vector and the molecule embedding vector in the embedding vector combination, and realize three-modal contrastive semantic alignment. Moreover, the image embedding vector, the text embedding vector and the molecule embedding vector of this embedding vector combination are marked as processed in the vector data set, and the processed image embedding vector, text embedding vector and molecule embedding vector no longer participate in other embedding vector combinations.

[0066] If the image embedding vector, the text embedding vector and the molecule embedding vector in the embedding vector combination do not correspond to the same chemical structure object, the next set of embedding vector combinations is determined based on the vector dataset and input into the three-modal contrastive semantic alignment model for three-modal contrastive semantic alignment discrimination until all image embedding vectors, text embedding vectors corresponding to each text data, three-modal contrastive semantic alignment of the embedding vector combination is completed or all embedding vector combinations are traversed, and a three-modal contrastive semantic alignment result corresponding to the input dataset is obtained.

[0067] Accordingly, the present embodiment can realize alignment of chemical structure image data, text data and SMILES string data in the input dataset through the three-modal contrastive semantic alignment model.

[0068] The present scheme designs a three-modal contrastive semantic alignment model, which utilizes a chemical vision encoder, a biochemistry field special text encoder and a molecule graph encoder in the system to realize corresponding embedding vector conversion. Such an embedding vector conversion manner can consider the characteristics of chemical structure image data, text data and SMILES string data of a chemical structure object and effectively convert the embedding vectors. The three-modal contrastive semantic alignment model introduces bidirectional contrastive loss and cohesion loss (which directly punishes the geometric dispersion degree of three-modal embedding vectors of the same chemical entity in the semantic space, forces them to be not only close in angle but also as close as possible in distance in the space, forms a tight cluster, and effectively solves the geometric ambiguity problem in high-dimensional space), and cooperates to realize contrastive learning between sample pairs of three modalities, so that the model can effectively shorten the distance between positive sample pairs of three modalities and lengthen the distance between negative sample pairs of three modalities, effectively avoid the "tearing" or "inconsistency" problem in the semantic space, and truly realize three-modal semantic alignment of a chemical structure object. The present scheme can be well applied to multi-modal representation of a chemical structure object extracted from scientific literature, thereby improving the understanding accuracy of the system for scientific literature.

[0069] On the basis of three-modal contrastive semantic alignment, the present embodiment further provides a data processing method for a multi-modal task of a chemical structure object, as shown in Figure 2 After the three-modal contrastive semantic alignment of a chemical structure object is performed by the three-modal contrastive semantic alignment method of the present embodiment, the electronic device can run step S21.

[0070] Step S21: Obtain the image embedding vector, the text embedding vector and the molecule embedding vector of each chemical structure object subjected to three-modal contrastive semantic alignment.

[0071] In this embodiment, the electronic device can obtain the image embedding vector, the text embedding vector and the molecule embedding vector of each chemical structure object subjected to three-modal contrast semantic alignment. After obtaining the image embedding vector, the text embedding vector and the molecule embedding vector of each chemical structure object, the electronic device can perform step S22.

[0072] Step S22: For each chemical structure object: based on the image embedding vector, the text embedding vector and the molecule embedding vector of the chemical structure object, bidirectional attention interaction is realized through an attention module, the image embedding vector, the text embedding vector and the molecule embedding vector are enhanced, the enhanced image embedding vector, the enhanced text embedding vector and the enhanced molecule embedding vector are spliced to obtain a spliced vector of the chemical structure object, the spliced vector is compressed back to a target length through a fully connected layer to obtain a fusion vector of the chemical structure object, which is used for a multi-modal task of the chemical structure object, wherein the multi-modal task includes a cross-modal retrieval task and an attribute prediction task of the chemical structure object.

[0073] In this embodiment, the electronic device can realize bidirectional attention interaction through an attention module based on the image embedding vector, the text embedding vector and the molecule embedding vector of the chemical structure object, and enhance the image embedding vector, the text embedding vector and the molecule embedding vector.

[0074] The attention module is designed as a six-unit attention module to realize bidirectional attention interaction. v2t: vision to text (image guides text modeling) v2g: vision to molecule (image guides molecule graph modeling) t2g: text to molecule (text guides molecule graph modeling) t2v: text to vision (text guides image modeling) g2v: molecule to vision (molecule graph guides image modeling) g2t: molecule to text (molecule graph guides text modeling) The formula of the attention module is: , wherein, represents a query vector, i.e., a target embedding vector that needs to be updated at present, is a key vector, representing a source embedding vector that provides information, is a value vector, representing a source embedding vector actually used to transfer information (which can be understood as a source embedding vector after linear transformation), represents the dimension of the key vector, and in this embodiment, 768 is taken as an example. Taking the v2t attention unit as an example, the image guides text modeling, then the text embedding vector is the query vector , indicating that it needs to be enhanced by the image embedding vector, and the image embedding vector is the key vector value vector It can be understood that the image embedding vector (key vector ) needs to be transmitted to the text embedding vector (query vector ) after linear transformation, and the is the dimension of the key vector, since the text embedding vector and the image embedding vector are both 768 dimensions, no dimension transformation is needed, 768 dimensions.

[0075] The original embedding vector of each modality is added to the output of the corresponding attention unit to realize the enhancement of the embedding vector by the attention unit.

[0076] Then, the electronic device can concatenate the enhanced image embedding vector, text embedding vector and molecular embedding vector to obtain the concatenation vector of the chemical structure object. Then, the concatenation vector is compressed back to the target length through the full connection layer to obtain the fusion vector of the chemical structure object, which can be used for the multi-modal task of the chemical structure object. The multi-modal task is, for example, the cross-modal retrieval task and the attribute prediction task of the chemical structure object, and the main task is still to build a knowledge graph, which can be used as the basis for various downstream tasks.

[0077] The pseudo code of this process is as follows (the encoding process of each modality data in the pseudo code does not need to be repeated in this embodiment, and the aligned embedding vector combination can be directly obtained, but there are other scenarios, for example, obtaining three modal data that have been aligned, but there is no corresponding embedding vector, then encoding is needed): class TriModalAlignmentModel(nn.Module): def __init__(self): super().__init__() # Three encoders self.visual_encoder = ChemicalVisualEncoder() # Output: [B, 768] self.text_encoder = ChemicalTextEncoder() # Output: [B, 768] self.graph_encoder = MolecularGraphEncoder() # Output: [B, 768] # Cross-modal fusion layer self.cross_modal_attention = nn.ModuleDict({ 'v2t': CrossModalAttention(768, 768), # Visual to Text 'v2g': CrossModalAttention(768, 768), # Visual to Image 't2g': CrossModalAttention(768, 768), # Text to Image 't2v': CrossModalAttention(768, 768), # Text to Vision 'g2v': CrossModalAttention(768, 768), # Image to Vision 'g2t': CrossModalAttention(768, 768), # Image to Text }) # Modal Fusion Projection self.fusion_projection = nn.Linear(768 * 3, 768) def forward(self, images, texts, graphs): # Independent Encoding v = self.visual_encoder(images) t = self.text_encoder(texts) g = self.graph_encoder(graphs) # Cross-modal attention # Each modality focuses on the other two modalities. v_attended = v + self.cross_modal_attention['t2v'](t, v) + \ self.cross_modal_attention['g2v'](g, v) t_attended = t + self.cross_modal_attention['v2t'](v, t) + \ self.cross_modal_attention['g2t'](g, t) g_attended = g + self.cross_modal_attention['v2g'](v, g) + \ self.cross_modal_attention['t2g'](t, g) # Fusion projection fused = torch.cat([v_attended, t_attended, g_attended], dim=-1) unified_repr = self.fusion_projection(fused) return v_attended, t_attended, g_attended, unified_repr According to this, the data processing of applying the three-modal contrast semantic alignment result to the downstream task can be completed. In this way, the multi-modal information can be fused through splicing + projection to generate a compact unified representation, which is more suitable for application to the downstream task.

[0078] In summary, the embodiment of the present application provides a three-modal contrast semantic alignment method and data processing method for chemical structure objects. The chemical visual encoder is used to realize adaptive Patch segmentation of the preprocessed chemical structure image data, the image blocks are segmented and embedded into a high-dimensional vector space, the position information and ring system identification are added to each image block, the mixed position coding is performed, the Transformer coding is performed on all image blocks, the multi-level features (including atomic type, chemical bond pattern, functional group and global structure) are extracted, and the image embedding vector of the target length (such as 768 dimensions) is generated by pooling. The biological and chemical field special text encoder (Sci-BERT or BERT further fine-tuned on the biological and chemical field corpus) based on the Transformer language model architecture is used to code and process the text data segmented from the text, and the [CLS] token representation is taken as the text embedding vector. The text embedding vector is of the target length (such as 768 dimensions). The SMILES string data is converted into a molecular graph by using RDKit, the molecular graph encoder is used to realize the initialization of the molecular graph, the GNN is used to aggregate the initialized molecular graph, and the global pooling processing is performed, the node embedding average value or maximum value is read out, and the molecular embedding vector of the target length (such as 768 dimensions) is obtained. Such embedding vector conversion mode can consider the characteristics of the chemical structure image data, the text data and the SMILES string data of the chemical structure object, and effectively convert the embedding vectors. In order to avoid the disadvantages of double-modal alignment or double-modal alignment (if the double-modal contrast learning method is simply combined in pairs, it cannot guarantee the realization of true three-modal alignment, and is easy to cause the “tearing” or “inconsistency” in the semantic space, which will cause the system to understand the chemical entity in a one-sided and unreliable manner), the three-modal contrast semantic alignment model is designed, the bidirectional contrast loss and the cohesion loss (which directly punishes the geometric dispersion degree of the three-modal embedding vectors of the same chemical entity in the semantic space, forces them to be close in angle and as close as possible in distance in the space, forms a tight cluster, and effectively solves the geometric ambiguity problem in the high-dimensional space) are introduced, and the contrast learning between the sample pairs of the three modalities is cooperatively realized. The model can effectively reduce the “distance” between the positive sample pairs of the three modalities, and increase the “distance” between the negative sample pairs of the three modalities, effectively avoid the “tearing” or “inconsistency” problem in the semantic space, truly realize the three-modal semantic alignment of the chemical structure object, and can be well applied to the multi-modal representation of the chemical structure object extracted from the scientific literature, thereby improving the understanding accuracy of the system for the scientific literature. The process of three-modal contrast learning is also the parameter learning process of the chemical visual encoder, the biological and chemical field special text encoder and the molecular graph encoder. The cosine similarity is used to measure the distance between the sample pairs, and the loss function is fed back to each encoder to realize the training of the parameters of each encoder.

[0079] In this document, relational terms such as first and second and the like can be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions.

[0080] The above description is merely illustrative of the application and not intended to be limiting. It will thus be appreciated that those skilled in the art will be able to devise various arrangements that, although not explicitly described or shown herein, embody the principles of the application and are thus within its spirit and scope. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the application should be included in the scope of the application.

Claims

1. A three-modal contrastive semantic alignment method for chemical structure objects, characterized in that, include: Obtain the input dataset, which includes several chemical structure image data, several text data, and several SMILES string data; Feature extraction is performed on the chemical structure image data, text data, and SMILES string data in the input dataset to determine the image embedding vector corresponding to each chemical structure image data, the text embedding vector corresponding to each text data, and the molecular embedding vector corresponding to each SMILES string data. The embedding vectors formed by image embedding vectors, text embedding vectors, and molecular embedding vectors are input into the trimodal contrastive semantic alignment model to determine whether the image embedding vectors, text embedding vectors, and molecular embedding vectors of the embedding vector combination correspond to the same chemical structure object. Finally, the trimodal contrastive semantic alignment result corresponding to the input dataset is obtained.

2. The three-modal contrastive semantic alignment method for chemical structure objects according to claim 1, characterized in that, Feature extraction is performed on the chemical structure image data, several text data, and several SMILES string data in the input dataset. The image embedding vector for each chemical structure image data, the text embedding vector for each text data, and the molecular embedding vector for each SMILES string data are determined, including: For each chemical structure image data: the chemical structure image data is converted into an image embedding vector using a chemical visual encoder; For each piece of text data: the text data is converted into a text embedding vector using a text encoder specifically designed for the field of biochemistry; For each SMILES string data: convert the SMILES string data into a molecular graph, and then use a molecular graph encoder to convert the molecular graph into a molecular embedding vector.

3. The three-modal contrastive semantic alignment method for chemical structure objects according to claim 2, characterized in that, Chemical structure image data is converted into image embedding vectors using a chemical visual encoder, including: Achieved through a chemical visual encoder: The chemical structure image data is preprocessed, including normalization, chemical structure centering, and contrast enhancement. The preprocessed chemical structure image data is subjected to adaptive patch segmentation, which divides the data into image patches and embeds them into a high-dimensional vector space. The adaptive patch segmentation adjusts the patch size in the bond region and the atom region. Location information and ring system identifiers are added to each image patch for hybrid location coding, which includes relative location coding and chemical bond orientation coding. The ring system identifier reveals whether the image patch belongs to a ring system. Transformer encoding is performed on all processed image patches to extract multi-level features, including atom type, chemical bond pattern, functional group and global structure; Pooling is performed on multi-level features to generate image embedding vectors of the target length.

4. The three-modal contrastive semantic alignment method for chemical structure objects according to claim 2, characterized in that, Text data is converted into text embedding vectors using a text encoder specific to the field of biochemistry, including: Perform text segmentation on text data; A biochemistry-specific text encoder based on a Transformer-based language model architecture is used to encode the segmented text data. The [CLS] token is used as the text embedding vector, where the text embedding vector is the target length. The biochemistry-specific text encoder is Sci-BERT or a BERT further fine-tuned on biochemistry corpora.

5. The three-modal contrastive semantic alignment method for chemical structure objects according to claim 2, characterized in that, The SMILES string data is converted into a molecular graph, and then the molecular graph is converted into molecular embedding vectors using a molecular graph encoder, including: Use RDKit to convert SMILES string data into a molecular graph, where node labels are of type atom and edge labels are of type bond. Implemented using a molecular graph encoder: Initialize the molecular diagram; The initial molecular graph is aggregated using GNN, then global pooling is performed, and the average or maximum value of the node embeddings is read out to obtain the molecular embedding vector of the target length. The output layer of GNN is set to the target length.

6. The three-modal contrastive semantic alignment method for chemical structure objects according to claim 1, characterized in that, The embedding vectors formed by image embedding vectors, text embedding vectors, and molecular embedding vectors are input into a three-modal contrastive semantic alignment model to determine whether the image embedding vectors, text embedding vectors, and molecular embedding vectors in the embedding vector combination correspond to the same chemical structure object. The final result is the three-modal contrastive semantic alignment of the input dataset, including: Based on the vector dataset corresponding to the input dataset, generate an embedding vector combination containing an image embedding vector, a text embedding vector, and a molecular embedding vector. The vector dataset contains the image embedding vector corresponding to each chemical structure image data, the text embedding vector corresponding to each text data, and the molecular embedding vector corresponding to each SMILES string data in the input dataset. The embedding vector combination is input into the trimodal contrastive semantic alignment model to calculate the similarity S(v,t) between the image embedding vector and the text embedding vector, the similarity S(v,g) between the image embedding vector and the molecular embedding vector, and the similarity S(t,g) between the text embedding vector and the molecular embedding vector. Based on the similarity S(v,t), S(v,g) and similarity S(t,g), it is determined whether the embedding vector combination corresponds to the same chemical structure object. If the embedding vector combination corresponds to the same chemical structure object, establish the mutual mapping relationship between the image embedding vector, text embedding vector and molecular embedding vector in the embedding vector combination to achieve trimodal contrast semantic alignment. In the vector dataset, the image embedding vector, text embedding vector and molecular embedding vector of this embedding vector combination are marked as processed. The processed image embedding vector, text embedding vector and molecular embedding vector will no longer participate in other embedding vector combinations. If they do not correspond to the same chemical structure object, the next set of embedding vector combinations is determined based on the vector dataset and input into the trimodal contrast semantic alignment model for trimodal contrast semantic alignment discrimination, until the trimodal contrast semantic alignment of all image embedding vectors and the text embedding vectors corresponding to each text data is completed, or all embedding vector combinations are traversed, and the trimodal contrast semantic alignment result corresponding to the input dataset is obtained.

7. The three-modal contrastive semantic alignment method for chemical structure objects according to claim 6, characterized in that, The loss function of the three-modal contrastive semantic alignment model is designed as follows: ， in, For trimodal contrastive semantic fusion loss, For three-modal contrast loss, These are the weighting coefficients. This represents the three-modal cohesion loss.

8. The three-modal contrastive semantic alignment method for chemical structure objects according to claim 7, characterized in that, Three-modal contrast loss for: ， ， ， ， ， ， ， in, This represents the InfoNCE loss with the image embedding vector as the anchor vector and the text embedding vector as the contrast vector. For positive sample vectors, For the first One negative sample vector, The total number of negative sample vectors. Anchor vector With positive sample vector similarity, Anchor vector With the negative sample vectors similarity, Hyperparameters are used to train the loss function and control its gradient. This represents the InfoNCE loss with text embedding vectors as anchor vectors and image embedding vectors as contrast vectors. For positive sample vectors, For the first There are n negative sample vectors, where N is the total number of negative samples. Anchor vector With positive sample vector similarity, Anchor vector With the negative sample vectors similarity, Hyperparameters are used to train the loss function and control its gradient. This represents the InfoNCE loss with the image embedding vector as the anchor vector and the molecular embedding vector as the contrast vector. For positive sample vectors, For the first One negative sample vector, The total number of negative sample vectors. Anchor vector With positive sample vector similarity, Anchor vector With the negative sample vectors similarity, Hyperparameters are used to train the loss function and control its gradient. This represents the InfoNCE loss with the molecular embedding vector as the anchor vector and the image embedding vector as the contrast vector. For positive sample vectors, For the first One negative sample vector, The total number of negative sample vectors. Anchor vector With positive sample vector similarity, Anchor vector With the negative sample vectors similarity, Hyperparameters are used to train the loss function and control its gradient. This represents the InfoNCE loss with the text embedding vector as the anchor vector and the molecular embedding vector as the contrast vector. For positive sample vectors, For the first One negative sample vector, The total number of negative sample vectors. Anchor vector With positive sample vector similarity, Anchor vector With the negative sample vectors similarity, Hyperparameters are used to train the loss function and control its gradient. This represents the InfoNCE loss with the molecular embedding vector as the anchor vector and the text embedding vector as the contrast vector. For positive sample vectors, For the first One negative sample vector, The total number of negative sample vectors. Anchor vector With positive sample vector similarity, Anchor vector With the negative sample vectors similarity, These are used to train hyperparameters, which control the gradient of the loss function.

9. The three-modal contrastive semantic alignment method for chemical structure objects according to claim 7, characterized in that, Three-modal cohesive loss for: ， in, The total number of samples in the batch. Indicates the first One sample, Indicates sample The image embedding vector, Indicates sample The text embedding vector, Indicates sample molecular embedding vector, Indicates the same sample The distance between the image embedding vector and the text embedding vector. Indicates the same sample The distance between the image embedding vector and the molecular embedding vector. Indicates the same sample The distance between the text embedding vector and the molecular embedding vector.

10. A data processing method for multimodal tasks involving chemical structure objects, characterized in that, After performing trimodal contrastive semantic alignment of a chemical structure object using the trimodal contrastive semantic alignment method of any one of claims 1-9, the method includes: Obtain the image embedding vector, text embedding vector, and molecular embedding vector for each chemical structure object after trimodal contrast semantic alignment; For each chemical structure object: Based on the image embedding vector, text embedding vector, and molecular embedding vector of the chemical structure object, a bidirectional attention interaction is achieved through an attention module to enhance the image embedding vector, text embedding vector, and molecular embedding vector. Then, the enhanced image embedding vector, text embedding vector, and molecular embedding vector are concatenated to obtain the concatenated vector of the chemical structure object. The concatenated vector is compressed back to the target length through a fully connected layer to obtain the fusion vector of the chemical structure object, which is used for multimodal tasks of chemical structure objects. Among them, multimodal tasks include cross-modal retrieval tasks and attribute prediction tasks of chemical structure objects.

Images