Methods, systems, devices and media for cross-modal molecular information retrieval

Abstract

The application discloses a cross-modal molecular information retrieval method, system, device and medium, wherein the application adopts a multi-modal fusion method, calculates a contrast loss through a modal extraction encoder to obtain single-modal representations of graph structure, image and text data of molecular information, so that the graph structure information and the image information are aligned to supplement and enrich the text information, and calculates a fusion loss through a modal fusion encoder to obtain multi-modal representations of the molecular information, so that the fusion of different modal data such as the graph structure, the image and the text is realized, and then in the application process, cross-modal retrieval of molecules can be performed based on the multi-modal representations of the molecular information in a molecular database according to target modal data and a target query condition of target molecular information, the demand query of multi-modal information processing is met, and therefore the accuracy and efficiency of molecular information retrieval are improved.

Description

Technical Field

[0001] This invention relates to the field of data retrieval technology, and in particular to a cross-modal molecular information retrieval method, system, device, and medium. Background Technology

[0002] In recent years, the rapid development of information technology and big data has had a profound impact on the field of data retrieval. In the field of molecular information, traditional data retrieval methods mostly rely on a single data modality, such as text, image, or graph structure. These methods perform well when processing simple queries. For example, graph structure can effectively reveal complex relationship networks between entities, images can provide intuitive and rich visual information, while text can describe specific concepts and facts in detail.

[0003] However, when dealing with complex queries and improving information retrieval efficiency, single-modal data sources have limitations in expressing information. Different data types, such as text, images, and graph structures, each have their own unique ways of expressing information and advantages. Therefore, when processing queries that require multimodal information, single-modal molecular information retrieval systems often fail to achieve ideal results and may even lead to information omissions or decreased retrieval accuracy. Summary of the Invention

[0004] The following is an overview of the subject matter described in detail herein. This overview is not intended to limit the scope of the claims.

[0005] This invention provides a cross-modal molecular information retrieval method, electronic device, and medium, aiming to solve the technical problems of poor retrieval performance and difficulty in handling multimodal information query requirements in existing single-modal molecular information retrieval systems. It can effectively integrate data from different modalities such as graph structures, images, and text, thereby improving the accuracy and efficiency of molecular information retrieval.

[0006] In a first aspect, embodiments of this application provide a cross-modal molecular information retrieval method, including:

[0007] Multimodal data characterizing molecular information is obtained from a preset dataset, including text data, image data, and graph structure data;

[0008] A multimodal molecular information model is constructed, comprising a modality extraction encoder and a modality fusion encoder. The modality extraction encoder is used to obtain multiple single-modality representations based on the multimodal data, and the modality fusion encoder is used to obtain a multimodal representation based on the multiple single-modality representations.

[0009] The multimodal data is input into the multimodal molecular information model, the contrast loss corresponding to multiple single-modal representations is calculated, and the modality extraction encoder is trained and optimized based on the contrast loss; and the fusion loss corresponding to the multimodal representations is calculated, and the modality fusion encoder is trained and optimized based on the fusion loss;

[0010] The target modality data and target query conditions of the target molecule information are obtained, and the target modality data is input into the multimodal molecular information model to obtain the multimodal retrieval results corresponding to the target modality data and the target query conditions.

[0011] In some embodiments, the modality extraction encoder includes a text encoder, an image encoder, and a graph structure encoder; the unimodal representation includes text modality representation, image modality representation, and graph structure modality representation; and obtaining multiple unimodal representations based on the multimodal data includes:

[0012] The text data is input into the text encoder to obtain the text modal representation;

[0013] The image data is input into the image encoder to obtain the image modal representation;

[0014] The graph structure data is input into the graph structure encoder to obtain the graph structure modal representation.

[0015] In some embodiments, calculating the contrastive loss corresponding to the plurality of the single-modal representations, and training and optimizing the modality extraction encoder based on the contrastive loss, includes:

[0016] Positive sample pairs are generated based on the text data, image data, and graph structure data that represent the same molecular information;

[0017] Negative sample pairs are generated based on the text data, image data, and graph structure data that characterize different molecular information;

[0018] The positive sample pairs and the negative samples are substituted into a preset triplet loss function to calculate the first loss and the second loss respectively. The first loss represents the contrastive learning loss between the text modality representation and the image modality representation, and the second loss represents the contrastive learning loss between the text modality representation and the graph structure modality representation.

[0019] The sum of the first loss and the second loss is determined as the contrast loss, and the modality extraction encoder is trained and optimized with the goal of minimizing the contrast loss.

[0020] In some embodiments, the modality fusion encoder includes multiple cross-attention layers, and obtaining a multimodal representation based on multiple single-modal representations includes:

[0021] The text modality representation, the image modality representation, and the graph structure modality representation, which represent the same molecular information, are input as a modality group into the multi-layered cross-attention layer;

[0022] The fusion weight coefficient group corresponding to the modality group is calculated by using the preset normalized exponential function in the multi-layer cross-attention layer;

[0023] The multimodal representation is obtained by weighting each modal representation in the modal group according to the fusion weight coefficient group.

[0024] In some embodiments, calculating the fusion loss corresponding to the multimodal representation, and training and optimizing the modal fusion encoder based on the fusion loss, includes:

[0025] Calculate the representation similarity between the text modal representation, the image modal representation, the graph structure modal representation, and the multimodal representation;

[0026] The fusion loss is calculated based on the representation similarity, and the fusion weight coefficient group in the modal fusion encoder is updated with the goal of minimizing the fusion loss.

[0027] In some embodiments, inputting the target modality data into the multimodal molecular information model to obtain the multimodal retrieval results corresponding to the target modality data and the target query conditions includes:

[0028] The target modal data is input into the multimodal molecular information model to obtain the target modal representation corresponding to the target modal data. The target modal representation includes a target single-modal representation and a target multimodal representation.

[0029] Calculate the similarity score between the modality representations corresponding to multiple multimodal data in the preset dataset and the target modality representation;

[0030] The modal representations corresponding to the multiple multimodal data are sorted according to the similarity scores, and the multimodal retrieval results are obtained based on the sorting results and the target query conditions.

[0031] In some embodiments, calculating the similarity score between the modality representations corresponding to multiple multimodal data in the preset dataset and the target modality representation includes:

[0032] Multiple feature pairs are determined between the modality representation corresponding to the multimodal data in the preset dataset and the target modality representation;

[0033] Based on a pre-set molecular knowledge dataset, multiple feature weights for multiple feature pairs are calculated using unsupervised learning.

[0034] Calculate the cosine similarity of multiple feature pairs;

[0035] The cosine similarity scores are calculated by weighting the cosine similarity scores according to the multiple feature weights to obtain a similarity score.

[0036] In some embodiments, obtaining multimodal retrieval results based on the sorting results and the target query conditions includes:

[0037] Determine the target query modality type corresponding to the target query condition;

[0038] Based on the target query modality type, the modality representations of multiple multimodal data corresponding to the sorting results are filtered by type to obtain multimodal retrieval results that conform to the target query modality type.

[0039] In some embodiments, the text data in the modal data is obtained in the following manner:

[0040] Text filtering processing is performed on the original text data in the preset dataset to obtain science-related text;

[0041] The aforementioned science-related texts are cleaned to obtain the cleaned text;

[0042] The cleaned text is subjected to key information extraction and text proofreading to obtain text data representing molecular information.

[0043] Secondly, embodiments of this application provide a cross-modal molecular information retrieval system, including:

[0044] The data acquisition module is used to obtain multimodal data characterizing molecular information from a preset dataset, wherein the multimodal data includes text data, image data, and graph structure data;

[0045] A model building module is used to build a multimodal molecular information model, which includes a modality extraction encoder and a modality fusion encoder. The modality extraction encoder is used to obtain multiple single-modality representations based on the multimodal data, and the modality fusion encoder is used to obtain a multimodal representation based on the multiple single-modality representations.

[0046] The model training module is used to input the multimodal data into the multimodal molecular information model, calculate the contrast loss corresponding to multiple single-modal representations, train and optimize the modality extraction encoder based on the contrast loss; and calculate the fusion loss corresponding to the multimodal representations, train and optimize the modality fusion encoder based on the fusion loss.

[0047] The information retrieval module is used to obtain target modal data and target query conditions of target molecule information, input the target modal data into the multimodal molecule information model, and obtain multimodal retrieval results corresponding to the target modal data and the target query conditions.

[0048] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the cross-modal molecular information retrieval method as described in any embodiment of the first aspect.

[0049] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions for performing the cross-modal molecular information retrieval method as described in any embodiment of the first aspect.

[0050] This application has at least the following beneficial effects: It proposes a cross-modal molecular information retrieval method. The method obtains multimodal data representing molecular information from a pre-set dataset, including text data, image data, and graph structure data; constructs a multimodal molecular information model, which includes a modality extraction encoder and a modality fusion encoder. The modality extraction encoder is used to obtain multiple single-modal representations from the multimodal data, and the modality fusion encoder is used to obtain multimodal representations from the multiple single-modal representations; inputs the multimodal data into the multimodal molecular information model, calculates the contrast loss corresponding to the multiple single-modal representations, and trains and optimizes the modality extraction encoder based on the contrast loss; calculates the fusion loss corresponding to the multimodal representations, and trains and optimizes the modality fusion encoder based on the fusion loss; and obtains target modal data of the target molecule information. The target modal data is input into a multimodal molecular information model along with the target query conditions to obtain multimodal retrieval results corresponding to the target modal data and the target query conditions. This application employs a multimodal fusion method, using a modality extraction encoder to calculate contrast loss and obtain single-modal representations of molecular information such as graph structures, images, and text data. This aligns graph structure and image information to supplement and enrich text information. A modality fusion encoder then calculates fusion loss to obtain multimodal representations of molecular information, achieving the fusion of different modal data such as graph structures, images, and text. In application, based on the multimodal representations of molecular information in the molecular database, cross-modal molecular retrieval can be performed according to the target modal data and target query conditions of the target molecular information, meeting the needs of multimodal information processing and thus improving the accuracy and efficiency of molecular information retrieval. Attached Figure Description

[0051] Figure 1This is a flowchart of a cross-modal molecular information retrieval method proposed in an embodiment of this application;

[0052] Figure 2 This is a flowchart of a cross-modal molecular information retrieval method proposed in another embodiment of this application;

[0053] Figure 3 This is a flowchart of a cross-modal molecular information retrieval method proposed in another embodiment of this application;

[0054] Figure 4 This is a flowchart of a cross-modal molecular information retrieval method proposed in another embodiment of this application;

[0055] Figure 5 This is a flowchart of a cross-modal molecular information retrieval method proposed in another embodiment of this application;

[0056] Figure 6 This is a flowchart of a cross-modal molecular information retrieval method proposed in another embodiment of this application;

[0057] Figure 7 This is a flowchart of a cross-modal molecular information retrieval method proposed in another embodiment of this application;

[0058] Figure 8 This is a flowchart of a cross-modal molecular information retrieval method proposed in another embodiment of this application;

[0059] Figure 9 This is an architecture diagram of a cross-modal molecular information retrieval system proposed in another embodiment of this application;

[0060] Figure 10 This is a structural diagram of an electronic device proposed in another embodiment of this application. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0062] In some embodiments, although functional modules are divided in the system diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the system or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the foregoing drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0063] Furthermore, unless otherwise explicitly specified and limited, the term "connection / linkage" should be interpreted broadly, for example, it can be a fixed connection or a movable connection, a detachable connection or a non-detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection or a connection that can communicate with each other; it can be a direct connection or an indirect connection through an intermediate medium.

[0064] In the description of the embodiments in this application, the terms "one embodiment / implementation," "another embodiment / implementation," or "some embodiments / implementations," "in the above embodiments / implementations," etc., refer to specific features, structures, materials, or characteristics described in conjunction with embodiments or examples that are included in at least two embodiments or implementations disclosed in this application. In this disclosure, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or implementation. It should be noted that although a logical order is shown in the flowcharts, in some cases, the steps shown or described may be performed in a different order than that shown in the flowcharts.

[0065] In recent years, the rapid development of information technology and big data has had a profound impact on the field of data retrieval. In the field of molecular information, traditional data retrieval methods mostly rely on a single data modality, such as text, images, or graph structures. These methods perform well when handling simple queries. For example, graph structures can effectively reveal complex relationship networks between entities, images can provide intuitive and rich visual information, and text can describe specific concepts and facts in detail. However, when dealing with complex queries and improving information retrieval efficiency, single-modal data sources have limitations in expressing information. Different data types, such as text, images, and graph structures, each have unique information expression methods and advantages. Therefore, when dealing with queries that require multimodal information, single-modal molecular information retrieval systems often fail to achieve ideal results and may even lead to information omissions or decreased retrieval accuracy.

[0066] To at least address the aforementioned problems, embodiments of the present invention provide a cross-modal molecular information retrieval method, electronic device, and medium. These aim to solve the technical problems of poor retrieval performance and difficulty in handling multimodal information query requirements in existing single-modal molecular information retrieval systems. The system can effectively integrate data from different modalities such as graph structures, images, and text, thereby improving the accuracy and efficiency of molecular information retrieval.

[0067] The embodiments of this application will be further described below with reference to other accompanying drawings.

[0068] Please refer to Figure 1 , Figure 1 This is a flowchart of a cross-modal molecular information retrieval method proposed in an embodiment of this application. In a first aspect, this application provides a cross-modal molecular information retrieval method, including:

[0069] Step S110: Obtain multimodal data representing molecular information from a preset dataset. The multimodal data includes text data, image data, and graph structure data.

[0070] Step S120: Construct a multimodal molecular information model. The multimodal molecular information model includes a modality extraction encoder and a modality fusion encoder. The modality extraction encoder is used to obtain multiple single-modality representations based on multimodal data, and the modality fusion encoder is used to obtain multimodal representations based on multiple single-modality representations.

[0071] Step S130: Input multimodal data into the multimodal molecular information model, calculate the contrast loss corresponding to multiple single-modal representations, train and optimize the modality extraction encoder based on the contrast loss; and calculate the fusion loss corresponding to the multimodal representations, train and optimize the modality fusion encoder based on the fusion loss.

[0072] Step S140: Obtain the target modal data and target query conditions of the target molecule information, input the target modal data into the multimodal molecular information model, and obtain the multimodal retrieval results corresponding to the target modal data and target query conditions.

[0073] In some embodiments, corresponding to steps S110 to S140, this application can obtain text, image, and graph structure data from a preset dataset. These data represent molecular information to varying degrees. During model training, the modality extraction encoder is used to extract features of each modality and encode them into single-modality representations, such as converting text data into text modality representations, image data into image modality representations, and graph structure data into graph structure modality representations. The modality fusion encoder is responsible for fusing these single-modality representations to obtain a unified multimodal representation. Specifically, for the modality extraction encoder, the contrastive loss for each single-modality representation is used to train and optimize the encoder. On the other hand, the modality fusion encoder is optimized with fusion loss as the guide. The fusion loss focuses more on the overall performance of the model. Thus, after obtaining the target modality data and query conditions of the target molecule, the multimodal retrieval results are obtained by extracting and fusing features through the already trained model.

[0074] In some embodiments, this application employs a multimodal fusion method. A modal extraction encoder calculates contrast loss to obtain single-modal representations of molecular information, such as graph structures, images, and text data. This aligns graph structure and image information to supplement and enrich text information. A modal fusion encoder then calculates fusion loss to obtain multimodal representations of molecular information, achieving the fusion of different modal data such as graph structures, images, and text. In application, based on the multimodal representations of molecular information in the molecular database, cross-modal molecular retrieval can be performed according to the target modal data of the target molecular information and the target query conditions, satisfying the query requirements for processing multimodal information and thus improving the accuracy and efficiency of molecular information retrieval.

[0075] It is conceivable that the purpose of this application's technical solution is to address the problem faced by the field of modern molecular information data retrieval, which relies on a single type of data source, such as plain text or a single image. This may lead to information omissions or decreased retrieval accuracy when processing complex queries. Since various data types of molecular information (such as graph structures, images, and text) have unique ways of expressing information, for example, graph structures can effectively represent the relationships between entities, images are rich in visual information, and text details specific content, a single data source cannot fully reveal the multidimensional characteristics of information. Therefore, the goal of this application is to integrate these different data modalities, leverage their complementary effects, and achieve multimodal molecular information fusion and corresponding multimodal retrieval.

[0076] It is understandable that the multimodal retrieval method in this application has higher flexibility and accuracy in handling complex and fuzzy queries. For example, combining text descriptions of molecular information with image content for querying can more accurately locate information and reduce the false detection rate. At the same time, since different types of data sources have significant differences in format, structure and semantics, this application integrates the processing and analysis of graph structure, image and text data through the above steps S110 to S140. By standardizing and preprocessing data of different modalities, it ensures that they can be effectively fused and analyzed. The modality extraction encoder calculates the contrast loss to obtain single-modal representations of molecular information such as graph structure, image and text data. Then, a Transformer model-based architecture is adopted, and on this basis, three cross-modal interactions are added. The attention mechanism modality fusion encoder can process and understand different modal data such as graph structures, images, and text, and integrate them into a unified latent space. In molecular science scenarios, latent space vectors can be used to represent molecular multimodalities, and representation vectors can be used for downstream tasks such as molecular attribute prediction, molecular cross-modal retrieval, molecular generation, and molecular optimization. This enables the integration of multiple types of data to fully leverage the advantages of different data sources and significantly improve retrieval efficiency and accuracy.

[0077] In some embodiments, the textual description of the molecule in this application includes the molecule's IUPAC name, chemical properties, applications and historical information, and the SMILES molecular expression, and the image data includes molecular images.

[0078] In some embodiments, the preset dataset can be a preset molecular information database, which may contain graph structures, images, and text data corresponding to multiple molecular information. Since this application can obtain multimodal representations and multiple unimodal representations of multiple molecular information from the preset molecular information database, when obtaining target modal data and target query conditions for target molecular information and inputting the target modal data into the multimodal molecular information model, the target modal data can be any one of a multimodal representation or multiple unimodal representations. The resulting multimodal retrieval results corresponding to the target modal data and target query conditions can also be any one of a multimodal representation or multiple unimodal representations. The target query conditions can be used to limit the modality types of the multimodal retrieval results; specifically, this can include, but is not limited to, the following scenarios:

[0079] Single-modal retrieval involves obtaining multiple single-modal representations of a single single-modal data point. For example: if the target modal data is a molecular image, the SMILES expression corresponding to the molecular image is retrieved using a multimodal molecular information model; if the target modal data is a molecular image, the molecular image is converted into a corresponding molecular image using a multimodal molecular information model for retrieval; if the target modal data is a molecular image, the molecular text description corresponding to the molecular image is retrieved using a multimodal molecular information model; if the target modal data is a molecular image and a molecular information database stores molecular images, the molecular text description, such as the IUPAC name, corresponding to the molecular image is obtained using a multimodal molecular information model for retrieval; and so on. Other single-modal retrieval methods include: using the SMILES expression as input to retrieve the text description of a molecule; retrieving detailed molecular text descriptions from a molecular image; retrieving the IUPAC name of a molecule based on the SMILES expression; and deriving the IUPAC name of a molecule from a molecular image.

[0080] Multimodal retrieval retrieves multimodal representations of multiple unimodal data. For example, if the target modal data includes SMILES expressions and molecular images, a multimodal molecular information model can be used to combine SMILES expressions and molecular images to retrieve the molecular text description; or if the target modal data includes SMILES expressions and molecular images, a multimodal molecular information model can be used to combine SMILES expressions and molecular images to retrieve the molecular IUPAC name; and so on, including the following multimodal retrieval methods: combining SMILES expressions and molecular images to retrieve the molecular text description; combining SMILES expressions and molecular images to locate the molecular IUPAC name; combining SMILES expressions, images, and graph structures to locate the molecular IUPAC name; and combining SMILES expressions, images, and graph structures to retrieve the molecular text description.

[0081] In the above scenario, it is understandable that the single-modal or multimodal representation of the target modal data cannot be directly generated from the retrieval modality using traditional tools such as the RDKit toolkit. However, this application can overcome this limitation by fusing multimodal data to achieve cross-modal retrieval and realize more accurate and efficient cross-modal retrieval functions.

[0082] In some embodiments, the cross-modal molecular information retrieval system corresponding to the method of this application may include a data layer, a model layer, and an application layer. In the data layer, corresponding to step S110 above, this application may first collect and organize cross-modal data, which includes images and structural diagrams of molecules and related text descriptions (e.g., the IUPAC name of the molecule, the chemical properties of the molecule, its applications, and historical information), to provide data support for subsequent model training and prediction.

[0083] In some embodiments, corresponding to steps S120 and S130 above, at the model layer, this application may adopt an architecture based on an improved Transformer model and introduce a cross attention mechanism for three modalities. This mechanism is mainly used to fuse data from three modalities: image, text, and graph structure. Centered on the text modality, the preprocessed embedding vectors are processed using the Transformer's self-attention layer. Then, through the text-image and text-graph structure cross attention mechanism with text as the core, effective interaction and information fusion between different modalities are achieved.

[0084] In some embodiments, corresponding to step S140 above, at the application layer, by combining data preprocessing at the data layer and multimodal fusion at the model layer, this application can obtain single-modal or multimodal data representations of the retrieved entity data and data representations of multiple entity target modalities. Then, the system can perform cross-modal retrieval tasks based on retrieval algorithms that calculate cosine similarity or combine molecular domain expert knowledge to obtain retrieval results.

[0085] It is conceivable that, at the application layer, this application can also include a classifier. The classifier can classify the multimodal representations and multiple unimodal representations of multiple molecular information in the molecular information database based on the calculated cosine similarity or by combining molecular domain expert knowledge, so as to obtain the multimodal retrieval results corresponding to the target modality data and the target query conditions.

[0086] Please refer to Figure 2 , Figure 2 This is a flowchart of a cross-modal molecular information retrieval method proposed in another embodiment of this application. In some embodiments, the modality extraction encoder includes a text encoder, an image encoder, and a graph structure encoder; the single-modal representation includes text modality representation, image modality representation, and graph structure modality representation; and multiple single-modal representations are obtained based on multimodal data, including:

[0087] Step S210: Input the text data into the text encoder to obtain the text modal representation;

[0088] Step S220: Input the image data into the image encoder to obtain the image modal representation;

[0089] Step S230: Input the graph structure data into the graph structure encoder to obtain the graph structure modal representation.

[0090] In some embodiments, steps S210 to S230 specifically specify the structure and function of the modality extraction encoder, explaining how the text encoder, image encoder, and graph structure encoder obtain the corresponding single-modal representation. This allows the cross-modal molecular information retrieval method of this application to fully utilize the information advantages of different data sources, improving the accuracy and completeness of retrieval. Especially when facing complex queries, this method can effectively acquire and combine multimodal information, avoid information omissions, and improve the efficiency and quality of information retrieval.

[0091] In some embodiments, prior to performing steps S210 to S230, the collection of image, graph structure, and textual data is first involved. This includes obtaining molecular images and structural maps from public databases, as well as collecting relevant literature and textual descriptions from databases. Textual characterizations (SMILES) expressions, images, and graph structure data of molecules are obtained from public databases such as PubChem, while textual data related to these molecules, such as literature abstracts and descriptions of chemical properties, are also collected. There are no unified standards or databases for textual descriptions of molecules; most are extracted from literature.

[0092] In some embodiments, preprocessing of data of different modalities is also included after data acquisition. First, image data is resized and formatted to ensure consistency and ease of processing. For graph-structured data, such as molecules represented by SMILES, the RDKit toolkit is used to convert them into molecular objects, thereby achieving vectorized representation for subsequent calculations and analysis. Simultaneously, text data also needs to be standardized and cleaned to remove irrelevant information and noise (text without scientific meaning), retaining key content and ensuring that the text length follows a normal distribution, thus enabling more effective data mining and analysis. These steps collectively ensure the extraction of high-quality, analyzable information from data of different sources and formats.

[0093] In some embodiments, corresponding to steps S210 to S230, the text encoder, image encoder, and graph structure encoder are all different encoders. For image data, this application can use the Swin Transformer as the image encoder, which utilizes a hierarchical Transformer structure to effectively capture image details and contextual information. For graph structure data, this application can use the GIN (Graph Isomorphism Network) from graph neural networks as the graph structure encoder, which can effectively extract graph structure features while preserving the graph's structural information. As for text data, this application can use MolT5 as the text encoder. This model is based on the T5 end-to-end text model, which performs excellently in processing natural language text. MolT5 is pre-trained for specific corpora in the chemical field, providing a better understanding of chemical molecules. Thus, three different encoders yield three corresponding monomodal representations.

[0094] Please refer to Figure 3 , Figure 3 This is a flowchart of a cross-modal molecular information retrieval method proposed in another embodiment of this application. In some embodiments, the method calculates the contrastive loss corresponding to multiple single-modal representations, and trains and optimizes the modality extraction encoder based on the contrastive loss, including:

[0095] Step S310: Generate positive sample pairs based on text data, image data, and graph structure data that represent the same molecular information;

[0096] Step S320: Generate negative sample pairs based on text data, image data, and graph structure data that characterize different molecular information;

[0097] Step S330: Substitute the positive sample pairs and negative samples into the preset triplet loss function to calculate the first loss and the second loss respectively. The first loss represents the contrastive learning loss between text modality representation and image modality representation, and the second loss represents the contrastive learning loss between text modality representation and graph structure modality representation.

[0098] Step S340: The sum of the first loss and the second loss is determined as the contrastive loss. The modality extraction encoder is trained and optimized with the goal of minimizing the contrastive loss.

[0099] In some embodiments, corresponding to steps S310 to S330, firstly, this application generates positive sample pairs using text data, image data, and graph structure data representing the same molecular information, and generates negative sample pairs using data representing different molecular information, aiming to provide training data for subsequent contrastive learning; further, the above sample pairs are substituted into a preset triplet loss function to calculate the contrastive loss. The triplet loss function is usually designed to increase the distance between negative sample pairs while decreasing the distance between positive sample pairs, and contrastive learning is performed in this way. The first loss and the second loss represent the contrastive learning loss between the text and image modalities and the text and graph structure modalities, respectively; further, the modality extraction encoder is trained and optimized by minimizing the calculated contrastive loss (i.e., the sum of the first loss and the second loss), prompting the encoder to learn modal representations that can distinguish different molecular information, improving the modality extraction encoder's representation ability for different modal data, and improving the similarity between different modalities, so that the encoder can better capture the common features of molecular information in different modalities, and through contrastive learning, enhance the model's ability to distinguish between positive and negative samples, thereby improving retrieval accuracy and efficiency.

[0100] Specifically, this application, based on the Transformer model, establishes dedicated modality extraction encoders (including image encoders, graph structure encoders, and text encoders) for various data modalities, and constructs a modality fusion encoder that integrates a cross-attention mechanism. This modality fusion encoder is specifically designed for processing and fusing image, text, and graph structure data modalities. Its core concept is to fully utilize the advantages of the Transformer model in processing text sequence data, using the text modality as the core, and constructing a text-image and text-graph structure cross-attention mechanism.

[0101] Within the aforementioned encoder framework, the input data is first preprocessed, converting each modality's data into embedding vectors. These embedding vectors are then fed into the Transformer self-attention layer of the modality fusion encoder to capture key features and relationships within each modality. Next, a specific cross-attention mechanism enables interaction and fusion between different modalities, allowing the model to more comprehensively understand and integrate various types of information. Furthermore, this application employs a contrastive learning strategy involving text-image and text-graph structures to align different entities in the latent space during training, thereby further enhancing the model's fusion capabilities.

[0102] It is conceivable that conventional cross-attention mechanisms can only achieve bimodal cross-learning, but have limitations in the stable alignment of entity latent space vectors. The distribution of latent space vectors is prone to mutual influence. Therefore, this application introduces a contrastive learning strategy to achieve the alignment of entity data of different modalities in the latent space. In particular, compared with traditional bimodal methods, the trimodal alignment strategy of this application can enhance the stability of the core text modality. In addition, the trimodal fusion strategy of this application also has the following advantages: it can more comprehensively capture and integrate the unique information of each modality, such as the visual details of images, the semantic content of text and the relational features of graph structures, thereby improving the accuracy and depth of retrieval and analysis. This not only improves the depth and breadth of multimodal data fusion, but also significantly enhances the stability and alignment effect of core modality data.

[0103] Specifically, corresponding to steps S310 to S340 above, the contrastive learning strategy can be included in the model training process. By inputting a data set of modality A-modality B-modality C, contrastive learning is used to continuously align the various modal data of entities in the model's latent space during training. Specifically, this can include the following steps: Generating positive and negative sample pairs: Creating positive and negative sample pairs for each molecular entity from three modalities (image, text, and graph structure). Positive sample pairs refer to different modal representations of the same molecule, such as an image of a molecule and its SMILES expression. Negative sample pairs consist of modal representations of different molecules. Feature embedding: Performing feature embedding on the data of each modality. For example, for image data, this application may use a convolutional neural network (Swin Transformer) to extract features. Similarly, text and graph structure data will be converted into embedding vectors through an appropriate model for subsequent processing. Contrastive loss function: This application uses a contrastive loss function to train the model, aiming to make different modal representations of the same entity closer in the embedding space, while representations of different entities are further apart. This can be achieved through triplet loss, the formula of which is:

[0104]

[0105] L total =L t-g +L t-i

[0106] Among them, a i p i n i These represent anchor points, positive samples, and negative samples, respectively. 'd' represents the distance metric, and 'margin' is the set boundary value. Using this method, the model can learn to distinguish and emphasize the features of identical and different entities across different modalities. L total L represents the overall loss function for model training. t-g and L t-i These represent the text-image contrastive learning loss function and the text-image contrastive learning loss function, respectively. The modality extraction encoder can then be trained and optimized with the goal of minimizing the contrastive loss. It's conceivable that, compared to traditional bimodal (AB) input, if bimodal learning aims to achieve trimodal learning, the input form is (AB, BC, AC), while the trimodal input form is directly (ABC). The loss function in the trimodal training process is a combination of text-image and text-graph structure loss functions, which can effectively improve the model's performance in multimodal tasks.

[0107] Please refer to Figure 4 , Figure 4 This is a flowchart of a cross-modal molecular information retrieval method proposed in another embodiment of this application. In some embodiments, the modality fusion encoder includes multiple cross-attention layers to obtain a multimodal representation based on multiple unimodal representations, including:

[0108] Step S410: The text modal representation, image modal representation, and graph structure modal representation representing the same molecular information are input as a modal group into the multi-layer cross-attention layer;

[0109] Step S420: Calculate the fusion weight coefficient group corresponding to the modality group using the preset normalized exponential function in the multi-layer cross-attention layer;

[0110] Step S430: The multimodal representations are obtained by weighting the modal representations in the modal group according to the fusion weight coefficient group.

[0111] In some embodiments, corresponding to steps S410 to S430, this application utilizes multi-layer cross-attention layers to process modal representations from text, images, and graph structures. The cross-attention layers allow the model to focus on the interactions and information complementarity between different modalities. By using a normalized exponential function in the cross-attention layers to calculate the contribution of each modality to the final multimodal representation, i.e., the fusion weight coefficient set, the different modal representations are then weighted according to the fusion weight coefficient set to obtain the final multimodal representation. This step generates a comprehensive multimodal representation that characterizes molecular information by weighted fusion of information from different modalities. It is conceivable that the weighted calculation of the multimodal representation can automatically adjust the contribution of different modalities in the final representation based on their importance, thereby optimizing the relevance and accuracy of the retrieval results, promoting complementarity and integration between different modalities, and enhancing the comprehensiveness and accuracy of molecular information retrieval.

[0112] In some embodiments, Cross Attention in this application is an attention mechanism that allows the model to simultaneously attend to information from different modalities, such as images, text, and graph structures. The general formula for Cross Attention can be expressed as:

[0113]

[0114] Where Q (Query), K (Key), and V (Value) are representations of different modalities, and dk is the dimension of the Key vector. This application can calculate the dot product between Query and Key using the above formula, and then divide by... Scaling is performed, and the softmax (normalization exponent) function is applied to obtain the weights. Finally, these weights are used to weight the Value to obtain the weight coefficients (fusion weight coefficient set) of each single modality representation. It can be understood that the training process of the modality fusion encoder is the optimization process of the fusion weight coefficient set.

[0115] As can be conceivable, the Cross Attention mechanism in traditional bimodal fusion methods enables the model to understand the interrelationships between different modalities while processing single-modal data. For example, the model can learn how to combine molecular structure diagrams and images. Through the cross attention mechanism, the model fuses data from different modalities in a unified latent space, generating a comprehensive, multi-dimensional data representation. Multimodal data fusion refers to integrating information from different sources (such as images, text, and graph structures) into a unified representation. An example of bimodal fusion can be represented by a weighted sum:

[0116] M=α·V I +(1-α)·V G ;

[0117] Traditional approaches to trimodal fusion using conventional bimodal fusion methods require inputting data sets of modality A-modality B, modality B-modality C, and modality C-modality A during model training. The single CrossAttention layer continuously changes these learned weights, leading to inaccurate and unstable weight assignments. Our proposed solution, however, uses a modality extraction encoder to calculate contrastive loss and obtain single-modal representations of molecular information, such as graph structures, images, and text data. A modality fusion encoder then calculates fusion loss to obtain multimodal representations of molecular information. This allows our model to achieve trimodal fusion using CrossAttention sets. Different CrossAttention layers can stabilize weight assignments for different input modality combinations, and the text-centric training approach allows other modalities to remain relatively stable based on the training objective. A trimodal fusion example can be represented using a weighted sum:

[0118] M=α·V I +β·V G +(1-α-β)·V T ;

[0119] Where M is the fused multimodal representation, VI, VG, and VT are the vectorized representations of image, graph structure, and text, respectively, and α and β are the fusion weights of these modalities, corresponding to multiple fusion weights in the fusion weight coefficient group. It can be understood that in the field of molecular information retrieval, the fusion and interaction of data with three or more modalities has always been challenging, and most current multimodal fusion strategies are mainly aimed at bimodal data. Compared to traditional bimodal cross-attention fusion mechanisms, the integrated cross-attention architecture of this application can contain multiple independent cross-attention modules. Each module specifically handles the interaction between different modalities (such as images and graph structures) and the target text modality. This design enables the system to more accurately capture the correlation between modalities, especially the correlation with the text modality. In addition, this design uses the text modality of molecular information as the core for information integration. By introducing a contrastive learning training strategy, alignment between different entities in the latent space is achieved during training, thereby improving the fusion performance of the model. Information from other modalities is transformed and aligned to supplement and enrich the text content. For example, when processing chemical structures, the system can identify specific structures and match them with chemical terms in the text description to achieve trimodal fusion.

[0120] Please refer to Figure 5 , Figure 5This is a flowchart of a cross-modal molecular information retrieval method proposed in another embodiment of this application. In some embodiments, the method includes calculating the fusion loss corresponding to the multimodal representation, and training and optimizing the modal fusion encoder based on the fusion loss, including:

[0121] Step S510: Calculate the representation similarity between text modal representation, image modal representation, graph structure modal representation, and multimodal representation;

[0122] Step S520: Calculate the fusion loss based on representation similarity, and update the fusion weight coefficient group in the modal fusion encoder with the goal of minimizing the fusion loss.

[0123] In some embodiments, this application first calculates the representation similarity between text modality representation, image modality representation, graph structure modality representation, and multimodal representation to determine the consistency of information between different modalities. Further, it calculates the fusion loss based on the representation similarity to ensure, from a multimodal perspective, that the fused information does not lose or distort the original single-modal information. Further, with the goal of minimizing the fusion loss, it updates the fusion weight coefficient set in the modality fusion encoder to optimize the modality fusion process. The above steps of calculating representation similarity ensure that the multimodal representation correctly fuses single-modal information, thereby improving the feature preservation effect of each modality. Updating the fusion weight coefficient set with the goal of minimizing the fusion loss can improve the accuracy of cross-modal molecular information retrieval, thus effectively improving the model performance of the modality fusion encoder and enhancing the integration capability of cross-modal molecular information.

[0124] Please refer to Figure 6 , Figure 6 This is a flowchart of a cross-modal molecular information retrieval method proposed in another embodiment of this application. In some embodiments, target modal data is input into a multimodal molecular information model to obtain multimodal retrieval results corresponding to the target modal data and target query conditions, including:

[0125] Step S610: Input the target modal data into the multimodal molecular information model to obtain the target modal representation corresponding to the target modal data. The target modal representation includes the target single-modal representation and the target multimodal representation.

[0126] Step S620: Calculate the similarity score between the modal representations corresponding to multiple multimodal data in the preset dataset and the target modal representation;

[0127] Step S630: Sort the modal representations corresponding to multiple multimodal data according to the similarity scores, and obtain the multimodal retrieval results based on the sorting results and the target query conditions.

[0128] In some embodiments, the target query criteria may include the modalities of the query results expected by the user. Users can perform searches by inputting target modal representations such as images, structural maps, or text descriptions of molecules. The system compares the user's input with multimodal data in the database to find the best match. The retrieved molecules are presented in the form of images, structural maps, and detailed text descriptions, providing users with comprehensive molecular information. Users can also refine the search results through further queries, for example, by searching for specific chemical properties of molecules or their application areas (drug development / materials development).

[0129] In multimodal retrieval systems, ranking algorithms typically rely on calculating the similarity between the user input and each item in the database. A simple similarity measure can be cosine similarity:

[0130]

[0131] Where A is a single-modal or multi-modal fusion representation of user input and molecular information of a certain item in the molecular database, and B is the target modality representation. Cosine similarity is used as the similarity score, and then the modality representations corresponding to multiple multimodal data can be sorted according to the similarity score. Based on the sorting results and target query conditions, multimodal retrieval results are obtained.

[0132] Please refer to Figure 7 , Figure 7 This is a flowchart of a cross-modal molecular information retrieval method proposed in another embodiment of this application. In some embodiments, the similarity score between the modality representations corresponding to multiple multimodal data in a preset dataset and the target modality representation is calculated, including:

[0133] Step S710: Determine multiple feature pairs between the modality representations corresponding to the multimodal data in the preset dataset and the target modality representation;

[0134] Step S720: Based on the preset molecular knowledge dataset, calculate the feature weights of multiple feature pairs using unsupervised learning.

[0135] Step S730: Calculate the cosine similarity of multiple feature pairs.

[0136] Step S740: Perform feature weighting calculation on multiple cosine similarities based on multiple feature weights to obtain a similarity score.

[0137] In some embodiments, multiple feature pairs are determined between the modal representations corresponding to multimodal data in a preset dataset and the target modal representation to identify key features that can represent the similarity between data. Based on a preset molecular knowledge dataset, unsupervised learning is used to calculate the feature weights of multiple feature pairs to help identify which features are more important in representing similarity. Cosine similarity is calculated for multiple feature pairs to measure the similarity between two vector directions and enhance the generalization ability of the model. Feature weighted calculation is performed on multiple cosine similarities according to multiple feature weights to obtain a similarity score, which is used to combine the importance of features and the actual similarity calculation to obtain a more accurate and appropriate similarity assessment and improve the performance of the retrieval system.

[0138] In some embodiments, traditional cross-modal retrieval often relies on mathematical metrics such as cosine similarity. However, such methods may overlook the expertise of molecular scientists and the practical application context. Therefore, this application combines molecular domain expert knowledge to design a novel similarity evaluation method. This method, based on the cosine similarity between vectors, weights different features according to molecular domain expert knowledge corresponding to a pre-defined molecular knowledge dataset. The weighting can be implemented using unsupervised algorithms (such as PCA or t-SNE). These algorithms can extract representative and differentiated features from molecular feature data, such as specific functional groups or chemical properties. Combining these analysis results, a weighting scheme can be designed to assign higher weights to important features, as shown in the following formula:

[0139]

[0140] w i The weights of the i-th feature are determined based on PCA or other unsupervised learning methods. CosineSimilarity(Featurei) is the cosine similarity between two molecular vectors calculated based on featurei. Based on these similarity scores, the system can rank all items, placing the most relevant items at the top. It's conceivable that during training, the similarity score can be considered a loss function to guide model training. For intelligent molecular data management, this scheme is based on multimodal molecular characterization methods and combined with a retrieval and ranking algorithm based on molecular domain knowledge, significantly improving retrieval accuracy and efficiency. This is particularly evident in handling complex molecular structures and related data, where the scheme demonstrates high performance. Furthermore, in large-scale molecular databases, this scheme can effectively manage and retrieve multimodal data, greatly improving the intelligence level of data retrieval and analysis.

[0141] Please refer to Figure 8 , Figure 8This is a flowchart of a cross-modal molecular information retrieval method proposed in another embodiment of this application. In some embodiments, multimodal retrieval results are obtained based on ranking results and target query conditions, including:

[0142] Step S810: Determine the target query modality type corresponding to the target query condition;

[0143] Step S820: Based on the target query modality type, the modality representation of multiple multimodal data corresponding to the sorting results is filtered by type to obtain multimodal retrieval results that conform to the target query modality type.

[0144] In some embodiments, determining the target query modality type corresponding to the target query condition means identifying which type of modal data (such as text, image, or graph structure) the query is targeting. Then, the modal representation of the multimodal data corresponding to the sorting results can be filtered according to the target query modality type to obtain multimodal retrieval results that conform to the target query modality type, so as to ensure that the retrieval results are consistent with the user's query intent.

[0145] It is conceivable that determining the query modality type and type filtering can ensure that users receive the information type they actually need. This allows the application to further refine the retrieval process through type filtering, enabling the cross-modal retrieval system to respond more flexibly and accurately to different types of query needs and optimize the relevance of retrieval results. In other words, through refined similarity calculation and query condition matching steps, this application can improve the performance and user experience of the cross-modal molecular information retrieval system, ensuring that the method of this application only returns data of the modality type that matches the query conditions, thereby improving user satisfaction.

[0146] Specifically, the application scenarios of the cross-modal molecular information retrieval method proposed in this application can be as follows: When drug development researchers use the cross-modal molecular information retrieval system to search for molecules with specific structural features, they can upload the molecular structure map or input relevant chemical property text descriptions. After receiving the input, the system performs an efficient multimodal search in the backend database, quickly returning a list of candidate molecules with the highest matching degree. Simultaneously, users can also use the system to perform more complex queries, such as combining chemical properties and application fields (e.g., "anti-cancer drugs"). This helps users quickly find candidate substances that meet specific needs among a large number of molecules. After the search is completed, the system presents the retrieved molecules to the user in the form of images, structure maps, and detailed text descriptions. In addition, the system can provide additional analytical tools, such as molecular property prediction and possible bioactivity hints, providing users with more in-depth molecular information. Furthermore, the system design of this application also has compatibility with different terminals, allowing users to conduct information retrieval through personal computers, tablets, and smartphones.

[0147] In some embodiments, the text data in the modal data is obtained as follows: the original text data in the preset dataset is subjected to text filtering to obtain scientifically relevant text; the scientifically relevant text is cleaned to obtain cleaned text; the cleaned text is subjected to key information extraction and text proofreading to obtain text data representing molecular information. This step can effectively improve the quality and usability of the text data, remove irrelevant or redundant information, and extract key information related to molecules, providing higher quality data input for subsequent modal representation learning and similarity calculation. Text cleaning and key information extraction help improve the performance of information retrieval and machine learning models, reduce noise and unnecessary data complexity, allow the model to focus more on key and meaningful information, and further improve the accuracy of the data through text proofreading to ensure that the data provided to the model does not contain misleading or erroneous information.

[0148] It is conceivable that, since this application employs multiple independent cross-attention components, each specifically handling interactions between different modalities (such as images and graph structures) and the target text modality, and thus using the text modality as the core of information integration, while information from other modalities is transformed and aligned to supplement and enrich the text content, it is necessary to filter the text data when acquiring it. Specifically, this may include the following steps: using keyword search and regular expression techniques to exclude obviously irrelevant text (such as advertisements, copyright notices, etc.) to achieve preliminary filtering; using natural language processing (NLP) techniques, such as text classification models, to divide the filtered text into two categories: "scientifically relevant" and "non-scientifically relevant"; and ensuring text consistency by removing formatting symbols, non-standard abbreviations, and incorrect punctuation marks, and integrating synonyms or technical terms. For example, chemical entities may be described using different terms in different documents. To achieve text cleaning, information extraction algorithms (such as named entity recognition) are applied to identify and extract key scientific information, such as chemical names and molecular formulas. For long texts, abstract generation is performed to extract core content, ensuring that the information is concise and targeted, thereby optimizing the content. The consistency of terminology, format, and style in the processed text data is guaranteed. Proofreading algorithms are used to identify and correct spelling and grammatical errors in the text to achieve the quality control stage, effectively improving data quality and the accuracy of information retrieval.

[0149] Please refer to Figure 9 , Figure 9 This is an architectural diagram of a cross-modal molecular information retrieval system proposed in another embodiment of this application; in some embodiments, such as Figure 9As shown, the data layer acquires data such as images, text, and graph structures. The model layer uses image encoders, text encoders, and graph structure encoders combined with a cross-attention mechanism and a Transformer model to obtain the unimodal and multimodal representations of images, text, and graph structures. The processing in the data and model layers is primarily performed on the server side to leverage high-performance computing resources for large-scale data processing and complex model training. The application layer performs cross-modal molecular retrieval based on the target modal data and target query conditions of the target molecule information, obtaining the retrieval results, such as... Figure 9 As shown, the Top 10 search results represent the top 10 modal representations of the target molecular information and the target query conditions, thus meeting the needs of multimodal information processing. Conversely, the application layer, especially the part interacting with the user, can be executed on a local terminal (such as a mobile device or personal computer) to achieve real-time user experience and response. This allows the proposed solution to effectively integrate data from different modalities such as graph structures, images, and text, providing users with more comprehensive information and deeper insights, thereby improving the accuracy and efficiency of retrieval.

[0150] Secondly, embodiments of this application provide a cross-modal molecular information retrieval system, including:

[0151] The data acquisition module is used to obtain multimodal data characterizing molecular information from a preset dataset. The multimodal data includes text data, image data, and graph structure data.

[0152] The model building module is used to build a multimodal molecular information model. The multimodal molecular information model includes a modality extraction encoder and a modality fusion encoder. The modality extraction encoder is used to obtain multiple single-modal representations from multimodal data, and the modality fusion encoder is used to obtain multimodal representations from multiple single-modal representations.

[0153] The model training module is used to input multimodal data into the multimodal molecular information model, calculate the contrast loss corresponding to multiple single-modal representations, train and optimize the modality extraction encoder based on the contrast loss; and calculate the fusion loss corresponding to the multimodal representations, train and optimize the modality fusion encoder based on the fusion loss.

[0154] The information retrieval module is used to obtain target modal data and target query conditions for target molecule information, input the target modal data into the multimodal molecular information model, and obtain the multimodal retrieval results corresponding to the target modal data and target query conditions.

[0155] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the cross-modal molecular information retrieval method as described in any of the embodiments of the first aspect.

[0156] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions for performing the cross-modal molecular information retrieval method as described in any of the embodiments of the first aspect.

[0157] refer to Figure 10 , Figure 10 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application.

[0158] Some embodiments of this application provide an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the cross-modal molecular information retrieval method of any of the above embodiments.

[0159] The electronic device 1000 of this application embodiment includes one or more processors 1010 and memory 1020. Figure 10 The example uses a processor 1010 and a memory 1020.

[0160] The processor 1010 and the memory 1020 can be connected via a bus or other means. Figure 10 Taking the example of a connection between China and Israel via a bus.

[0161] Memory 1020, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory 1020 may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory 1020 may optionally include memory 1020 remotely located relative to processor 1010. These remote memories can be connected to electronic device 1000 via a network, and examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0162] In some embodiments, when the processor executes a computer program, it executes the cross-modal molecular information retrieval method of any of the above embodiments at preset intervals.

[0163] Those skilled in the art will understand that Figure 10The device structure shown does not constitute a limitation on the electronic device 1000, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0164] exist Figure 10 In the electronic device 1000 shown, the processor 1010 can be used to call the cross-modal molecular information retrieval method stored in the memory 1020, thereby realizing the cross-modal molecular information retrieval method.

[0165] Based on the hardware structure of the aforementioned electronic device 1000, various embodiments of the cross-modal molecular information retrieval system of this application are proposed. Meanwhile, the non-transitory software programs and instructions required to implement the cross-modal molecular information retrieval method of the aforementioned embodiments are stored in a memory. When executed by a processor, the cross-modal molecular information retrieval method of the aforementioned embodiments is executed.

[0166] This application also provides a computer-readable storage medium storing computer-executable instructions for performing the above-described cross-modal molecular information retrieval method. For example, the computer-executable instructions can cause one or more processors to perform the cross-modal molecular information retrieval method in the above-described method embodiments.

[0167] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network nodes. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0168] Those skilled in the art will understand that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer-readable storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer-readable storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer-readable storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0169] The above is a detailed description of the preferred embodiments of this application. However, this application is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this application. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A cross-modal molecular information retrieval method, characterized in that, The method comprises the following steps: obtaining multi-modal data representing molecular information from a preset data set, wherein the multi-modal data comprises text data, image data and graph structure data; constructing a multi-modal molecular information model, wherein the multi-modal molecular information model comprises a modal extraction encoder and a modal fusion encoder, the modal extraction encoder comprises a text encoder, an image encoder and a graph structure encoder, and the modal extraction encoder is used to obtain a plurality of single-modal representations from the multi-modal data, wherein the single-modal representations comprise a text modal representation, an image modal representation and a graph structure modal representation, the modal fusion encoder comprises a plurality of cross-attention layers, and the modal fusion encoder is used to input the text modal representation, the image modal representation and the graph structure modal representation representing the same molecular information as a modal group into the plurality of cross-attention layers, calculate a set of fusion weight coefficients corresponding to the modal group by using a preset normalization exponential function in the plurality of cross-attention layers, and obtain a multi-modal representation by performing weighted calculation on each modal representation in the modal group according to the set of fusion weight coefficients; inputting the multi-modal data into the multi-modal molecular information model, calculating a contrast loss corresponding to the plurality of single-modal representations, training and optimizing the modal extraction encoder according to the contrast loss, and calculating a representation similarity between the text modal representation, the image modal representation, the graph structure modal representation and the multi-modal representation, and calculating a fusion loss according to the representation similarity, and updating the set of fusion weight coefficients in the modal fusion encoder with the goal of minimizing the fusion loss; obtaining target modal data of target molecular information and a target query condition, inputting the target modal data into the multi-modal molecular information model, and obtaining a multi-modal retrieval result corresponding to the target modal data and the target query condition. 2.The cross-modality molecular information retrieval method of claim 1, wherein, The method comprises the following steps: inputting the text data into the text encoder to obtain a text modal representation; inputting the image data into the image encoder to obtain an image modal representation; inputting the graph structure data into the graph structure encoder to obtain a graph structure modal representation.

3. The cross-modality molecular information retrieval method of claim 2, wherein, The method comprises the following steps: generating a positive sample pair according to the text data, the image data and the graph structure data representing the same molecular information; generating a negative sample pair according to the text data, the image data and the graph structure data representing different molecular information; substituting the positive sample pair and the negative sample pair into a preset triplet loss function to calculate a first loss and a second loss, respectively, wherein the first loss represents a contrast learning loss between the text modal representation and the image modal representation, and the second loss represents a contrast learning loss between the text modal representation and the graph structure modal representation; determining a sum of the first loss and the second loss as a contrast loss, and training and optimizing the modal extraction encoder with the goal of minimizing the contrast loss. 4.The cross-modality molecular information retrieval method of claim 1, wherein, The target modality data is input into the multi-modal molecular information model to obtain a multi-modal retrieval result corresponding to the target modality data and the target query condition, including: The target modality data is input into the multi-modal molecular information model to obtain a target modality representation corresponding to the target modality data, the target modality representation including a target single-modality representation and a target multi-modality representation; Similarity scores between the target modality representation and modality representations corresponding to a plurality of multi-modal data in the preset data set are calculated; The modality representations corresponding to the plurality of multi-modal data are sorted according to the similarity scores, and a multi-modal retrieval result is obtained according to a sorting result and the target query condition.

5. The cross-modality molecular information retrieval method of claim 4, wherein, The similarity scores between the target modality representation and modality representations corresponding to a plurality of multi-modal data in the preset data set are calculated, including: A plurality of feature pairs are determined between the modality representations corresponding to the multi-modal data in the preset data set and the target modality representation; Based on a preset molecular knowledge data set, a plurality of feature weights of the plurality of feature pairs are calculated in an unsupervised learning manner; Cosine similarities corresponding to the plurality of feature pairs are calculated; The plurality of cosine similarities are feature-weighted to obtain similarity scores. 6.The cross-modality molecular information retrieval method of claim 4, wherein, The multi-modal retrieval result is obtained according to the sorting result and the target query condition, including: A target query modality type corresponding to the target query condition is determined; Modality representations of a plurality of multi-modal data corresponding to the sorting result are type-filtered according to the target query modality type to obtain a multi-modal retrieval result conforming to the target query modality type.

7. The cross-modality molecular information retrieval method of claim 1, wherein, The text data in the multi-modal data is obtained in the following manner: Original text data in the preset data set is text-filtered to obtain scientific related text; The scientific related text is text-cleaned to obtain cleaned text; The cleaned text is subjected to key information extraction and text proofreading to obtain text data representing molecular information.

8. A cross-modal molecular information retrieval system, characterized by, Including: A data acquisition module is configured to obtain multi-modal data representing molecular information from a preset data set, the multi-modal data including text data, image data, and graph structure data; The model construction module is configured to construct a multi-modal molecular information model, which comprises a modal extraction encoder and a modal fusion encoder. The modal extraction encoder comprises a text encoder, an image encoder, and a graph structure encoder, and is configured to obtain a plurality of single-modal representations from the multi-modal data, wherein the single-modal representations comprise a text modal representation, an image modal representation, and a graph structure modal representation. The modal fusion encoder comprises a plurality of cross-attention layers, and is configured to input the text modal representation, the image modal representation, and the graph structure modal representation representing the same molecular information as a modal group into the plurality of cross-attention layers, calculate a plurality of fusion weight sets corresponding to the modal group by using a preset normalization exponential function in the plurality of cross-attention layers, and calculate each modal representation in the modal group by using the fusion weight sets to obtain a multi-modal representation. The model training module is configured to input the multi-modal data into the multi-modal molecular information model, calculate a plurality of contrast losses corresponding to the single-modal representations, train and optimize the modal extraction encoder according to the contrast losses, calculate a representation similarity between the text modal representation, the image modal representation, the graph structure modal representation, and the multi-modal representation, calculate a fusion loss according to the representation similarity, and update the fusion weight sets in the modal fusion encoder by taking minimizing the fusion loss as an objective. The information retrieval module is configured to obtain target modal data of target molecular information and a target query condition, input the target modal data into the multi-modal molecular information model, and obtain a multi-modal retrieval result corresponding to the target modal data and the target query condition.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the cross-modal molecular information retrieval method in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer program is stored in the computer readable storage medium and is executed by the processor to implement the cross-modal molecular information retrieval method in any one of claims 1 to 7.

Images