Cross-modal molecular retrieval-oriented curriculum learning framework, product, medium and equipment

Abstract

The invention discloses a cross-modal molecular retrieval-oriented course learning framework, a product, a medium and equipment, and belongs to the technical field of artificial intelligence. The framework includes a multi-modal encoder configured to be capable of converting an input sample into a textual representation and a molecular representation; the sample difficulty quantizer is configured to perform difficulty grade division according to the average value of the text similarity and the molecular similarity of the samples and the instances in the training set, and quantify the sample difficulty; the sample scheduler is configured to introduce a curriculum learning strategy, samples can be sorted according to difficulty levels, and sample sets which are different in difficulty and can be applied to different training stages are obtained; and the adaptive intensity learning module is configured to dynamically adjust the training intensity according to the sample complexity of the model in different stages based on the sample set of the sample scheduler. According to the method, the problem that the performance and efficiency of a cross-modal text-molecule retrieval method cannot be optimized at the same time can be solved, and the performance is obviously improved when the method is compatible with any backbone network.

Description

Technical Field

[0001] The present invention belongs to the technical field of artificial intelligence, and particularly relates to a curriculum learning framework, product, medium and device for cross-modal molecular retrieval. Background Art

[0002] In fields such as drug research and development, developing a cross-modal text-molecule retrieval model is of great significance for studying molecular structures. However, existing models lack dynamic perception of sample difficulty and their learning status during training.

[0003] Existing cross-modal text-molecule retrieval methods can be divided into three categories: statistical-based methods, hash-based methods, and deep learning-based methods.

[0004] 1. Statistical-based methods: By analyzing the feature correlation in subspace projection to bridge the modality gap, maximizing the correlation between text and molecular embeddings in the shared latent space to achieve modality alignment, providing a basic solution for early cross-modal alignment.

[0005] 2. Hash-based methods: Mapping high-dimensional multi-modal features into binary codes to achieve efficient similarity search, projecting text and molecular representations into a shared Hamming space, and accelerating retrieval by comparing binary codes.

[0006] 3. Deep learning-based methods: Mapping semantic representations of different modalities into a shared embedding space to facilitate similarity evaluation. Methods such as adversarial learning, contrastive learning, and optimal transport are used to achieve multi-modal and multi-granularity alignment.

[0007] Existing cross-modal text-molecule retrieval methods have the following several defects:

[0008] 1. The contradiction between efficiency and complexity: Although methods such as adversarial learning can improve model performance, they also lead to an increase in model complexity and require a large amount of training resources.

[0009] 2. Lack of dynamic perception: Existing models lack dynamic perception of sample difficulty and learning status, are prone to overfitting on simple samples and underfitting on complex samples, and lack a dynamic adjustment strategy during the training phase, unable to adaptively optimize the training intensity according to the learning progress.

[0010] 3. Semantic relationship recognition problem: Existing methods may ignore or fail to capture semantic relationships between different modalities, affecting the accuracy of retrieval. Summary of the Invention

[0011] The present invention aims to solve at least one of the technical problems in the above related technologies to a certain extent.

[0012] To this end, the purpose of the present invention is to provide a curriculum learning framework, product, medium, and device for cross-modal molecular retrieval, which can solve the problem that the performance and efficiency of cross-modal text-molecular retrieval methods cannot be optimized simultaneously, and is compatible with any backbone network to achieve a significant improvement in performance.

[0013] To solve the above technical problems, the present invention is implemented as follows:

[0014] An embodiment of the present invention provides a curriculum learning framework for cross-modal molecular retrieval, and the framework includes:

[0015] A multi-modal encoder, configured to be able to convert input samples into text representations and molecular representations;

[0016] A sample difficulty quantizer, configured to be able to calculate the text similarity and molecular similarity between the input sample and each instance in the training set according to the cosine similarity of the input sample, and divide the difficulty level according to the average value of the two-modal similarities, thereby quantifying the difficulty of each sample;

[0017] A sample scheduler, configured to introduce a curriculum learning strategy, and be able to sort samples according to difficulty levels to obtain a sample set with different difficulties that can be applied in different training stages;

[0018] An adaptive intensity learning module, configured to be able to dynamically adjust the training intensity based on the sample set of the sample scheduler according to the sample complexity of the model in different stages.

[0019] In addition, according to the curriculum learning framework for cross-modal molecular retrieval of the present invention, it may also have the following additional technical features:

[0020] In some of the embodiments, the multi-modal encoder includes a text encoder and a molecular encoder.

[0021] In some of the embodiments, the text encoder is SciBERT;

[0022] The molecular encoder is Mol2vec.

[0023] In some of the embodiments, the work content of the text encoder includes:

[0024] Obtain a text description;

[0025] Connect the [CLS] token with the title of the text sequence to obtain the required sequence;

[0026] Input the sequence into the text encoder to extract the feature vector corresponding to [CLS] as the text representation; and input the sequence into the molecular encoder, thereby converting the molecular graph structure into a sentence of substructures to obtain the molecular representation.

[0027] In some of these embodiments, the way of dividing the difficulty levels in the sample difficulty quantizer is as follows:

[0028] Compare the average value of the two-modal similarity of the current input sample with a preset similarity threshold;

[0029] Count the number of instances in the training set that exceed the similarity threshold;

[0030] Take the counted number as the difficulty level of the input sample.

[0031] In some of these embodiments, the sample scheduler integrates the curriculum learning strategy into the entire training cycle to promote the retrieval learning of the model; the samples in the training set are arranged from easy to difficult as the k-th step parameter increases.

[0032] In some of these embodiments, the adaptive intensity learning module adaptively adjusts the training objective of the model according to the learning intensity curve;

[0033] The training objective of the model is obtained by calculating the loss of each introduced training sample through the sample scheduler.

[0034] The embodiment of the present invention also provides a computer program product, including a computer program, which when executed by a processor implements the content of the curriculum learning framework for cross-modal molecular retrieval as described in any one of the above.

[0035] The embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored, which when executed by a processor implements the content of the curriculum learning framework for cross-modal molecular retrieval as described in any one of the above.

[0036] The embodiment of the present invention also provides a computer device, including: a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor executes the computer program to implement the content of the curriculum learning framework for cross-modal molecular retrieval as described in any one of the above.

[0037] Compared with the prior art, the present invention has at least the following beneficial effects:

[0038] In the embodiment of the present invention, the provided curriculum learning framework for cross-modal molecular retrieval can sort the samples according to the difficulty by quantifying the sample difficulty and introducing the curriculum learning strategy, obtain a sample set with different difficulties that can be applied in different training stages, so as to arrange the training samples step by step according to the difficulty, optimize the model training and reduce the sample requirements;

[0039] In the embodiments of the present invention, the provided curriculum learning framework for cross-modal molecular retrieval can dynamically adjust the training objectives at different curriculum stages by introducing adaptive intensity learning, which can prevent overfitting on simple samples and improve the generalization ability in complex scenarios.

[0040] Using the curriculum learning framework for cross-modal molecular retrieval of the present invention can improve the model performance while achieving significant time savings.

[0041] Additional aspects and advantages of the present invention will be given in part in the following description, will become apparent in part from the following description, or will be understood through the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a schematic diagram of the working structure of the curriculum learning framework for cross-modal molecular retrieval disclosed in an embodiment of the present invention;

[0043] Figure 2 It is a bar chart of the proportion of training samples used by ORMA disclosed in the present invention and CLASS(ORMA) of the present invention during the entire training period;

[0044] Figure 3 It is a bar chart of the proportion of training samples used by the existing AMAN disclosed in the present invention and CLASS(AMAN) of the present invention during the total training period. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0045] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0046] Next, the embodiments of the present invention will be described in detail through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0047] Please refer to Figure 1 As shown, in some embodiments of the present invention, a curriculum learning framework for cross-modal molecular retrieval is provided. The framework is implemented using a cross-modal molecular training framework based on curriculum learning (CLASS), and includes a multi-modal encoder, a sample difficulty quantizer, a sample scheduler, and an adaptive intensity learning module.

[0048] When the curriculum learning framework of the present invention works:

[0049] First, the multi-modal encoder is responsible for processing data and inputting samples (z i ) and the training set (z jThe text and molecules in ( ) generate text representations and molecular representations through a text encoder and a molecular encoder respectively.

[0050] Next, the processed data is sent to a sample difficulty quantizer to calculate the similarity between samples. If there are many similar samples during training, the difficulty for the model to learn to distinguish these similar samples will increase. Therefore, this step measures the difficulty of samples according to the number N of similar samples. i to measure the difficulty of samples.

[0051] Then, the sample scheduler can introduce samples in the order from easy to difficult as training progresses based on the curriculum learning strategy.

[0052] Finally, the adaptive intensity learning gradually increases the training intensity as the complexity of the training samples increases. Dynamically adjust the training intensity of the model at all curriculum stages to control the global training process of the model.

[0053] In the entire framework process, each part is either closely connected or overall coordinated, and the simplified and dynamic regulation of model training is achieved with a precise structure, improving the efficiency and performance of the model.

[0054] In some embodiments of the present invention, the multimodal encoder is composed of a text encoder and a molecular encoder, and its role is to convert the text-molecule pairs z i 、z j in the input samples or training sets into text representations and molecular representations respectively. The working principle of the multimodal encoder is as follows: The text encoder uses SciBERT, which performs well in the chemical field. For a text description t, first connect the [CLS] token with the title of the text sequence, and then input the sequence into SciBERT to extract the feature vector corresponding to [CLS] as the text representation h t . m The molecular representation h

[0055] In some embodiments of the present invention, the working principle of the sample difficulty quantizer is: For a given text-molecule pair z i , first calculate its text similarity j and molecular similarity with each instance z in the training set according to the cosine similarity of formula (1) and formula (2), so as to obtain the average value S of the two-modal similarities shown in formula (3) i，j :

[0056]

[0057]

[0058]

[0059] Calculate the number N of samples in the training set whose similarity to the sample z i exceeds the threshold σ according to formulas (4) and (5): i to represent the difficulty level of the sample:

[0060]

[0061]

[0062] It is already certain that since it is not easy for the model to learn to distinguish similar samples, the more similar samples a sample has, the greater the challenge to the model during training. Therefore, the samples in the training set are sorted from low to high according to the number of similar samples to obtain the final training set Simply put, it is the set in which the samples follow an order from easy to difficult. During model training, arrange the samples in the order of so that the model gradually learns and adapts to samples of different difficulty levels.

[0063] In some embodiments of the present invention, the working principle of the sample scheduler includes: designing the sample scheduler according to the curriculum learning strategy in a paradigm from easy to difficult. Specifically, the curriculum learning strategy is integrated into the entire training cycle to promote the retrieval learning of the model. During this process, the samples in the training set will be arranged from easy to difficult as the k-th step parameter increases. In the k-th epoch, the proportion of samples used for training is:

[0064] λ = α + β·k (6)

[0065] where λ represents the proportion of samples introduced in the k-th epoch, and α and β are the initial proportion and growth factor of the training samples respectively. By studying different α and β, the training samples used for model learning in each epoch stage are from the set D k ={z i |z i ∈Z sort，i ≤λ|Z sort} composed. When all training samples are introduced into model learning, that is, |Z sort | = |D k |, the model continues to be optimized according to all samples in the remaining stages.

[0066] Previous cross-modal text-molecule retrieval work treated all samples equally, while the sample scheduler used in the present invention can quantify the similarity of samples, determine the difficulty level of samples according to the similarity, and arrange them into model training in sequence according to the training stage after sorting from easy to difficult. This means that the samples during model training change from static to dynamic, providing an implementation basis for the adaptive intensity learning in the next stage. It was found in the ablation experiment that when the text similarity used to quantify the sample difficulty in the sample scheduler was discarded, the performance of the model all decreased to a certain extent, indicating that combining the similarities of the two modalities when quantifying the sample difficulty is effective and helps to play a higher utility in curriculum learning.

[0067] In some embodiments of the present invention, the working principle of the adaptive intensity learning module includes: during the model training process, adaptive intensity learning is achieved through the curriculum learning strategy. For each curriculum stage k, while controlling the complexity of the training samples, the training objective is modified to dynamically adjust the degree of model training. The model learns simple samples in the initial stage of the curriculum, and at this time, the training intensity is appropriately reduced to prevent overfitting. As the curriculum progresses, the sample complexity increases, and at this time, the training intensity of the model is gradually increased to prompt the model to better learn complex modal features, thereby improving the performance and generalization ability of the model in complex scenarios. It is worth mentioning that the framework can be combined with any backbone network using methods such as contrastive learning or adversarial learning, which can bring an expected improvement to the performance of the model. More precisely, two learning intensity curves, formula (7) and formula (8), are designed, which can achieve adaptive curriculum learning as the number of epochs k increases:

[0068]

[0069]

[0070] The training objective of any model is adaptively adjusted according to the learning intensity curve γ ∈ {γ1, γ2}. As shown in formula (9), the training objective of the model in the curriculum learning stage k is defined as L epoch , which is calculated by the sample scheduler for the loss c of each introduced training sample. For a sample z i , v i = 1 represents that z i is selected to train the model in epoch k, and v i = 0 indicates that it is not selected. This strategy ensures that the model can be trained with low intensity in the curriculum learning of the simple sample stage and gradually increase the training intensity as the complexity of the training samples increases.

[0071]

[0072]

[0073] The adaptive intensity learning module of the present invention, based on the curriculum learning strategy, can dynamically adjust the training intensity according to the sample complexity of the model at different curriculum stages, rather than statically fixing the training target. This means that based on the sample scheduler sorting the sample difficulties, for the simple samples in the initial curriculum stage, the training intensity of the model will correspondingly decrease, and vice versa. Generally speaking, the adaptive intensity learning structure is designed to solve the problem of uneven sample difficulties. By dynamically regulating the training intensity of the model for samples of different difficulty levels, it not only optimizes the learning efficiency, avoids the problem of overfitting, but also enables the model to better learn complex modal features, thereby improving the performance and generalization ability of the model.

[0074] Table 1 shows the main results of the text-molecule retrieval task of the method of the present invention and the baseline model on the ChEBI-20 dataset. The best results are shown in bold. It can be seen from the table that both the AMAN and ORMA models using the CLASS framework are always better than the baseline model, with an improvement of 1.0%-1.9% in Hits@1 and a reduction of 1.76-3.35 in the average ranking. At the same time, compared with the relatively good baseline model Atomas-base, AMAN integrating the method of the present invention still has significant improvements of 1.0%, 0.5%, and 0.007 in Hits@1, Hits@10, and MRR. Table 2 also shows the main results of the molecule-text retrieval task under the same experimental conditions. The AMAN and ORMA models using the CLASS framework show better performance and are in the leading position in Hits@1 and MRR, with improvements of 1.1%-1.6% and 0.005-0.009 respectively.

[0075] The results shown in Table 1 and Table 2 indicate that the method using the CLASS framework demonstrates strong performance in both text-molecule retrieval and molecule-text retrieval. Especially when applied based on the AMAN model, the performance of this method is particularly outstanding. These scenarios can be regarded as the optimal usage states of this product.

[0076] Experimental Example 1:

[0077] The dataset of the present invention is the ChEBI-20 dataset collected from PubChem and ChEBI. It consists of 26,408 training samples, 3,301 validation samples, and 3,301 test samples, with a total of 33,010 pairs.

[0078] All experiments were completed on an A100 GPU. For text encoding, a vector representation with a dimension of 300 was learned using SciBERT, and molecular representations were generated via Mol2Vec. The AMAN and ORMA models were used as the backbone networks of the CLASS framework. The Adam optimizer was used throughout the experiment, with an initial learning rate fixed at 1e-4. The training period was fixed at 60 epochs, and the batch size was uniformly set to 32. Additionally, the hyperparameters of the model were optimized through grid search. When AMAN was used as the backbone network, the trade-off parameters for the text-molecule retrieval task were α = 40 and β = 3, while for the molecule-text retrieval task, they were α = 20 and β = 4; when using the ORMA model, for text-molecule retrieval, α = 70 and β = 3, and for molecule-text retrieval, α = 60 and β = 2. The similarity threshold σ for sample difficulty assessment was set to 0.99.

[0079] The evaluation in the present invention includes two tasks: text-molecule retrieval and molecule-text retrieval. Similar to previous studies, considering the one-to-one correspondence between texts and molecules in the dataset, the present invention uses the mean reciprocal rank (MRR), mean rank (MeanRank), Hits@1, and Hits@10 as evaluation metrics, where:

[0080] MRR: It is a metric used to evaluate the quality of results in information retrieval and other fields, which calculates the average of the reciprocals of the ranks of the first correct answer in a set of queries.

[0081] Mean rank: The average rank of relevant items in a ranked list. It sums up the ranks of all relevant items in a set of queries and then divides by the number of queries.

[0082] Hits@1: It is a measurement metric that refers to the proportion of queries in which the correct answer is retrieved as the first candidate result during the retrieval process.

[0083] Hits@10: Similar to Hits@1, but instead of checking whether the correct answer is ranked first, it checks whether the correct answer is among the top 10 results in the ranked list.

[0084] To verify the superiority of the CLASS framework, the following methods were used as baselines for comparison in the experiment:

[0085] MLP-Ensemble, GCN-Ensemble, All-Ensemble: The Ensemble baselines use models based on MLP or GCN to encode molecules for retrieval learning, and each model is initialized with different parameters.

[0086] MLP+Atten, MLP+FPG: The MLP+Atten baseline uses cross-modal attention to capture information and re-rank the retrieval results. MLP+FPG is consistent with the MLP+Atten baseline and uses the FPGrowth algorithm to re-rank the top results in the MLP model.

[0087] Atomas-base: This model is pre-trained on a large-scale dataset where molecules are represented by Simplified Molecular-Input Line-Entry System (SMILES), and the two modalities are aligned at three granularities through a clustering method. Atomas is trained on the ChEBI-20 dataset with the same configuration as this study.

[0088] MemoryBank: This method proposes a memory bank-based feature projector to extract modality-shared features and calculates four similarities to narrow the gap.

[0089] The present invention conducts ablation experiments on the above test data to study the specific effects of the sample scheduler and adaptive intensity learning. The present invention conducts ablation experiments based on CLASS(AMAN) in two retrieval tasks, and the results are shown in Table 3. When the sample scheduler no longer quantifies the text similarity of sample difficulty, the performance of the model decreases to varying degrees in both retrieval tasks. At the same time, ablating the molecular similarity when quantifying the sample difficulty causes the model performance to decrease by 1.6%-1.9%, 0.7%-0.8%, 0.02-0.014, and 0.26-1.12 in all metrics. Eliminating the adaptive intensity learning strategy results in a 0.8%-2.3% and 0.013-0.009 decrease in the performance of Hits@1 and MRR, respectively. This shows that both quantifying the sample difficulty by combining the similarities of the two modalities and controlling the training intensity are effective.

[0090] Table 1 Results of the text-molecule retrieval task of the present invention and the baseline on the ChEBI-20 dataset

[0091]

[0092] Table 2 Results of the molecule-text retrieval task of the present invention and the baseline on the ChEBI-20 dataset

[0093] Model Hits@1(↑) Hits@10(↑) MRR(↑) MeanRank(↓) All-Ensemble 25.2％ 74.1％ 0.408 21.77 Atomas-base 45.6％ 90.3％ 0.614 15.12 MemoryBank 52.3％ 93.3％ 0.673 12.29 AMAN 46.2％ 91.1％ 0.622 19.93 CLASS(AMAN) 47.8％ 91.6％ 0.631 15.14 ORMA 60.9％ 93.2％ 0.733 10.71 CLASS(ORMA) 62.0％ 92.7％ 0.738 14.59

[0094] Table 3 Ablation study of the present invention based on AMAN

[0095]

[0096] Table 4: Influence of different learning intensity curves in adaptive intensity learning on the text-molecule retrieval task

[0097] Learning intensity curve Hits@1(↑) Hits@10(↑) MRR(↑) MeanRank(↓) γ1 50.5％ 91.6％ 0.653 20.50 γ2 51.1％ 92.6％ 0.660 16.80

[0098] Figure 2 is a bar chart of the proportion of training samples used by ORMA and the present invention CLASS(ORMA) during the entire training period; Figure 3 is a bar chart of the proportion of training samples used by the existing AMAN and the present invention CLASS(AMAN) during the total training period.

[0099] For the parts not described in detail in the present invention, reference can be made to the prior art or the well-known technologies to those skilled in the art. This embodiment does not make any limitations in this regard and will not be described in detail herein.

[0100] The embodiments of the present invention have been described above in conjunction with the accompanying drawings. However, the present invention is not limited to the above specific embodiments. The above specific embodiments are merely illustrative and not restrictive. Under the inspiration of the present invention, those of ordinary skill in the art can also make many forms without departing from the spirit of the present invention and the scope protected by the claims, and all of them fall within the protection scope of the present invention.

Claims

1. A curriculum learning framework for cross-modal molecular retrieval, characterized in that The framework includes: A multi-modal encoder configured to convert an input sample into a text representation and a molecular representation; A sample difficulty quantizer configured to calculate the text similarity and molecular similarity between the input sample and each instance in the training set according to the cosine similarity of the input sample, and divide the difficulty level according to the average value of the two-modal similarities, so as to quantify the difficulty of each sample; A sample scheduler configured to introduce a curriculum learning strategy, capable of sorting samples according to difficulty, and obtaining a sample set of different difficulties that can be applied in different training stages; An adaptive intensity learning module configured to dynamically adjust the training intensity based on the sample set of the sample scheduler according to the sample complexity at different stages of the model.

2. The curriculum learning framework for cross-modal molecular retrieval according to claim 1, characterized in that The multi-modal encoder includes a text encoder and a molecular encoder.

3. The curriculum learning framework for cross-modal molecule retrieval according to claim 2, wherein The text encoder is SciBERT; the molecular encoder is Mol2vec.

4. The curriculum learning framework for cross-modal molecular retrieval according to claim 2, characterized in that, The working content of the text encoder includes: Obtaining a text description; Connecting the [CLS] token with the title of the text sequence to obtain the required sequence; Inputting the sequence into the text encoder to extract the feature vector corresponding to [CLS] as the text representation; and inputting the sequence into the molecular encoder, thereby converting the molecular graph structure into a sentence of sub-structures to obtain the molecular representation.

5. The curriculum learning framework for cross-modal molecular retrieval according to claim 1, wherein The method for dividing the difficulty level in the sample difficulty quantizer is: Comparing the average value of the two-modal similarities of the current input sample with a preset similarity threshold; Counting the number of instances in the training set that exceed the similarity threshold; Taking the counted number as the difficulty level of the input sample.

6. The curriculum learning framework for cross-modal molecular retrieval according to claim 1, wherein The sample scheduler integrates the curriculum learning strategy into the entire training cycle to promote the retrieval learning of the model; The samples in the training set are arranged from easy to difficult as the value of the k-th step parameter increases.

7. The curriculum learning framework for cross-modal molecule retrieval according to claim 1, characterized in that, The adaptive intensity learning module adaptively adjusts the training objective of the model according to the learning intensity curve; The training objective of the model is obtained by calculating the loss of each introduced training sample through the sample scheduler.

8. A computer program product comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the content of the curriculum learning framework for cross-modal molecular retrieval described in any one of claims 1-7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the content of the curriculum learning framework for cross-modal molecular retrieval described in any one of claims 1-7.

10. A computer device, comprising: A memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the computer program to implement the content of the curriculum learning framework for cross-modal molecular retrieval described in any one of claims 1-7.

Images