A data selection method and device based on protein pre-training scenarios

Abstract

The application discloses a data screening method and device based on protein pre-training scene, comprising: obtaining protein data, and representing each protein data as a three-dimensional graph structure; using a graph coding model based on a graph neural network to encode each three-dimensional graph structure to obtain a hidden vector of the protein; modeling the data screening problem as a maximum coverage problem, defining a sample volume according to the distance between the hidden vector of each protein and the hidden vector of other proteins, and screening the protein data according to the sample volume. The method does not involve human intervention, and screens the protein data in an end-to-end manner, and guides the screening through the characteristics of the data itself, thereby guaranteeing the theoretical rationality and robustness of the screening, and being widely applied to various scenes, and achieving a balance between the minimum data size and the optimal performance on various data. The screened protein data is directly applied to pre-training, thereby improving the training efficiency and guaranteeing the global optimality of the solution.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of artificial intelligence and protein combination, and particularly relates to a data selection method and device based on a protein pre-training scene. BACKGROUND

[0002] Pre-trained models are ubiquitous in current artificial intelligence research. The general patterns obtained from pre-trained models trained on massive amounts of unlabeled data can be well applied to different downstream tasks. These methods have achieved new state-of-the-art performance in many fields, from language modeling and translation to image understanding and network analysis, as well as knowledge graph representation and protein function and structure prediction.

[0003] The study of protein function and structure using neural networks relies on a large amount of unlabeled protein data. Protein sequences and protein structures are naturally rich in information, and many studies currently use deep learning models to learn embedding representations of protein sequences or protein structures to facilitate the prediction of protein properties and functions, protein folding structures, and the binding of large and small molecules. However, protein molecules have a much larger molecular weight than general small molecules, and the corresponding data is more complex. In the neural network training process, there are problems such as long training time and consumption of a large amount of training resources. Existing research has shown that data quality has a great impact on the effectiveness of pre-trained models. If a dataset that has not been well screened is used, the model's generalization ability is poor, and it cannot fully utilize the advantages of neural network models in many practical problems, limiting its application in practical scenarios.

[0004] Generally speaking, larger models have stronger ability to capture more complex data patterns, resulting in better performance. Based on this belief, recent pre-trained models have become very large, usually with millions or even billions of neurons as parameters. For example, a Transformer has about 1.6 trillion parameters. Good training of this model requires a huge amount of data, and the Transformer uses a 750GB Colossal Clean Crawled Corpus dataset. Although unlabeled data can be easily obtained, they are usually unfiltered and may contain redundant data, or even have a negative impact on the pre-training of models for certain specific tasks. In addition, more data requires more computing resources, and many artificial intelligence researchers cannot obtain such resources. Taking the Switch Transformer as an example, it takes nearly three days to pre-train on 30 TPUs, and only a few people can afford such resource overhead. SUMMARY

[0005] The pre-training model for the above protein is faced with problems of too large training data and too much training resource consumption, and existing models have problems of too high artificial participation and poor generalization ability, and the application proposes a data screening method and device based on a protein pre-training scene, which models and prunes data using a tree structure and combines downstream scenes for joint optimization.

[0006] To achieve the above application purposes, the embodiment provides a data screening method based on a protein pre-training scene, which comprises:

[0007] Protein data is acquired, and each piece of protein data is represented as a three-dimensional graph structure;

[0008] A graph encoding model based on a graph neural network is used to encode each three-dimensional graph structure to obtain a hidden vector of the protein;

[0009] The data screening problem is modeled as a maximum coverage problem, the sample volume is defined according to the distance between the hidden vector of each piece of protein data and the hidden vector of other protein data, and the protein data is screened according to the sample volume.

[0010] Preferably, each piece of protein data is represented in a PDB format, including atomic coordinates, atomic species and chemical bond information, and in the represented three-dimensional graph structure, the atomic coordinates and atomic species are nodes, and the chemical bond information between atoms represents the edges between nodes.

[0011] Preferably, the graph encoding model is a pluggable model, including SchNet, PNA, SphereNet and PointNet.

[0012] Preferably, the sample volume is defined according to the distance between the hidden vector of each piece of protein data and the hidden vector of other protein data, and the protein data is screened according to the sample volume, comprising:

[0013] An empty set is initialized, and a priority queue is created;

[0014] Selection process: for the hidden vector of the protein in the current space, the partition dimension is calculated based on the KD algorithm, that is, the variance of the element value in each dimension is calculated, the value corresponding to the median of the dimension corresponding to the maximum variance is saved to the set, and the dimension corresponding to the maximum variance is selected as the partition dimension, the current space is divided into two subspaces according to the partition dimension, and the volume of each subspace is calculated, each subspace is pushed into the priority queue, and each subspace is sorted in descending order according to the volume in the priority queue;

[0015] The subspace with the largest volume in the priority queue is taken as the current space to execute the selection process until the amount of data in the set meets the requirement.

[0016] Preferably, the calculating the volume of each subspace comprises:

[0017] Multiplying the element values of all dimensions within the subspace as the volume of the subspace.

[0018] Preferably, the screening method further comprises: further screening in combination with protein data of a downstream task, comprising:

[0019] According to the optimal transport algorithm, a transfer matrix of the protein data screened according to the sample volume to the downstream task data is calculated, and the protein data screened according to the sample volume is further screened according to the transfer matrix to obtain data having a correlation density with the downstream task data.

[0020] Preferably, the method further comprises: training a downstream task model using the data having the correlation density with the downstream task data.

[0021] To achieve the above-mentioned object, the embodiments further provide a data screening device based on a protein pre-training scenario, comprising:

[0022] A graph construction module is configured to obtain protein data and represent each piece of protein data as a three-dimensional graph structure;

[0023] An encoding module is configured to encode each three-dimensional graph structure using a graph encoding model based on a graph neural network to obtain an implicit vector of the protein;

[0024] A screening module is configured to model a data screening problem as a maximum coverage problem, define a sample volume according to the distance between the implicit vector of each protein and the implicit vectors of other proteins, and screen the protein data according to the sample volume.

[0025] Preferably, in the screening module, the sample volume is defined according to the distance between the implicit vector of each protein and the implicit vectors of other proteins, and the protein data is screened according to the sample volume, comprising:

[0026] An empty set is initialized, and a priority queue is created;

[0027] Selection process: for the implicit vector of the protein in the current space, the partition dimension is calculated based on the KD algorithm, that is, the variance of the element values in each dimension is calculated, the value corresponding to the median of the dimension corresponding to the maximum variance is saved to the set, and the dimension corresponding to the maximum variance is screened as the partition dimension, the current space is divided into two subspace according to the partition dimension, and the volume of each subspace is calculated, each subspace is pushed into the priority queue, and each subspace is sorted in descending order according to the volume in the priority queue;

[0028] The subspace with the largest volume in the priority queue is taken as the current space to execute the selection process until the amount of data in the set meets the requirement.

[0029] Preferably, it further comprises a re-screening module for further screening of protein data combined with downstream task data, including:

[0030] According to the optimal transmission algorithm, a migration matrix of the protein data screened according to the sample volume to the downstream task data is calculated, and the protein data screened according to the sample volume is further screened according to the migration matrix, to obtain data with a correlation density with the downstream task data.

[0031] Compared with the prior art, the present application has at least the following beneficial effects:

[0032] On the basis of encoding the protein data as the latent vector of the protein, the sample volume is defined according to the distance between the latent vector of each protein and the latent vector of other proteins, and the protein data is screened according to the sample volume. This process does not involve human intervention, and the protein data is screened end-to-end, guided by the characteristics of the data itself, ensuring the theoretical rationality and robustness of the screening, and the application scenario is wide, and the balance between the minimum data size and the optimal performance is achieved on various data. The screened protein data is directly applied to pre-training, improving the training efficiency, while ensuring the global optimality of the solution. BRIEF DESCRIPTION OF DRAWINGS

[0033] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0034] Figure 1 is a flowchart of the data screening method provided by the embodiment under the protein pre-training scenario;

[0035] Figure 2 is a flowchart of the data screening method provided by the embodiment under the protein pre-training scenario;

[0036] Figure 3 is a structural schematic diagram of the data screening device provided by the embodiment under the protein pre-training scenario. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical scheme and advantages of the present application more clear, the following will further describe the present application in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and do not limit the protection scope of the present application.

[0038] The purpose of this invention is to find a subset from a given protein dataset while preserving key information and relevant knowledge for downstream tasks. However, few methods have systematically studied how to filter sample data to train better pre-trained models with limited sample data. Existing methods can be roughly divided into two groups: (1) manual design of selection strategies based on intuition, but these methods cannot adapt to different tasks without careful tuning; (2) methods that score and rank data using specific metrics and then select the top few items as the training set. However, both of these methods can only capture some data attributes, and in this sense, they cannot achieve the selection of general and adaptive data from the pre-trained dataset.

[0039] Ideally, data filtering methods should be adaptable to different data and tasks, meaning the sampling process should be unaffected by human factors. In this invention, semantics is considered as an indicator for extracting important samples from the original protein dataset. Intuitively, if a subset of samples has nearly identical vector representations in the feature space, then they likely describe the same things in the real world. Therefore, if we want to reduce the size of the protein dataset, we need to select only a small number of samples from this subset as representatives of this type of semantics.

[0040] In this embodiment, the data filtering process is modeled as a maximum coverage problem. The volume is defined based on the distance between the latent vectors of the sample data and the latent vectors of other selected sample data. Sample data is selected based on the volume, and sample data is further filtered based on the correlation density with downstream task data.

[0041] Based on the above analysis, such as Figure 1 As shown in the embodiment, the data filtering method based on protein pre-training scenario includes the following steps:

[0042] Step 1: Acquire protein data and characterize each protein data point as a three-dimensional graph structure.

[0043] In the embodiment, the protein data is derived from Protein Bank, AlphaFold and other data sets, and is stored in PDB (Protein Data Bank) file format, which can be read and written by many programs. The complete PDB file specification provides a lot of information, including author, literature reference and structure determination method. The PDB format is composed of information lines in a text file. Each line of information in the file is called a record. A PDB file usually contains several different types of records arranged in a specific order to describe a protein structure. The PDB format is a file standard containing atomic coordinates. Through the PDB file, atomic coordinates, atomic species and chemical bond information can be obtained, and based on the obtained data, a three-dimensional graph structure is constructed, in which atomic coordinates and atomic species are nodes, and chemical bond information between atoms represents edges between nodes.

[0044] Step 2, encoding each three-dimensional graph structure by using a graph encoding model based on a graph neural network to obtain a hidden vector of the protein.

[0045] In the embodiment, the graph encoding model for extracting data hidden vectors needs to have strong expression ability, so that the distribution of the hidden vector in the high-dimensional vector space is related to its actual meaning. Therefore, the graph encoding model adopts SchNet, PNA, SphereNet, PointNet and the like, especially SphereNet. The three-dimensional graph corresponding to each protein data is input into the graph encoding model, and a hidden vector containing semantics is obtained after calculation.

[0046] The hidden vectors of the original protein data set are scattered in a high-dimensional vector space. Under the representation of the graph encoding model with strong expression ability, the hidden vectors can represent the semantics of the original protein data to some extent. The semantics are used as an index for extracting important sample data from the original protein data set. The area where the vectors are concentrated means that there are a large number of redundant data of proteins with similar structure and function; the sparse place of the vectors means that the number of proteins with this function and structure is small.

[0047] Step 3, modeling the data screening problem as a maximum coverage problem, defining a sample volume according to the distance between the hidden vector of each protein and the hidden vector of other proteins, and screening the protein data according to the sample volume.

[0048] In the embodiment, the data screening problem is modeled as a maximum coverage problem, and the data is optimized and selected by using the KD tree algorithm. The specific process includes:

[0049] (a) initialize an empty set, and create a priority queue in which a random element is stored.

[0050] (b) Selection process: for the latent vector of the protein in the current space, calculate the partition dimension based on the KD algorithm, that is, calculate the variance of the element value in each dimension, save the value corresponding to the median of the dimension corresponding to the maximum variance to the set, and at the same time, filter the dimension corresponding to the maximum variance as the partition dimension, divide the current space into two subspaces according to the partition dimension, and calculate the volume of each subspace, push each subspace into the priority queue, and sort each subspace in the priority queue in descending order according to the volume;

[0051] (c) The subspace with the largest volume in the priority queue is taken as the current space to execute the selection process until the amount of data in the set meets the requirement.

[0052] The KD algorithm is a binary tree, and each node represents a subset of the data set. One feature of the KD tree is that each non-leaf node can also be regarded as a hyperplane, that is, a partition dimension, which divides the space into two subspaces, and each subspace contains half the number of variables.

[0053] In the embodiment, the volume of the subspace divided by the hyperplane corresponding to the partition dimension can be calculated by multiplying the element values of all dimensions in the subspace. Therefore, the volume of the subspace is directly calculated by the above method, and the subspace is pushed into the priority queue.

[0054] After the above screening process, the data obtained by simplifying the original protein data set containing redundant information can still retain the information amount of the original protein data.

[0055] Step 4, further screening of protein data combined with downstream tasks.

[0056] In the embodiment, for a certain specific downstream task, a new indicator is introduced to measure the importance of protein data in the protein data set to the downstream task data. Specifically, a transfer matrix is used to express the correlation between the upstream and downstream data. The present application hopes to only select a small part of sample data from the source protein data that is most relevant to the downstream task data (i.e. target data).

[0057] Specifically, the optimal transport algorithm is used to calculate the transfer matrix of the protein data screened according to the sample volume to the downstream task data, and the protein data screened according to the sample volume is further screened according to the transfer matrix to obtain data with a correlation density to the downstream task data.

[0058] The further screened protein data subset is a globally optimal data subset, and using the globally optimal data subset as the model upstream training set can maximize the performance of the model and improve the training efficiency.

[0059] Based on the same inventive concept, the embodiment also provides a data screening device based on protein pre-training scene, as shown in the figure, comprising a graph construction module, an encoding module and a screening module, wherein the graph construction module is used to obtain protein data and represent each protein data as a three-dimensional graph structure; the encoding module is used to encode each three-dimensional graph structure by using a graph encoding model based on a graph neural network to obtain a hidden vector of the protein; and the screening module is used to model a data screening problem as a maximum coverage problem, define a sample volume according to the distance between the hidden vector of each protein and the hidden vector of other proteins, and screen the protein data according to the sample volume. Figure 3

[0060] It should be noted that the data screening device based on the protein pre-training scene provided by the above embodiment should be illustrated by the division of the above functional modules when performing data screening, and the above functions can be completed by different functional modules as needed, that is, the internal structure of the terminal or server is divided into different functional modules to complete all or part of the functions described above. In addition, the data screening device based on the protein pre-training scene provided by the above embodiment and the data screening generation method embodiment based on the protein pre-training scene belong to the same concept, and the specific implementation process is detailed in the data screening method embodiment based on the protein pre-training scene, which will not be repeated here.

[0061] In the existing data screening method, data is screened for specific data and tasks, and the generalization performance of the method is poor. The data screening method and device based on the protein pre-training scene provided by the above embodiment do not join human intervention, screen the input data end to end, guide the screening through the characteristics of the data itself, and directly apply to pre-training, thereby improving the training efficiency and ensuring the global optimality of the solution.

[0062] A large number of research results have shown that data quality has a great influence on the performance of protein pre-training models. In the protein pre-training task, there are various pre-training tasks, each of which learns the functional and structural information of proteins from different aspects, and each pre-training task has its applicable scene, but they all rely on high-quality unlabeled protein data. The screening method and device provided by the embodiment ensure the theoretical rationality and robustness of screening, and are widely applicable in various data to balance between the minimum data size and the optimal performance.

[0063] The specific embodiments described above have detailed the technical solutions and beneficial effects of the present application. It should be understood that the above description is only the most preferred embodiment of the present application and is not intended to limit the present application. Any modifications, supplements and equivalent replacements made within the principle range of the present application shall be included in the protection scope of the present application.​

Claims

1. A data screening method based on protein pre-training scenarios, characterized in that, include: Acquire protein data and characterize each protein data point as a three-dimensional graph structure; The latent vector of the protein is obtained by encoding each three-dimensional graph structure using a graph coding model based on a graph neural network. Model the data filtering problem as a maximum coverage problem; The sample volume is defined based on the distance between the latent vectors of each protein and the latent vectors of other proteins. Protein data is then filtered based on the sample volume. Specifically, this involves: calculating the partitioning dimension for the latent vectors of proteins in the current space using the KD algorithm, i.e., calculating the variance of the element values ​​in each dimension, saving the median value corresponding to the dimension with the largest variance to the set, and selecting the dimension corresponding to the largest variance as the partitioning dimension. The current space is then divided into two subspaces based on the partitioning dimension, and the volume of each subspace is calculated. Each subspace is then prioritized and sorted in descending order of volume in the priority queue. The subspace with the largest volume in the priority queue is selected as the current space, and the selection process continues until the amount of data in the set meets the requirements.

2. The data filtering method based on protein pre-training scenario according to claim 1, characterized in that, Each protein data point is represented in PDB format, including atomic coordinates, atom types, and chemical bond information. In the three-dimensional graph structure, atomic coordinates and atom types serve as nodes, and the chemical bond information between atoms represents the edges between nodes.

3. The data filtering method based on protein pre-training scenario according to claim 1, characterized in that, The graph coding model is a pluggable model, including SchNet, PNA, SphereNet, and PointNet.

4. The data filtering method based on protein pre-training scenario according to claim 1, characterized in that, The calculation of the volume of each subspace includes: The volume of the subspace is the product of the element values ​​of all dimensions within the subspace.

5. The data filtering method based on protein pre-training scenario according to claim 1, characterized in that, The screening method further includes: further screening in conjunction with protein data from downstream tasks, including: The optimal transfer algorithm is used to calculate the migration matrix from protein data selected based on sample volume to downstream task data. Based on the migration matrix, the protein data selected based on sample volume is further filtered to obtain data with correlation density with downstream task data.

6. The data filtering method based on protein pre-training scenario according to claim 5, characterized in that, The method further includes training the downstream task model using data with correlation density to the downstream task data.

7. A data filtering device based on a protein pre-training scenario, characterized in that, include: The graph construction module is used to acquire protein data and characterize each protein data point as a three-dimensional graph structure. The encoding module is used to encode each three-dimensional graph structure using a graph encoding model based on graph neural networks to obtain the hidden vector of the protein. The filtering module is used to model the data filtering problem as a maximum coverage problem. It is also used to define the sample volume based on the distance between the latent vectors of each protein and the latent vectors of other proteins, and to filter protein data based on the sample volume. Specifically, it includes: calculating the partitioning dimension based on the KD algorithm for the latent vectors of proteins in the current space, that is, calculating the variance of the element values ​​in each dimension, saving the value corresponding to the median of the dimension with the largest variance to the set, and selecting the dimension corresponding to the largest variance as the partitioning dimension. Based on the partitioning dimension, the current space is divided into two subspaces, and the volume of each subspace is calculated. Each subspace is pushed into the priority queue and sorted in descending order of volume. The subspace with the largest volume in the priority queue is used as the current space to perform the selection process until the amount of data in the set meets the requirements.

8. The data filtering device based on protein pre-training scenario according to claim 7, characterized in that, It also includes a re-screening module for further screening of protein data in conjunction with downstream tasks, including: The optimal transfer algorithm is used to calculate the migration matrix from protein data selected based on sample volume to downstream task data. Based on the migration matrix, the protein data selected based on sample volume is further filtered to obtain data with correlation density with downstream task data.

Images