Gene function classification method and device based on Sensor-BERT and KMeans model

Abstract

The invention discloses a gene function classification method and a gene function classification device based on a Sension-BERT model and a KMeans model. The method comprises the following steps: obtaining data required by a model, wherein the data required by the model is listed subsequently; through pre-training and task fine tuning, the SBERT can better capture the fine-grained semantic difference in the gene function description, and the data is input into the pre-trained SBERT to generate Sensor Embedding; and continuously inputting the Sensor Embedding into the KMeans model for training, and adjusting parameters through an elbow method to obtain an optimal K value (a classification cluster number). Based on the established gene function summary knowledge base, a reliable method is provided for gene classification through a modeling method combining an SBERT deep learning model and a KMeans unsupervised machine learning model.

Description

Technical Field

[0001] The present invention relates to a gene function classification method based on Sentence-BERT (SBERT) and KMeans models, and also relates to a corresponding gene function classification method and device based on SBERT and KMeans models, belonging to the cross-research field of data science and bioinformatics. Background Art

[0002] Gene expression data or sequence features are often extremely high-dimensional, and direct input into deep learning models may lead to overfitting or feature redundancy. Gene function classification divides features into biologically meaningful subsets, which makes it easier for the model to learn the relationship between functional modules more effectively. Feature block input has the advantages of (1) reducing the complexity of the feature space, (2) incorporating biological knowledge and enhancing model interpretability, and (3) improving the ability of deep learning models to capture functional modules. Gene function classification is an important task in understanding biological systems. By analyzing the functional information of genes, we can reveal the role of genes in biological processes, the potential mechanisms of diseases, and the discovery of drug targets. Traditional text analysis methods (such as feature representation based on bag-of-words models) usually ignore the semantic information in the text, resulting in classification results that are difficult to accurately reflect the true relationship between gene functions.

[0003] Sentence-BERT (SBERT) based on the BERT model can learn deep semantic representations of natural language texts through deep neural networks and generate high-dimensional semantic embeddings. This method can effectively capture the potential semantics in gene function descriptions and make up for the shortcomings of traditional methods in semantic processing. However, due to the huge amount of data in gene function classification, how to effectively process high-dimensional gene embedding vectors remains a difficult problem.

[0004] The KMeans clustering algorithm is a classic unsupervised learning algorithm that can divide data into different clusters based on similarity. In gene function classification, the KMeans algorithm can classify genes with similar functions into one category by clustering the generated gene function embeddings, thereby forming a gene function module with biological significance. Summary of the invention

[0005] The primary technical problem to be solved by the present invention is to provide a gene function classification method based on Sentence-BERT and KMeans models.

[0006] Another technical problem to be solved by the present invention is to provide a gene function classification device based on Sentence-BERT and KMeans models.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] According to a first aspect of an embodiment of the present invention, a gene function classification method based on Sentence-BERT and KMeans model is provided, comprising the following steps:

[0009] Obtain the input data required by the model, which is a gene function description text library;

[0010] Step S2: Input the gene summary text into the Sentence-BERT model to generate vectorized SentenceEmbeddings;

[0011] Step S3: Use Sentence Embeddings to build a KMeans machine learning model and pre-train it to obtain the best hyperparameter K; use Sentence Embeddings and hyperparameter K to build the final KMeans clustering model;

[0012] Step S4: Output the clustering results, perform dimensionality reduction on the Sentence Embeddings and output a clustering effect diagram.

[0013] Preferably, when establishing a gene function description text library, the following processing is performed:

[0014] Step S11, gene summary text from NCBI (https: / / www.ncbi.nlm.nih.gov / ), GeneCards (https: / / www.genecards.org / ) and UniProtKB / Swiss-Prot (https: / / www.uniprot.org / ) databases;

[0015] Step S12: After obtaining the gene summary text, organize it into a dataframe and use Python and other related libraries to clean the data.

[0016] Preferably, when obtaining data, the following processing is performed:

[0017] Step 111: The examples cited in the present invention are based on the above three databases. These three databases are highly authoritative and all descriptions are non-repetitive, so as to establish a high-quality knowledge base for training.

[0018] Preferably, when performing data cleaning,

[0019] Step 121, converting text type data into string format;

[0020] Step 122, delete meaningless sentences, which do not describe gene functions and may misjudge the clustering effect;

[0021] Step 123, delete the corresponding gene names. The gene names are all non-repeating topics and are meaningless to the functional description of the genes. They are deleted to optimize the classification effect.

[0022] Step 124, convert all text into lowercase to facilitate subsequent processing;

[0023] Step 125, deleting common stop words in text processing. The common stop words used in the present invention are ENGLISH_STOP_WORDS in the Python library SKLEARN.

[0024] Step 126, deleting scattered single letters (26 lowercase letters) in the text description paragraph, which will cause ambiguity in the feature description;

[0025] Step 127, deleting additional stop words. The present invention considers that some uncommon stop words are meaningless in gene classification, and is provided here for reference;

[0026] Step 128: Delete unnecessary spaces, punctuation marks, line breaks, etc.

[0027] The better one is that when generating vectorized Sentence Embeddings,

[0028] Step S21: Input one or more sentences, segment them and convert them into tokens.

[0029] Step S22: Add special tags. Special tags [CLS] (beginning of sentence) and [SEP] (end of sentence) are added to each sentence so that the model can recognize the beginning and end of the sentence.

[0030] Step S23: Word embedding generation: the input tokens are mapped to the embedding layer to generate the corresponding word vectors, including: Token Embeddings: representing the semantic information of each token. Position Embeddings: representing the position of the token in the sentence. Segment Embeddings (if there are multiple sentences): distinguishing the relationship between sentences.

[0031] Step S24, BERT encoding, uses the Transformer encoder of the BERT model to process the tokens of the sentence and generate contextualized embeddings for each token.

[0032] Step S25: Extract Sentence Embeddings. Usually, the vector corresponding to the [CLS] tag is extracted from the BERT output as the representative vector of the entire sentence. The vectors of all tokens are pooled (such as average pooling) to generate Sentence Embeddings of fixed length.

[0033] The best way is to get the optimal hyperparameter K of KMeans:

[0034] Step S31: define a possible range of cluster numbers (e.g., from 1 to a maximum value K). max ), K max Usually chosen based on the size and characteristics of the data;

[0035] Step S32, training the model and calculating the clustering performance index;

[0036] Step S33, draw a "number of clusters vs. SSE" graph, with K as the horizontal axis and SSE as the vertical axis, and draw a line graph. As K increases, SSE will gradually decrease because more clusters can better fit the data.

[0037] Step S34, identify the "elbow" and observe the "elbow point" in the figure, that is, the point where the SSE reduction rate slows down significantly. The K corresponding to this point is the optimal number of clusters, indicating that the marginal benefit of adding more clusters to reduce the error has become smaller.

[0038] Step S35: Select the best K, and use the K obtained by the elbow method as the best number of clusters for cluster analysis.

[0039] Preferably, when training the model and calculating the clustering performance index, for each candidate K, the following steps are performed:

[0040] Step 321, fitting a KMeans model using K clusters;

[0041] Step 322, calculating the total sum of squared errors (SSE) of clustering;

[0042] Step 323: Save the corresponding KKK and SSE.

[0043] Preferably, when using Sentence Embeddings and hyperparameter K to build the final KMeans clustering model:

[0044] Step S36: Input the Sentence Embeddings generated in step S25 into the KMeans model. The model hyperparameter K is obtained in step S35 and set in the model. The random number seed is fixed to facilitate the reproduction of the results.

[0045] Preferably, when outputting the clustering results:

[0046] Step S41, predicting the classification outcome of the input data according to the KMeans model established in step S36, and outputting a table;

[0047] Step S42: Reduce the dimension of Sentence Embeddings and perform visualization operation based on the classification outcome predicted in step S41 to output a clustering effect diagram.

[0048] According to a second aspect of an embodiment of the present invention, a gene function classification device based on Sentence-BERT and KMeans model is provided, comprising a processor, a memory and a display, wherein the processor reads a computer program in the memory, and the result is displayed on the display, and is used to perform the following operations:

[0049] Obtaining the required input data, the input data is a gene function description text library;

[0050] Input the data into the receiving interface of the whole device;

[0051] The clustering results are generated and saved by the processor, and the Sentence Embeddings are reduced in dimension to output a clustering effect diagram which is displayed on the display.

[0052] Beneficial effects of the present invention

[0053] The gene function classification method and device based on Sentence-BERT and KMeans model provided by the present invention performs gene function classification by using gene function information summarized by previous studies in multiple databases, which can be used in other studies. This model can classify genes more accurately and has potential application value in fields such as functional enrichment analysis. Through this clustering method, genes can be classified more accurately through gene summary information. Researchers can use the classification results of this tool to optimize enrichment analysis and use gene features to optimize the modeling process. For example, when using genomic data for modeling, the model can be input after classification according to the classification results, so that each category carries certain functional classification information, enhancing the learning ability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 A schematic diagram of a flow chart of a gene function classification method based on Sentence-BERT and KMeans models provided in an embodiment of the present invention.

[0055] Figure 2 The operation flow chart of the example provided for the embodiment of the present invention is Figure 1 Additional details include the data flow of the entire instance.

[0056] Figure 3 A schematic diagram of the structure of a gene function classification device based on Sentence-BERT and KMeans model provided in an embodiment of the present invention.

[0057] Figure 4 Additional custom stop words provided for embodiments of the present invention.

[0058] Figure 5 The elbow method (A, C) and clustering effect (B, D) of KMeans parameter selection provided in the embodiment of the present invention. DETAILED DESCRIPTION

[0059] The technical content of the present invention is described in detail below with reference to the accompanying drawings and specific examples. This description performs calculations for two examples, and the specific results are shown in Figure 5 -B and Figure 5 -D.

[0060] like Figure 1 As shown, the gene function classification method based on Sentence-BERT and KMeans model provided in the embodiment of the present invention includes the following steps:

[0061] Step S1, obtaining the input data required by the model, the input data being a gene function description text library;

[0062] Step S2: Input the gene summary text into the Sentence-BERT model to generate vectorized SentenceEmbeddings;

[0063] Step S3: Use Sentence Embeddings to build a KMeans machine learning model and pre-train it to obtain the best hyperparameter K; use Sentence Embeddings and hyperparameter K to build the final KMeans clustering model;

[0064] Step S4: Output the clustering results, perform dimensionality reduction on the Sentence Embeddings and output a clustering effect diagram.

[0065] Establish a gene function description text library, such as Figure 2 As shown, the following steps are included:

[0066] Step S11, gene summary text from NCBI (https: / / www.ncbi.nlm.nih.gov / ), GeneCards (https: / / www.genecards.org / ) and UniProtKB / Swiss-Prot (https: / / www.uniprot.org / ) databases;

[0067] Step S12: After obtaining the gene summary text, organize it into a dataframe and use Python and other related libraries to clean the data.

[0068] like Figure 2 As shown, when obtaining data, the source of the selected database is:

[0069] Step 111, the examples cited in the present invention are based on the above three databases. These three databases are highly authoritative and all descriptions are non-repetitive, so they are used to establish a high-quality knowledge base for training.

[0070] After the data is obtained, it needs to be cleaned to obtain the data for input model. The steps are as follows:

[0071] Step 121, converting text type data into string format;

[0072] Step 122, delete meaningless sentences, which do not describe gene functions and may misjudge the clustering effect;

[0073] Step 123, delete the corresponding gene names. The gene names are all non-repeating topics and are meaningless to the functional description of the genes. They are deleted to optimize the classification effect.

[0074] Step 124, convert all text into lowercase to facilitate subsequent processing;

[0075] Step 125, deleting common stop words in text processing. The common stop words used in the present invention are ENGLISH_STOP_WORDS in the Python library SKLEARN.

[0076] Step 126, deleting scattered single letters (26 lowercase letters) in the text description paragraph, which will cause ambiguity in the feature description;

[0077] Step 127: Delete extra stop words. The present invention considers that some uncommon stop words are meaningless in gene classification and provides references here (e.g. Figure 4 shown);

[0078] Step 128: Delete unnecessary spaces, punctuation marks, line breaks, etc.

[0079] To convert unstructured text data into structured vector data and generate vectorized SentenceEmbeddings, the following steps are required:

[0080] Step S21: Input one or more sentences, segment them and convert them into tokens.

[0081] Step S22: Add special tags. Add special tags [CLS] (beginning of sentence) and [SEP] (end of sentence) to each sentence so that the model can recognize the beginning and end of the sentence. Figure 2 shown.

[0082] Step S23: Word embedding generation: the input tokens are mapped to the embedding layer to generate the corresponding word vectors, including: Token Embeddings: representing the semantic information of each token. Position Embeddings: representing the position of the token in the sentence. Segment Embeddings (if there are multiple sentences): distinguishing the relationship between sentences.

[0083] Step S24, BERT encoding, uses the Transformer encoder of the BERT model to process the tokens of the sentence and generate contextualized embeddings for each token, such as Figure 2 shown.

[0084] Step S25: Extract Sentence Embeddings. Usually, the vector corresponding to the [CLS] tag is extracted from the BERT output as the representative vector of the entire sentence. The vectors of all tokens are pooled (such as average pooling) to generate Sentence Embeddings of fixed length, such as Figure 2 shown.

[0085] KMeans clustering can be performed based on Sentence Embeddings vectors. To obtain the optimal number of clusters for clustering analysis, the following steps are included:

[0086] Step S31: define a possible range of cluster numbers (e.g., from 1 to a maximum value K). max ), K max Usually chosen based on the size and characteristics of the data;

[0087] Step S32, training the model and calculating the clustering performance index;

[0088] like Figure 2As shown, where KMeans optimal hyperparameter K, when training the model and calculating the clustering performance index, for each candidate K, the following steps are performed:

[0089] Step 321, fitting a KMeans model using K clusters;

[0090] Step 322, calculating the total sum of squared errors (SSE) of clustering;

[0091] Step 323: Save the corresponding KKK and SSE.

[0092] Step S33, draw a "number of clusters vs SSE" graph, such as Figure 5 -A, 5-C, with K as the horizontal axis and SSE as the vertical axis, a line graph is drawn. As K increases, SSE will gradually decrease because more clusters can better fit the data.

[0093] Step S34, identify the "elbow" and observe the "elbow point" in the figure, that is, the point where the SSE reduction rate slows down significantly. The K corresponding to this point is the optimal number of clusters, indicating that the marginal benefit of adding more clusters to reduce the error has become smaller.

[0094] Step S35: Select the best K, and use the K obtained by the elbow method as the best number of clusters for cluster analysis.

[0095] When using Sentence Embeddings and hyperparameter K to build the final KMeans clustering model, the following steps are performed:

[0096] Step S36: Input the Sentence Embeddings generated in step S25 into the KMeans model. The model hyperparameter K is obtained in step S35 and set in the model. The random number seed is fixed to facilitate the reproduction of the results.

[0097] When outputting clustering results, perform the following steps:

[0098] Step S41: According to the KMeans model established in step S36, the classification outcome of the input data is predicted and a table is output. The specific result format is shown in Figure 2 (Results are shown);

[0099] Step S42: Reduce the dimension of Sentence Embeddings and perform visualization based on the classification results predicted in step S41 to output a clustering effect diagram, such as Figure 5 -B (corresponding to 5-A) and Figure 5-D (corresponding to 5-C), this model accurately classifies genes through gene summary information. Researchers can use the classification results of this tool to optimize enrichment analysis and modeling processes. For example, when using genomic data for modeling, you can classify according to the classification results and input them into the model, so that each category carries certain functional classification information, enhancing the learning ability of the model.

[0100] In order to implement the gene function classification method based on Sentence-BERT and KMeans model provided by the present invention, the present invention also provides a gene function classification device based on Sentence-BERT and KMeans model, such as Figure 3 As shown, it includes a memory 31, a processor 32, and a display 33. The processor 32 reads the computer program in the memory 31, and the result is displayed on the display 33. The processor 32 is used to execute a gene function classification method based on Sentence-BERT and KMeans models proposed in this application.

[0101] The specific embodiments described herein are merely examples of the spirit of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in similar ways, but they will not deviate from the spirit of the present invention or exceed the scope defined by the appended claims.

Claims

1. A gene function classification method based on Sentence-BERT and KMeans model, characterized in that The steps include: Step S1, establishing a gene function description text library, wherein the gene function description text library includes gene summary text; Step S2: Input the gene summary text into the Sentence-BERT model to generate vectorized SentenceEmbeddings; Step S3: Use Sentence Embeddings to build a KMeans clustering model and pre-train it to obtain the best hyperparameter K; Use Sentence Embeddings and the best hyperparameter K to build the final KMeans clustering model; Step S4: Output the clustering results, perform dimensionality reduction on the Sentence Embeddings and output a clustering effect diagram.

2. The gene function classification method according to claim 1, characterized in that In step S1, a gene function description text library is established, including the following steps: Step S11, obtaining gene summary text from NCBI, GeneCards and UniProtKB / Swiss-Prot databases; Step S12: Obtain the gene summary text and organize it into a dataframe for data cleaning.

3. The gene function classification method according to claim 2, characterized in that In step S12, data cleaning specifically includes the following operations: Step 121, converting text type data into string format; Step 122, delete meaningless sentences, which do not describe gene functions and may misjudge the clustering effect; Step 123, delete the corresponding gene names. The gene names are all non-repeating topics and are meaningless to the functional description of the genes. They are deleted to optimize the classification effect. Step 124, convert all text into lowercase to facilitate subsequent processing; Step 125, deleting the commonly used stop words "stop words" in text processing; Step 126, deleting scattered single letters in the text description paragraph, which may cause ambiguity in the feature description; Step 127, delete additional stop words; Step 128: Delete unnecessary symbols, including spaces, punctuation marks, and line breaks.

4. The gene function classification method according to claim 1, characterized in that In step S2, the Sentence-BERT model generates vectorized Sentence Embeddings, including the following steps: Step S21, input one or more sentences, segment them and convert them into tokens, wherein the sentences are gene summary texts; Step S22: Add special tags. A special tag [CLS] is added at the beginning of each sentence, and a special tag [SEP] is added at the end of each sentence so that the model can recognize the beginning and end of the sentence. Step S23: Word embedding generation: the input tokens are mapped to the embedding layer to generate the corresponding word vectors, including: Token Embeddings: represents the semantic information of each token; Position Embeddings: indicates the position of the token in the sentence; Segment Embeddings: used for multiple sentences to distinguish the relationship between sentences; Step S24, BERT encoding, using the Transformer encoder of the BERT model to process the tokens of the sentence and generate context-sensitive embeddings for each token; Step S25: Extract Sentence Embeddings. Extract the vector corresponding to the [CLS] tag from the BERT output as the representative vector of the entire sentence. Pool the vectors of all tokens to generate Sentence Embeddings of fixed length.

5. The gene function classification method according to claim 1, characterized in that In step S3, the optimal hyperparameter K of KMeans is obtained, including: Step S31: define the possible range of cluster numbers, K max Choose based on the size and characteristics of the data; Step S32, training the model and calculating the clustering performance index; Step S33, draw a "number of clusters vs. SSE" graph, with K as the horizontal axis and SSE as the vertical axis, and draw a line graph; Step S34, identify the "elbow" of the line graph, and observe the "elbow point" in the graph, that is, the point where the SSE reduction rate slows down significantly. The K corresponding to this point is the optimal number of clusters, indicating that the marginal benefit of adding more clusters to reduce the error has become smaller; Step S35: The optimal number of clusters K obtained by the elbow method is used as the optimal hyperparameter K of KMeans.

6. The gene function classification method according to claim 5, characterized in that In step S32, when training the model and calculating the clustering performance index, for each candidate K, the following steps are performed respectively: Step 321, fitting a KMeans model using K clusters; Step 322, calculating the total sum of square errors SSE of clustering; Step 323: Save the corresponding K and SSE.

7. The gene function classification method according to claim 5, characterized in that Use Sentence Embeddings and the optimal hyperparameter K to build the final KMeans clustering model, including: Step S36: Input the Sentence Embeddings generated in step S25 into the KMeans model, and use the result obtained in step S35 to set the optimal hyperparameter number K in the KMeans model, and fix the random number seed to facilitate the reproduction of the result.

8. The gene function classification method according to claim 1, characterized in that In step S4, outputting the clustering results includes the following steps: Step S41, predicting the classification outcome of the input data according to the KMeans clustering model established in step S3, and outputting a table; Step S42: Reduce the dimension of Sentence Embeddings and perform visualization operation based on the classification outcome predicted in step S41 to output a clustering effect diagram.

9. A gene function classification device based on Sentence-BERT and KMeans model, characterized in that The invention comprises a processor, a memory and a display, wherein the processor reads a computer program in the memory and displays the result on the display, and is used to execute a gene function classification method based on Sentence-BERT and KMeans model as described in any one of claims 1 to 8.

Images