A method for extracting yeast transcription factor-gene relationship in biological text

Abstract

This invention discloses a method for extracting yeast transcription factor-gene relationships from biological text. The method includes: retrieving literature related to yeast transcription factor-gene relationship extraction; constructing an input representation; constructing a relationship extraction model; inputting the input representation into the trained relationship extraction model to obtain a pre-selected output representation containing transcription factor-gene binding or regulatory information; inputting the output representation into the trained relationship extraction model to obtain a first probability; inputting the output representation into a syntactic analyzer and a semantic analyzer respectively, and then into a graph neural network to obtain a second probability; setting a loss function for secondary training; inputting real-time collected literature into the model each time; and finally obtaining a joint probability based on the first and second probabilities. If the joint probability is greater than a preset value, the output representation is considered to contain transcription factor-gene binding or regulatory information, and the corresponding output representation is output. The advantage of this invention is that the relationship extraction is more accurate.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and more specifically to a method for extracting yeast transcription factor-gene relationships from biological text. Background Technology

[0002] With the deepening and expansion of research in the biomedical field, the number of related documents is growing exponentially. This massive amount of unstructured biomedical text information contains rich knowledge and value, which is of great significance to researchers and medical personnel in disease diagnosis, treatment, and drug development. However, the large volume of unstructured text information also presents a significant challenge to medical personnel and researchers in finding and mining the information they need, making the information acquisition process cumbersome and time-consuming. Therefore, effectively processing biomedical text information and improving information mining efficiency is crucial for promoting the development of the biomedical field. Relation extraction and entity recognition play key roles in processing this text data. Relation extraction helps us discover the connections between entities in the text, while entity recognition helps detect key biological entities in the text. However, existing relation extraction and entity recognition methods often suffer from problems such as insufficient preprocessing, limited extraction accuracy, and high computational complexity when processing unstructured biomedical text.

[0003] Research on yeast transcription factor-gene associations in the biomedical field is of great significance to biological and medical research. Studying yeast transcription factor-gene associations in a vast amount of unstructured biomedical text information helps to gain a deeper understanding of gene expression regulation mechanisms, thus providing important insights into gene regulation in complex organisms. Therefore, the extraction of yeast transcription factor-gene relationships has become a new research direction and hot topic in the industry.

[0004] Current relation extraction techniques still have many shortcomings in solving the problem of information extraction from yeast transcription factor-genes. For example, before neural information extraction models, rule-based entity recognition and relation extraction frameworks were widely used. These frameworks focused on mining rules based on hard templates or soft features from text and matching them with instances. Rule formulation requires domain expert knowledge or automatic discovery from large corpora, resulting in cumbersome data processing and incomplete rule coverage. In recent years, end-to-end neural network methods have made significant progress in information extraction. To utilize these rules, many works have begun to combine traditional rule-based methods with neural matching modules. However, these methods still require manually formulating seed rules or labeling seed relation triggers and iteratively expanding them, making the overall relation extraction method relatively cumbersome.

[0005] Entity recognition and relation extraction are considered mutually beneficial tasks. However, existing relation extraction methods do not fully leverage the interaction between entity recognition and relation extraction. For example, Chinese Patent Publication No. CN114282539A discloses a named entity recognition method based on a pre-trained model in the biomedical field, which only involves entity recognition and does not involve relation extraction. It fails to fully utilize the dependency between the two tasks, resulting in inaccurate relation extraction results. Furthermore, existing relation extraction methods do not differentiate between the syntactic and semantic importance of words in sentences. In practical applications, some words have important syntactic functions but contribute little to the semantics of the sentence, such as prepositions and conjunctions. Conversely, some words, such as nouns and lexical verbs, contribute significantly to the semantics. When encoding context, most methods are too simplistic, failing to inject syntactic features into word vectors. They ignore the differences in semantic and syntactic importance between words, considering only the influence of semantics or syntax on relation extraction without simultaneously considering the overall impact of both, leading to inaccurate relation extraction results. Summary of the Invention

[0006] The technical problem to be solved by this invention is that the existing yeast transcription factor-gene relationship extraction method does not extract accurate results, which is not conducive to researchers’ in-depth understanding of gene expression regulation mechanisms.

[0007] This invention solves the above-mentioned technical problems through the following technical means: a method for extracting yeast transcription factor-gene relationships from biological text, the method comprising:

[0008] Step a: Retrieve relevant literature on yeast transcription-gene relationships from biomedical text databases;

[0009] Step b: Preprocess the literature and construct the input representation;

[0010] Step c: Construct a relation extraction model, train the model, and obtain a trained relation extraction model;

[0011] Step d: Input the pre-trained relation extraction model into the input representation to obtain a preliminary filtered output representation containing transcription factor-gene binding or regulatory information;

[0012] Step e: Input the output representation into the trained relation extraction model for classification to obtain the first probability of whether the output representation contains transcription factor-gene binding or regulatory information. Input the output representation into the syntactic analyzer and semantic analyzer respectively, and then into the graph neural network. After vector fusion and normalization, obtain the second probability of whether the output representation contains transcription factor-gene binding or regulatory information. Set the loss function for secondary training. After training, input the real-time collected literature into the model each time. Finally, obtain the joint probability based on the first probability and the second probability. If the joint probability is greater than the preset value, it is considered that the output representation contains transcription factor-gene binding or regulatory information, and the corresponding output representation is output.

[0013] Beneficial effects: This invention retrieves literature related to the research topic from a biomedical text database to achieve entity recognition. After preprocessing the retrieved literature (i.e., the entity recognition results), it inputs the retrieved literature into a constructed relation extraction model to achieve relation extraction. It fully utilizes the dependency between entity recognition and relation extraction tasks to improve the accuracy of relation extraction. Secondly, after the relation extraction model outputs a preliminary screening of the output representation containing transcription factor-gene binding or regulatory information, it uses structures such as syntactic analyzers, semantic analyzers, and graph neural networks to consider the overall impact of syntax and semantics on relation extraction. It integrates syntactic and semantic information, enabling the model to better understand sentence semantics and improve the accuracy of relation extraction. In addition, through two training sessions, the accuracy of relation extraction is further improved.

[0014] Further, step b includes:

[0015] Step b1: Segment the document into words and build a corresponding vocabulary list V, thereby representing the document as a sequence of words;

[0016] Step b2: Convert the segmented documents into a vocabulary index to obtain text vectors;

[0017] Step b3: Combine the text vector, mask, and segments into an input vector; where the mask represents a valid or invalid binary vector of data, and the segments are vectors that mark which segment of the original text a word belongs to;

[0018] Step b4: Concatenate the input vector with the additional feature representation of yeast transcription factor-gene to obtain the input representation X_extended.

[0019] Furthermore, the performance evaluation method for the vocabulary list V is as follows:

[0020] Through formula Construct an objective function, and evaluate the performance of the vocabulary V by the value of the objective function. The smaller the value of the objective function, the better the performance of the vocabulary V. Here, n represents the number of words participating in the performance evaluation, and p(w i p(w) represents the probability that word w appears in the training text. i |V) represents the probability that a word wi is represented as a sequence of words given a vocabulary V.

[0021] Furthermore, step c includes:

[0022] The relation extraction model includes a self-attention computation layer, a multi-head attention layer, a first linear layer, an activation layer, a second linear layer, a normalization layer, a fully connected layer, and a Softmax layer. The input representation of the document is input to the self-attention computation layer, and the output of the self-attention computation layer is input to the multi-head attention layer. The outputs of the self-attention computation layer and the multi-head attention layer are weighted and fused to obtain the self-attention output. The self-attention output is input to the first linear layer and then sequentially passes through the activation layer, the second linear layer, the normalization layer, the fully connected layer, and the Softmax layer to output the final output representation.

[0023] Furthermore, the calculation process of the self-attention calculation layer is as follows:

[0024] Q = X_extended * W_Q

[0025] K = X_extended * W_K

[0026] v = X_extended * W_V

[0027] A = Q * K T / sqrt(hidden_size)

[0028] W_A = softmax(A)

[0029] Y_SA=W_A*v

[0030] Where Q is the query matrix of the self-attention computation layer, K is the key matrix of the self-attention computation layer, v is the value matrix of the self-attention computation layer, and K T W_Q represents the transpose of the key matrix of the self-attention computation layer, W_K and W_V are the weight matrices of the query matrix, key matrix, and value matrix of the self-attention computation layer, respectively, A is the attention score matrix of the self-attention computation layer, hidden_size is the dimension of the input representation, W_A is the normalized attention score matrix of the self-attention computation layer, and Y_SA is the output result of the self-attention computation layer.

[0031] Furthermore, the calculation process of the multi-head attention layer is as follows:

[0032] Q′=Y_SA*W_Q′

[0033] K′=Y_SA*W_K′

[0034] v′=Y_SA*W_V′

[0035] A′=Q′*K′ T / sqrt(hidden_size)

[0036] W_A′=Softmax(A′)

[0037] Y_SA′=W_A′*v′

[0038] Where Q′ is the query matrix of the multi-head attention layer, K′ is the key matrix of the multi-head attention layer, v′ is the value matrix of the multi-head attention layer, and K′ T Y_Q′ represents the transpose of the key matrix of the multi-head attention layer, W_Q′, W_K′, and W_V′ are the weight matrices of the query matrix, key matrix, and value matrix of the multi-head attention layer, respectively, A′ is the attention score matrix of the multi-head attention layer, W_A′ is the normalized attention score matrix of the multi-head attention layer, and Y_SA′ is the output result of the multi-head attention layer.

[0039] Furthermore, the processing steps for the first linear layer, activation layer, second linear layer, normalization layer, fully connected layer, and Softmax layer are as follows:

[0040] The output of the first linear layer is FFN_1 = Y_SA * W_1 + b_1, where W_1 represents the weight matrix of the first linear layer and b_1 represents the bias vector of the first linear layer.

[0041] The output of the activation layer is ReLU_FFN_1 = ReLU(FFN_1), where ReLU represents the activation layer;

[0042] The output of the second linear layer is FFN_2 = ReLU_FFN_1 * W_2 + b_2, where W_2 represents the weight matrix of the second linear layer and b_2 represents the bias vector of the second linear layer.

[0043] The output of the normalization layer is Y_FFN = LayerNorm(Y_SA + FFN_2), where LayerNorm represents the normalization operation;

[0044] The output of the fully connected layer is Y_FC = Y_FFN*W_FC + b_FC, where W_FC represents the weight matrix of the fully connected layer and b_FC represents the bias vector of the fully connected layer.

[0045] The calculation formula for the Softmax layer is P_i=exp(Y_FC_i) / ∑_j exp(Y_FC_j), where Y_FC_i is the vector of the i-th sample output by the fully connected layer, P_i is the probability distribution predicted by the i-th sample, and the i-th sample refers to the description of the i-th sentence in the literature.

[0046] Furthermore, the training process of the relation extraction model is as follows:

[0047] The weighted cross-entropy loss function L = -sum(w_i*y_i*log(p_i)) is constructed using the formula, where w_i is the weight of the i-th sample and y_i is the true transcription factor-gene pair label.

[0048] The parameters of each layer in the relation extraction model are continuously adjusted to train the model. The weighted cross-entropy loss function is calculated until the value of the weighted cross-entropy loss function is minimized, at which point training stops and a well-trained relation extraction model is obtained.

[0049] Further, step e includes:

[0050] Through formula Obtain the classifier label vector. The classifier label vector is obtained by formula p1=softmax(FW) to obtain the first probability of whether the output representation contains transcription factor-gene binding or regulatory information. Here, FrozenBioBERT represents the relation extraction model, S is the output representation of the relation extraction model, W is the training weight matrix, f1 is the function formed by the last five layers of the relation extraction model, and ° represents the function combination.

[0051] Furthermore, step e also includes:

[0052] The output is input into the syntactic analyzer and semantic analyzer respectively to extract the syntactic structure and semantic dependencies of the sentence, generating a syntactic-semantic graph, where nodes represent words and edges represent syntactic and semantic relations in the sentence. The graph neural network uses the formula G = GNN(S) syntax ,S semantic The syntactic semantic graph is encoded into a syntactic semantic graph representation vector. This vector is then fused with the classifier label vector using the formula H = concat(F,G). The fused representation is input into a fully connected layer and a softmax layer, and then processed using the formula p2 = softmax(W). h H+b h The second probability is obtained to represent whether the output representation contains transcription factor-gene binding or regulatory information, where GNN represents a graph neural network, and S... syntax S represents the syntactic structure of a sentence.semantic This indicates the semantic dependencies between sentences; `concat` represents the concatenation and merging operation; W h b represents the weight term. h This indicates the bias term.

[0053] Furthermore, step e also includes:

[0054] Set the loss function Among them, L i Let represent the weighted cross-entropy loss function value of the i-th sample, and N be the total number of sentences describing it in the literature. α is the number of sentence descriptions related to transcription factor-gene pair j in the same article for the i-th sample. i The weighting coefficients and β is the first hyperparameter;

[0055] The parameters of the syntactic parser, semantic parser, and graph neural network are continuously adjusted for secondary training until the loss function L′ is minimized, at which point training stops. After training, real-time collected literature is input into the model each time, and finally, the result is calculated according to formula p. comb =α*p1+(1-α)*p2 to obtain the joint probability, where α is the second hyperparameter. If the joint probability is greater than the preset value, it is considered that the output representation contains transcription factor-gene binding or regulatory information, and the corresponding output representation is output.

[0056] The advantages of this invention are:

[0057] (1) This invention retrieves literature related to the research topic from a biomedical text database to achieve entity recognition. After preprocessing the retrieved literature, i.e. the results of entity recognition, it is input into the constructed relation extraction model to achieve relation extraction. It makes full use of the dependency between entity recognition and relation extraction to improve the accuracy of relation extraction. Secondly, after the relation extraction model outputs the output representation containing transcription factor-gene binding or regulatory information that has been initially screened, it uses structures such as syntactic analyzer, semantic analyzer, and graph neural network to consider the overall influence of syntax and semantics on relation extraction. It integrates syntactic and semantic information so that the model can better understand the semantics of sentences and improve the accuracy of relation extraction. In addition, the accuracy of relation extraction is further improved through two training sessions.

[0058] (2) Compared to the traditional BERT model, the Fine-tuned BioBERT model of this invention has been pre-trained specifically in the biomedical field, enabling it to better understand domain-specific terminology and semantic relationships. It exhibits higher accuracy in relation extraction tasks, thereby improving the effectiveness of yeast transcription factor-gene relation extraction.

[0059] (3) This invention employs a trigger-based memory flow framework, which provides more effective key information capture. It can capture key information in the literature based on the appearance of trigger words, thereby extracting relationships more effectively. Through targeted capture of key information, the trigger-based memory flow framework improves the accuracy of relationship extraction, avoiding missed or false detections. By combining the Fine-tuned BioBERT model, the trigger-based memory flow framework, and syntactic-semantic graph fusion techniques, a highly efficient and accurate yeast transcription factor-gene pair relationship extraction process is achieved. This makes it easier to extend and adapt to other tasks, including relationship extraction tasks in other biomedical fields.

[0060] (4) In this invention, a bilinear design (i.e., a first linear layer and a second linear layer) is employed. When describing the relationship between transcription factors and genes, it is usually necessary to consider their interactions. The bilinear layer allows the model to consider transcription factors and genes simultaneously in a shared space, which helps to capture the complex interactions between them, thereby improving the overall relationship extraction accuracy.

[0061] (5) This invention employs syntactic and semantic analysis design. In the yeast transcription factor-gene extraction task, the structural information of sentences is very important. A transcription factor may be associated with a gene through a specific verb, and the syntactic role of this verb may affect the meaning of this relationship. Through syntactic and semantic analysis, the model can obtain this important structural information.

[0062] (6) This invention employs self-attention and multi-head attention mechanisms. In the yeast transcription factor-gene extraction task, different words may have different effects on the extraction results. Some words may directly indicate the relationship between a transcription factor and a gene, while others may provide contextual information about this relationship. The self-attention mechanism allows the model to assign different attention to different words in a sentence, while multi-head attention allows the model to capture different information in the sentence across different attention heads. This helps to fully understand how transcription factors and genes are associated in a sentence. Attached Figure Description

[0063] Figure 1 This is a flowchart of a method for extracting yeast transcription factor-gene relationships from biological text, as disclosed in an embodiment of the present invention. Detailed Implementation

[0064] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0065] like Figure 1 As shown, this invention provides a method for extracting yeast transcription factor-gene relationships from biological text, the method comprising:

[0066] Step 1: Use keywords to search for relevant literature on yeast transcription-gene relationships in biomedical text databases (such as PubMed, Web of Science, etc.) and initially screen out literature that meets the requirements; Figure 1 In this context, TF-gene represents yeast transcription factor-gene.

[0067] Step 2: Preprocess the literature to construct the input representation; the specific process is as follows:

[0068] Step 2.1: Use WordPiece to segment the document and build a corresponding vocabulary V, thus representing the document as a sequence of words; define an objective function to evaluate the performance of the vocabulary V, specifically through the formula... Construct an objective function, and evaluate the performance of the vocabulary V by the value of the objective function. The smaller the value of the objective function, the better the performance of the vocabulary V. Here, n represents the number of words participating in the performance evaluation, and p(w i p(w) represents the probability that word w appears in the training text. i |V) represents the probability that a word wi is represented as a sequence of words given a vocabulary V.

[0069] A vocabulary V is a list containing all possible words (words or phrases) used to map words in the raw text to unique indices or codes. This enables computers to process and analyze text data using these codes, achieving the goal of automated extraction of yeast transcription factor-gene associations. In this process, the objective function L(V) measures the performance of the vocabulary V in representing text data. The smaller the value of the objective function, the less uncertainty the vocabulary V introduces in representing text, meaning it is easier for computers to understand and process the text data. Therefore, to obtain a high-performance vocabulary V, the value of the objective function L(V) needs to be minimized.

[0070] Step 2.2: Convert the segmented documents into a vocabulary index to obtain text vectors; thus, each word or subword has a unique ID, corresponding to its position in the vocabulary.

[0071] Step 2.3: Combine the text vector, mask, and segments into an input vector X; where the mask represents a valid or invalid binary vector of data, and the segments are vectors that mark which segment of the original text a word belongs to;

[0072] Step 2.4: The input vector X is concatenated with the additional feature representations of yeast transcription factors and genes, X_features, using the formula X_extended = concat(X, X_features), to obtain the input representation X_extended. X_features are the additional feature representations of transcription factors and genes, such as sequence features and functional annotations, while concat represents the concatenation operation.

[0073] Step 3: Construct and train the relation extraction model to obtain the trained relation extraction model. The relation extraction model includes a self-attention calculation layer, a multi-head attention layer, a first linear layer, an activation layer, a second linear layer, a normalization layer, a fully connected layer, and a softmax layer. The input representation of the document is input to the self-attention calculation layer. The output of the self-attention calculation layer is input to the multi-head attention layer. The outputs of the self-attention calculation layer and the multi-head attention layer are weighted and fused to obtain the self-attention output. The self-attention output is input to the first linear layer, which then passes through the activation layer, the second linear layer, the normalization layer, the fully connected layer, and the softmax layer to output the final output representation. The specific process is as follows:

[0074] Step 3.1: Perform self-attention calculation on the input representation X_extended. The calculation process of the self-attention calculation layer is as follows:

[0075] Q = X_extended * W_Q

[0076] K = X_extended * W_K

[0077] v = X_extended * W_V

[0078] A = Q * K T / sqrt(hidden_size)

[0079] W_A = softmax(A)

[0080] Y_SA=W_A*v

[0081] Where Q is the query matrix of the self-attention computation layer, K is the key matrix of the self-attention computation layer, v is the value matrix of the self-attention computation layer, and KT W_Q represents the transpose of the key matrix of the self-attention computation layer, W_K and W_V are the weight matrices of the query matrix, key matrix, and value matrix of the self-attention computation layer, respectively, A is the attention score matrix of the self-attention computation layer, hidden_size is the dimension of the input representation, W_A is the normalized attention score matrix of the self-attention computation layer, and Y_SA is the output result of the self-attention computation layer.

[0082] Step 3.2: To capture transcription factor-gene relationships, an additional attention head can be added to the Fine-tuned BioBERT model. This additional attention head will be specifically used to capture the relationship between transcription factors and genes; this additional attention head is also known as a multi-head attention layer. The calculation process for the multi-head attention layer is as follows:

[0083] Q′=Y_SA*W_Q′

[0084] K′=Y_SA*W_K′

[0085] v′=Y_SA*W_V′

[0086] A′=Q′*K′ T / sqrt(hidden_size)

[0087] W_A′=Softmax(A′)

[0088] Y_SA′=W_A′*v′

[0089] Where Q′ is the query matrix of the multi-head attention layer, K′ is the key matrix of the multi-head attention layer, v′ is the value matrix of the multi-head attention layer, and K′ T Y_SA represents the transpose of the key matrix of the multi-head attention layer. W_Q′, W_K′, and W_V′ are the weight matrices of the query matrix, key matrix, and value matrix of the multi-head attention layer, respectively. A′ is the attention score matrix of the multi-head attention layer, W_A′ is the normalized attention score matrix of the multi-head attention layer, and Y_SA′ is the output of the multi-head attention layer. The weighted fusion of the outputs from the self-attention computation layer and the multi-head attention layer is used as the updated self-attention output, which will still be represented as Y_SA below.

[0090] Step 3.3: In the yeast transcription factor-gene pair extraction task, the position feedforward network (FFN) in the Fine-tuned BioBERT model can be considered as a way to capture more complex features in the input text. These features may be related to the relationship between yeast transcription factors and gene pairs. Specifically, the processing steps for the first linear layer, activation layer, second linear layer, normalization layer, fully connected layer, and softmax layer are as follows:

[0091] Starting with the output representation (Y_SA) of the self-attention layer, this representation captures contextual information from the text. The input dimension is (batch_size, seq_length, hidden_size). Y_SA is input to the first linear layer, and the output of the first linear layer is FFN_1 = Y_SA * W_1 + b_1, where W_1 represents the weight matrix of the first linear layer with dimension (hidden_size, ffn_hidden_size), and b_1 represents the bias vector of the first linear layer with dimension (ffn_hidden_size). The first linear layer is applied to map the input representation from the hidden_size dimension to the ffn_hidden_size dimension.

[0092] The ReLU activation layer is applied to introduce nonlinear characteristics. The output of the activation layer is ReLU_FFN_1 = ReLU(FFN_1), where ReLU represents the activation layer.

[0093] The output of the second linear layer is FFN_2 = ReLU_FFN_1 * W_2 + b_2, where W_2 represents the weight matrix of the second linear layer with dimensions (ffn_hidden_size, hidden_size), and b_2 represents the bias vector of the second linear layer with dimensions (hidden_size). Applying the second linear layer maps the output of the activation function from the ffn_hidden_size dimension back to the hidden_size dimension.

[0094] The output of the second linear layer is added to the original self-attention output, and then layer normalization is applied. The output of the normalized layer is Y_FFN = LayerNorm(Y_SA + FFN_2), where LayerNorm represents the normalization operation.

[0095] In this process, the position-feedforward network learns to extract higher-level features relevant to the yeast transcription factor-gene pair extraction task from the output of the self-attention layer. These features help the model more accurately identify and extract yeast transcription factor-gene pair relationships in downstream tasks.

[0096] Step 3.4: Pass the FFN output representation to the fully connected layer. The purpose of the fully connected layer is to map the representation at each position to a vector equal to the number of labels. The output of the fully connected layer is Y_FC = Y_FFN*W_FC + b_FC, where W_FC represents the weight matrix of the fully connected layer with dimensions (hidden_size, num_labels), and b_FC represents the bias vector of the fully connected layer with dimensions (num_labels).

[0097] The output of the fully connected layer is passed to the Softmax layer. The Softmax layer transforms the output vector at each position into a probability distribution, allowing us to select the predicted label based on the highest probability. The Softmax layer is calculated as P_i = exp(Y_FC_i) / ∑_j exp(Y_FC_j), where Y_FC_i is the vector of the i-th sample output from the fully connected layer, P_i is the probability distribution predicted for the i-th sample, and the i-th sample refers to the description of the i-th sentence in the literature. The output dimension of the Softmax layer is (batch_size, seq_length, num_labels). The output representation is obtained after processing by the fully connected layer and the Softmax layer.

[0098] Step 3.5: Design a weighted cross-entropy loss function to measure the degree of matching between the predicted results and the actual transcription factor-gene pairs. The training process of the relation extraction model is as follows:

[0099] The weighted cross-entropy loss function L = -sum(w_i*y_i*log(p_i)) is constructed using the formula, where w_i is the weight of the i-th sample and y_i is the true transcription factor-gene pair label.

[0100] The parameters of each layer in the relation extraction model are continuously adjusted to train the model. The weighted cross-entropy loss function is calculated until the value of the weighted cross-entropy loss function is minimized, at which point training stops and a well-trained relation extraction model is obtained.

[0101] Step 4: Input the trained relation extraction model into the input representation to obtain the output representation containing transcription factor-gene binding or regulatory information, which has been preliminarily screened.

[0102] Step 5: The output representation is input into the trained relation extraction model for classification to obtain the first probability representing whether the output representation contains transcription factor-gene binding or regulatory information. The output representation is then input into the syntactic analyzer and semantic analyzer, and finally into the graph neural network. After vector fusion and normalization, the second probability representing whether the output representation contains transcription factor-gene binding or regulatory information is obtained. A loss function is set for secondary training. After training, real-time collected literature is input into the model each time. Finally, a joint probability is obtained based on the first and second probabilities. If the joint probability is greater than a preset value, the output representation is considered to contain transcription factor-gene binding or regulatory information, and the corresponding output representation is output. The specific process is as follows:

[0103] Step 5.1: For a given relation extraction model's output representation S, first input it into the FrozenBioBERT model. The FrozenBioBERT model outputs the word fragment tensor for each sentence, which is obtained through the formula... Obtain the classifier label vector;

[0104] The classifier label vector is obtained by the formula p1 = softmax(FW) to obtain the first probability representing whether the output representation contains transcription factor-gene binding or regulatory information;

[0105] Where FrozenBioBERT represents the relation extraction model, S is the output representation of the relation extraction model, W represents the training weight matrix, f1 represents the function formed by the last five layers of the relation extraction model, and ° represents the function combination.

[0106] Step 5.2: Input the output representation into the syntactic analyzer and semantic analyzer respectively to extract the syntactic structure and semantic dependencies of the sentence, generating a syntactic-semantic graph, where nodes represent words and edges represent syntactic and semantic relations in the sentence. The graph neural network uses the formula G = GNN(S) syntax ,S semantic Encode the syntactic semantic graph into a syntactic semantic graph representation vector;

[0107] The syntactic semantic graph representation vector and the classifier label vector are fused using the formula H = concat(F,G);

[0108] The fused representation is input into a fully connected layer and a Softmax layer, using the formula p2 = softmax(W). h H+b h The second probability is obtained from the characterization output representation of whether it contains transcription factor-gene binding or regulatory information.

[0109] Where GNN represents Graph Neural Network, S syntaxS represents the syntactic structure of a sentence. semantic This indicates the semantic dependencies between sentences; `concat` represents the concatenation and merging operation; W h b represents the weight term. h This indicates the bias term.

[0110] It should be noted that the syntactic analyzer and semantic analyzer are not improvements of this invention, but are existing technologies. The syntactic analyzer and semantic analyzer described in the paper "QingrongXia, Zhenghua Li*, Min Zhang. A Syntax-aware Multi-task Learning Framework for Chinese Semantic Role Labeling" published by the Institute of Artificial Intelligence, School of Computer Science and Technology, Soochow University, China can be used.

[0111] Step 5.3: Set the loss function Among them, L i Let represent the weighted cross-entropy loss function value of the i-th sample, and N be the total number of sentences describing it in the literature. α is the number of sentence descriptions related to transcription factor-gene pair j in the same article for the i-th sample. i The weighting coefficients and β is the first hyperparameter; β is an adjustable hyperparameter used to control the degree of attention given to a minority of samples. When β > 0, the weighting coefficient α is used for transcription factor-gene pairs that occur less frequently. i This will increase the model's focus on these samples. By optimizing the loss function of this innovative design, the model can pay more attention to transcription factor-gene pairs that appear less frequently, while still focusing on all samples, which helps improve the model's performance when dealing with imbalanced data.

[0112] The parameters of the syntactic parser, semantic parser, and graph neural network are continuously adjusted for secondary training until the loss function L′ is minimized, at which point training stops. After training, real-time collected literature is input into the model each time, and finally, the result is calculated according to formula p. comb =α*p1+(1-α)*p2 to obtain the joint probability, where α is the second hyperparameter. If the joint probability is greater than the preset value, it is considered that the output representation contains transcription factor-gene binding or regulatory information, and the corresponding output representation is output.

[0113] Through the above technical solutions, this invention retrieves literature related to the research topic from a biomedical text database to achieve entity recognition. After preprocessing the retrieved literature, i.e., the entity recognition results, it inputs the retrieved literature into a constructed relation extraction model to achieve relation extraction. It fully utilizes the dependency between entity recognition and relation extraction to improve the accuracy of relation extraction. Secondly, after the relation extraction model outputs the output representation containing transcription factor-gene binding or regulatory information through preliminary screening, it uses structures such as syntactic analyzers, semantic analyzers, and graph neural networks to consider the overall impact of syntax and semantics on relation extraction. It integrates syntactic and semantic information, enabling the model to better understand sentence semantics and improve the accuracy of relation extraction. In addition, through two training sessions, the accuracy of relation extraction is further improved.

[0114] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for extracting yeast transcription factor-gene relationship in biological text, characterized in that, The method comprises: Step a: retrieving literatures related to yeast transcription-gene relationship extraction from a biomedical text database; Step b: preprocessing the literatures to construct input representations; Step c: constructing a relation extraction model, training the model, and obtaining the trained relation extraction model; the relation extraction model comprises a self-attention calculation layer, a multi-head attention layer, a first linear layer, an activation layer, a second linear layer, a normalization layer, a full connection layer, and an output layer; the input representation of the literature is input into the self-attention calculation layer, the output result of the self-attention calculation layer is input into the multi-head attention layer, the output results of the self-attention calculation layer and the multi-head attention layer are weighted and fused to serve as a self-attention output, the self-attention output is input into the first linear layer, and the first linear layer is sequentially connected with the activation layer, the second linear layer, the normalization layer, the full connection layer, and the output layer to output a final output representation; Step d: inputting the input representations into the trained relationship extraction model to obtain output representations containing preliminary screened transcription factor-gene binding or regulation information; Step e: inputting the output representations into the trained relationship extraction model to obtain a first probability representing whether the output representations contain transcription factor-gene binding or regulation information, inputting the output representations into a syntax analyzer and a semantic analyzer and then into a graph neural network, and then performing vector fusion and normalization operations to obtain a second probability representing whether the output representations contain transcription factor-gene binding or regulation information, setting a loss function for secondary training, inputting real-time collected literatures into the model after the training is completed, and finally obtaining a joint probability according to the first probability and the second probability, regarding the output representations as containing transcription factor-gene binding or regulation information when the joint probability is greater than a preset value, and outputting the corresponding output representations.

2. The method for extracting yeast transcription factor-gene relationships from biological text according to claim 1, characterized in that, The step b comprises: Step bl, tokenizing the documents and establishing a corresponding word vocabulary such that the documents are represented as sequences of words; Step b2: converting the segmented literatures into a word table index to obtain a text vector; Step b3: grouping the text vector, a mask, and a segment into an input vector; wherein the mask is a binary vector representing the validity or invalidity of data, and the segment is a vector indicating which segment of the original text the word belongs to; Step b4, splicing the input vector with the additional features representation of the yeast transcription factor-gene, to obtain the input representation .

3. The method for extracting yeast transcription factor-gene relationships from biological text according to claim 1, characterized in that, The calculation process of the self-attention calculation layer is as follows: wherein is a query matrix of the self-attention computation layer, is a key matrix of the self-attention computation layer, is a value matrix of the self-attention computation layer, denotes the transpose of the key matrix of the self-attention computation layer, , , are weight matrices of the query matrix of the self-attention computation layer, the key matrix of the self-attention computation layer, and the value matrix of the self-attention computation layer, respectively, is an attention score matrix of the self-attention computation layer, is a dimension of the input representation, is a normalized attention score matrix of the self-attention computation layer, is an output result of the self-attention computation layer.

4. The method of claim 3, wherein the method further comprises: determining the relationship between the yeast transcription factor and the gene based on the relationship between the yeast transcription factor and the gene in the biological text. The calculation process of the multi-head attention layer is as follows: wherein is a query matrix of the multi-headed attention layer, is a key matrix of the multi-headed attention layer, is a value matrix of the multi-headed attention layer, denotes the transpose of the key matrix of the multi-headed attention layer, , , are weight matrices of the query matrix of the multi-headed attention layer, the key matrix of the multi-headed attention layer, and the value matrix of the multi-headed attention layer, respectively, is an attention score matrix of the multi-headed attention layer, is a normalized attention score matrix of the multi-headed attention layer, is an output result of the multi-headed attention layer.

5. The method for extracting yeast transcription factor-gene relationships from biological text according to claim 4, characterized in that, the first linear layer, the activation layer, the second linear layer, the normalization layer, the fully connected layer, and The processing procedure of the layers is as follows: The output of the first linear layer is wherein, denotes the weight matrix of the first linear layer, denotes the bias vector of the first linear layer; The output of the activation layer is wherein, denotes the activation layer; The output of the second linear layer is wherein, denotes the weight matrix of the second linear layer, denotes the bias vector of the second linear layer; The output of the normalization layer is wherein, denotes a normalization operation; The output of the fully connected layer is wherein, denotes the weight matrix of the fully connected layer, denotes the bias vector of the fully connected layer; The formula for calculating the layer is: ,in, It is the first output of the fully connected layer. A vector of samples, It is the first The probability distribution predicted for the nth sample, the th The sample refers to the first sample in the literature. Describe in one sentence.

6. The method of claim 5, wherein the method further comprises: determining the relationship between the yeast transcription factor and the gene based on the extracted information. The training process of the relationship extraction model is as follows: By formula construction Weighted cross-entropy loss function, where, is the weight of the th sample, is the true transcription factor-gene pair label; Continuously adjust the parameters of each layer in the relationship extraction model, train the model, calculate the weighted cross-entropy loss function, and stop training when the value of the weighted cross-entropy loss function is the smallest, to obtain the trained relationship extraction model.

7. The method of claim 1, wherein the method is characterized by, The step e comprises: The classifier label vector is obtained by formula The classifier label vector is obtained by formula A first probability is obtained that indicates whether the transcription factor-gene binding or regulation information is contained in the output representation, wherein, represents a relation extraction model, is an output representation output by the relation extraction model, represents a training weight matrix, represents a function formed by the last five layers of the relation extraction model, represents a function combination.

8. The method according to claim 7, wherein the method is characterized by, The step e further comprises: The output representations are respectively input into a syntax analyzer and a semantic analyzer to extract the syntactic structure and semantic dependency of the sentence, and generate a syntax semantic graph, wherein the nodes represent words and the edges represent the syntactic and semantic relationships in the sentence. The graph neural network utilizes the formula The syntax semantic graph is encoded into a syntax semantic graph representation vector through the formula The syntax semantic graph representation vector is fused with the classifier label vector, and the fused representation is input into a fully connected layer and a Softmax layer through the formula A second probability representing whether the transcription factor-gene binding or regulation information is contained in the output representation is obtained, wherein represents the graph neural network, represents the syntactic structure of the sentence, represents the semantic dependency of the sentence, represents the concatenation fusion operation, represents the weight term, represents the bias term.

9. The method for extracting yeast transcription factor-gene relationships from biological text according to claim 8, characterized in that, The step e further comprises: Setting the loss function where, denotes the weighted cross-entropy loss function value for the th sample, is the total number of sentence descriptions in the literature, , is the number of sentence descriptions associated with the th sample in the same article, is the number of transcription factor-gene pairs associated with the th sample in the same article, is a weighting coefficient, and is the first hyperparameter; The parameters of the syntactic analyzer, the semantic analyzer, and the graph neural network are continuously adjusted for secondary training until the value of the loss function is minimized, and the training is stopped. After the training is completed, the literature collected in real time is input into the model each time, and the joint probability is obtained according to the formula , wherein, is a second hyperparameter, and the joint probability greater than a preset value is regarded as containing transcription factor-gene binding or regulation information in the output representation, and the corresponding output representation is output.

Images