Clinical data classification method based on BERT, Bi-LSTM and attention mechanism

Abstract

The invention relates to a clinical data classification method based on BERT, Bi-LSTM and an attention mechanism, and the method comprises the steps: collecting clinical data of a patient, and carrying out the tag processing; performing word segmentation processing by using a word segmentation device; bERT is used as an encoder, and feature representation of data after word segmentation is obtained; inputting the feature representation into a Bi-LSTM model to obtain an advanced representation of the data sequence; inputting the advanced representation of the sequence into an attention layer to calculate an attention weight; and multiplying the attention weight by the output of the Bi-LSTM, inputting a result into a full-connection layer to perform data classification, and performing data classification by means of a softmax activation function. According to the method, the BERT, the Bi-LSTM and the attention mechanism are combined, long-text clinical data of the patient can be better processed, and the long-distance dependency relationship in the text can be better understood, so that the clinical data of the patient can be better classified, the disease type of the patient can be better analyzed, and a doctor can be helped to accurately grasp the disease condition and make a treatment scheme.

Description

Technical Field

[0001] The present invention belongs to the field of natural language processing, and specifically is a clinical data classification method based on BERT, Bi-LSTM and attention mechanism. Background Art

[0002] Clinical data, including the patient's chief complaint, current medical history, previous treatment history, and examination and test records, are crucial for doctors to understand the patient's condition and provide a basis for formulating subsequent diagnosis and treatment plans. Given the widespread existence of misdiagnosis, the effective use of clinical data to assess disease probability and provide diagnosis and treatment recommendations is of great significance to improving diagnostic accuracy.

[0003] Traditional neural network models, such as RNN, LSTM and other existing technologies, perform poorly in processing complex texts. Faced with large data sets or long clinical text sequences, these models may have difficulty capturing long-distance dependencies and are prone to forgetting early information. Moreover, long clinical text sequences are usually high-dimensional and complex, and traditional neural network models may not be able to extract all of their relevant features, resulting in reduced classifier performance. Furthermore, clinical data may be missing, noisy, and inconsistent, and these data quality issues will affect the training and classification accuracy of the model.

[0004] With the rapid development of machine learning and deep learning technologies, a class of large language models (LLMs) with huge parameters and deep network structures have emerged in recent years, such as GPT and BERT. Its application potential in clinical care is gradually emerging, especially in processing and analyzing large data sets from patients, such as medical images, texts, and electronic health records. It learns rich meta-representations by pre-training on a large amount of text data, so it can better capture and learn deeper language features, better understand word contexts, and handle long-distance dependencies. In addition, LLM can handle real and noise problems through data enhancement technology, such as supplementing missing information by generating synthetic data. And LLM trains and fine-tunes on relevant large data sets through self-supervised learning to form professional domain models, and the mode of outputting according to downstream tasks has achieved great success, so LLM can be fine-tuned on specific medical data sets to improve its prediction quality. Summary of the invention

[0005] In order to make better use of patients' clinical data, help doctors better understand patients' conditions, reduce misdiagnosis, and specify follow-up treatment plans, this paper proposes a clinical data classification method based on BERT, Bi-LSTM and attention mechanism.

[0006] The technical solution adopted by the present invention to achieve the above-mentioned purpose is: a clinical data classification method based on BERT, Bi-LSTM and attention mechanism, comprising the following steps:

[0007] S1. Collect the long corpus clinical data of patients, perform data cleaning, label processing, and word segmentation processing to obtain the [disease type label, text corpus word segmentation] pair data set; randomly divide it into a fine-tuning data set and a training data set;

[0008] S2. Construct a disease type prediction model for clinical data classification based on BERT, Bi-LSTM and attention mechanism, the model includes: BERT layer, Bi-LSTM layer, attention mechanism layer, fully connected classification layer, softmax classifier; and perform two-way loop training on the classification disease type prediction model based on the fine-tuning data set and the training data set to obtain an ideal model;

[0009] S3. Collect clinical data of patients to be diagnosed, process them by word segmentation, and input them into the ideal model to automatically output the classification results of disease types.

[0010] Label the patient information; label the disease type based on the patient's diagnosis information given by the doctor.

[0011] The word segmentation process is to define a Dataset class as a word segmenter, which inherits from torch.utils.data.Dataset of PyTorch, receives a DataFrame containing text and labels, processes the text into a format readable by the model through a tokenizer, and stores the disease type label.

[0012] The two-way loop training includes:

[0013] 1) Add a fully connected layer after the original BERT model for classification tasks, use the fine-tuning dataset to train it and save the network weight parameters, obtain the fine-tuned BERT model and output it to the subsequent network;

[0014] 2) The fine-tuned BERT layer, as well as the Bi-LSTM layer, attention mechanism layer, fully connected classification layer, and softmax classifier connected in sequence, are trained using the training data set so that the network uses text corpus segmentation as input and disease type labels as output, and performs iterative fitting learning to predict the disease type of unknown text corpus.

[0015] The disease type prediction model for clinical data classification includes:

[0016] The fine-tuned BERT layer is used to encode the data and obtain the feature representation of the data, that is, the last hidden layer state of BERT;

[0017] Bi-LSTM layer: The last hidden layer state of the BERT model is used as the input of Bi-LSTM to obtain the high-level representation of the feature sequence LstmOutput i ;

[0018] Attention layer: Input the high-level representation of the feature sequence into the attention layer, calculate the attention score, apply L2 regularization to it to enhance the generalization ability of the model, and use the softmax function to normalize the regularized attention score to obtain the attention weight AttenWeights; multiply the attention weight and the output of the Bi-LSTM to obtain the weighted sequence representation AttenOutput and output it to the fully connected layer;

[0019] Fully connected layer: The output of the attention mechanism is used as the input of the fully connected layer, and the high-dimensional output of the attention mechanism is converted into a set of original logits vectors y. Each element in the vector y corresponds to the original score of a category. The specific calculation formula is as follows:

[0020] y=W×AttenOutput+b

[0021] Where W is the weight matrix and b is the bias vector;

[0022] The softmax classifier normalizes the original logits vector y, maps each original score in the vector y to a probability value between [0, 1], outputs the maximum value of the probability value, and obtains the disease classification result.

[0023] The Bi-LSTM structure has two LSTM layers, one for processing sequences from forward to backward and the other for processing sequences from backward to forward, for processing both forward and backward context information at the same time. When processing sequences, the input of each time step is passed to the two LSTM layers respectively, and the final Bi-LSTM output is obtained by merging the forward and reverse hidden states at each time step:

[0024] Input sequence X = (X1, X2, ..., X n )In Bi-LSTM, the forward calculation is The reverse calculation is Final output LstmOutput i Depend on and composition.

[0025] The LSTM layer specifically calculates the high-level representation of the feature sequence through the following formula:

[0026] i t =σ(W ix Xt +W ih h t-1 +b i )

[0027] f t =σ(W fx X t +W fh h t-1 +b f )~C t =tanh(W cx X t +W ch h t-1 +b c )

[0029] C t =tanh(f t ⊙C t-1 +i t ⊙~C t )

[0030] o t =σ(W ox X t +W oh h t-1 +b o )

[0031] h t =o t ⊙tanh(C t )

[0032] In the above formula, i i 、f t , o t Respectively represent input gate, forget gate, and output gate; W ix Represents the input feature X of the input gate t The weight matrix, W ih Represents the hidden layer unit h of the previous moment of the input gate t-1 The weight matrix, W fx Represents the input feature X of the forget gate t The weight matrix, W fh Represents the hidden layer unit h of the previous moment of the forget gate t-1 The weight matrix, W cx Input feature X representing the memory cell update t The weight matrix, W ch Indicates the memory unit and the latest hidden layer unit h at the previous moment t-1 The weight matrix, W ox Represents the input feature X of the output gate t The weight matrix, W ohRepresents the previous hidden layer unit h of the output gate t-1 The weight matrix of i represents the bias variable of the input gate, b f represents the bias variable of the forget gate, b c represents the bias variable of the memory unit, b o Represents the bias variable of the output gate; X t Represents the input of the current time step; h t-1 represents the previous hidden layer unit; ~C t Represents the updated memory unit; C t Represents the current memory unit; σ represents the sigmoid activation function; tanh represents the hyperbolic tangent activation function; ⊙ represents the principal element multiplication.

[0033] The queries of all elements in the attention layer are calculated using the same task token, and L2 regularization is introduced into the calculation of attention weights. The specific calculation formula is as follows:

[0034]

[0035] AttenWeights=softmax(Attention L2 (z))

[0036] Where z is the input sequence, the dimension is N×D, N is the sequence length, D is the embedding dimension; t represents the task token, the dimension is 1×D t , D t is the token dimension, λ is the coefficient of the L2 regularization term, which is used to control the strength of regularization; W q is the weight matrix of the query, W k is the weight matrix of the key, W v It is the weight matrix of value.

[0037] The softmax classifier is classified through the softmax activation function, and the output prediction vector P = {p1, p2, ..., p i , ..., p n}, where n represents the number of text categories, p i It represents the probability that the text belongs to the i-th category, and the category with the largest probability value is taken as the final classification result.

[0038] A clinical data classification system based on BERT, Bi-LSTM and attention mechanism, including a computing background and a front-end interface mounted on a computer; the computing background stores a program, and when the program is loaded, the method steps as described above are executed to achieve disease type prediction; the front-end interface is used for human-computer interaction to input long corpus data, label information, model initial parameters, training parameters and related instructions for controlling model operation, as well as visually displaying the process data of model calculation and the final disease type result.

[0039] The present invention has the following beneficial effects and advantages:

[0040] 1. Accuracy: By fine-tuning the BERT model, integrating Bi-LSTM and improving the attention mechanism to add L2 regularization, the model can better handle large batches of text, better capture and learn deeper language features, better understand word context, and handle long-distance dependencies.

[0041] 2. Convenience: Not restricted by time or region, the trained model can be used at any time to classify and predict the patient's clinical data.

[0042] 3. Visualization: After inputting the patient's clinical data, the possible type of disease of the patient can be intuitively obtained, which is convenient for doctors to refer to and formulate the next treatment plan. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 Flow chart of the clinical data classification method of the present invention using BERT, Bi-LSTM and attention mechanism;

[0044] Figure 2 This is a schematic diagram of the BERT model structure;

[0045] Figure 3 This is a schematic diagram of the Bi-LSTM model structure;

[0046] Figure 4 Schematic diagram of the attention mechanism structure;

[0047] Figure 5 This is a diagram showing the actual effect of the clinical data classification method. DETAILED DESCRIPTION

[0048] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation method of the present invention is described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the invention, so the present invention is not limited by the specific implementation disclosed below.

[0049] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0050] A clinical data classification method based on BERT, Bi-LSTM and attention mechanism, including the following steps:

[0051] Step 1: Collection of clinical data of patients. Relevant hospitals record clinical data such as gender, age, chief complaint, diagnosis and treatment records of patients when they visit the hospital.

[0052] Step 2: Preprocess the patient's clinical information.

[0053] The following data cleaning steps were used:

[0054] 1) All incomplete instances (i.e. data with empty attribute values) are removed. This process is manually applied to the entire dataset. For instances with missing values, these are removed instead of being filled and completed due to the small number of instances.

[0055] 2) Remove irrelevant attributes that are not important to the study, such as hospital name, staff ID, and patient occupation, which may cause confusion in the model.

[0056] 3) Delete all duplicate data records.

[0057] 4) Merge the data and remove stop words.

[0058] Step 3: Mark the patient as ill based on the patient's diagnostic information.

[0059] 0 indicates ovarian malignancy, 1 indicates cervical malignancy, 2 indicates uterine endometrial malignancy, and 3 indicates invasive hydatidiform mole. Finally, the information of each patient is merged into a text, that is, the data format becomes [CancerType, Text], which is convenient for subsequent input of data into the model for classification.

[0060] Step 4: Use the word segmenter to segment the data.

[0061] Define a Dataset class, which inherits from PyTorch's torch.utils.data.Dataset, receives a DataFrame containing text and labels, processes the text into a model-readable format through tokenizer, and stores the labels. The Dataset class can be used to perform word segmentation on the dataset T. First, we get the word vector set T′={t1,t2,…,tn}, t i is one of the tokens that have been segmented, that is, each text entry in the dataset is first converted into a series of tokens, and then these tokens are converted into a fixed-length word vector set t = {w1, w2, ..., w L}, as the input of the BERT model.

[0062] Step 5: Fine-tune the BERT model using the segmented clinical data. Since the BERT model is pre-trained with a general corpus, it is not effective to directly apply it to the medical field. Therefore, clinical data can be used to fine-tune the BERT model. First, add a fully connected layer to it, and then divide the data into two parts, one for fine-tuning the BERT model and the other for training the classification model, forming a nested two-way training loop. Each part of the data is divided into three parts: training set, validation set, and test set. The weight of the model is saved according to the test results and applied to subsequent experiments.

[0063] Step 6: Build and train the fine-tuned BERT model with Bi-LSTM and attention mechanism. The specific process is as follows:

[0064] 1) Input the segmented patient clinical data into the fine-tuned BERT, encode the data through BERT, and obtain the feature representation of the data, that is, the last hidden layer state of BERT. The hidden layer state of BERT is a tensor representation, and the shape of this tensor is [batch_size, sequence_length, hidden_size]. Among them, batch_size is the batch size of the input data, sequence_length is the maximum length of the sequence, and hidden_size is the hidden layer size of the BERT model.

[0065] 2) The last hidden layer state of BERT is used as the input of Bi-LSTM to obtain a high-level representation of the feature sequence, which is calculated by the following formula:

[0066] i t =σ(W ix X t +W ih h t-1 +b i )

[0067] f t =σ(W fx X t +W fh h t-1 +b f )~C t=tanh(W cx X t +W ch h t-1 +b c )

[0069] C t =tanh(f t ⊙C t-1 +i t ⊙~C t )

[0070] o t =σ(W ox X t +W oh h t-1 +b o )

[0071] h t =o t ⊙tanh(C t )

[0072] In the above formula, i i 、f t , o t Respectively represent input gate, forget gate, and output gate; W ix Represents the input feature X of the input gate t The weight matrix, W ih Represents the hidden layer unit h of the previous moment of the input gate t-1 The weight matrix, W fx Represents the input feature X of the forget gate t The weight matrix, W fh Represents the hidden layer unit h of the previous moment of the forget gate t-1 The weight matrix, W cx Input feature X representing the memory cell update t The weight matrix, W ch Indicates the memory unit and the latest hidden layer unit h at the previous moment t-1 The weight matrix, W ox Represents the input feature X of the output gate t The weight matrix, W oh Represents the previous hidden layer unit h of the output gate t-1 The weight matrix of i represents the bias variable of the input gate, b f represents the bias variable of the forget gate, b c represents the bias variable of the memory unit, b o Represents the bias variable of the output gate; X t is the input of the current time step; h t-1 represents the previous hidden layer unit; ~Ct Represents the updated memory unit; C t Represents the current memory unit; σ represents the sigmoid activation function; tanh represents the hyperbolic tangent activation function; ⊙ represents the principal element multiplication.

[0073] There are two LSTM layers in the Bi-LSTM structure, one processes the sequence from front to back, and the other processes the sequence from back to front; therefore, the model can use the previous and next context information at the same time. When processing the sequence, the input of each time step will be passed to the two LSTM layers separately, and the final Bi-LSTM output is the forward and reverse hidden states merged at each time step; that is, the input sequence X = (X1, X2, ..., X n )In Bi-LSTM, the forward calculation is The reverse calculation is Final output LstmOutput i Depend on and composition.

[0074] 3) The high-level representation of the feature sequence is input into the attention layer, the original attention score is calculated, L2 regularization is applied to it to enhance the generalization ability of the model, and the regularized attention score is normalized using the softmax function to obtain the attention weight.

[0075] In the model proposed in this article, the query of all elements in the attention mechanism is calculated using the same task token instead of the input token itself, and L2 regularization is introduced into the calculation of the attention weight. The specific calculation formula is as follows:

[0076]

[0077] AttenWeights=softmax(Attention L2 (z))

[0078] Where z is the input sequence, the dimension is N×D, N is the sequence length, D is the embedding dimension; t represents the task token, the dimension is 1×D t , D t is the token dimension; λ is the coefficient of the L2 regularization term, which is used to control the strength of regularization; W q is the weight matrix of the query, W k is the weight matrix of the key, W v It is the weight matrix of value.

[0079] 4) Multiply the attention weight and the output of Bi-LSTM to obtain the weighted sequence representation. The specific calculation formula is as follows:

[0080]

[0081] Where AttenWeights i is the attention weight, LstmOutput i is the output of the LSTM layer.

[0082] 5) The weighted sequence representation is passed through a fully connected layer to convert the high-dimensional output of the attention mechanism into a set of original logits vectors y. Each element in the vector y corresponds to the original score of a category. The specific calculation formula is as follows:

[0083] y=W×AttenOutput+b

[0084] Where W is the weight matrix and b is the bias vector.

[0085] Step 7: Use the training set data to train the built model until the model converges and the final model training loss value reaches 1e-2. During the training process, the model parameters are set as follows:

[0086] Table 1 Model parameter settings

[0087]

[0088] Step 8: Conduct ablation experiments. The ablation experiment gradually removes certain components in the model to show the ultimate contribution of each component to the performance of the model. There are three ablation experiments, namely, a single BERT model, a BERT+Bi-LSTM model, and a BERT+attention model.

[0089] The ablation experiment will use precision, recall, and F1-score indicators to evaluate the model performance. Let TP represent the positive sample predicted as the positive class; TN represent the negative sample predicted as the negative class; FP represent the negative sample predicted as the positive class; and FN represent the positive sample predicted as the negative class.

[0090] The definitions and calculation formulas of each evaluation index are as follows:

[0091] Precision is the proportion of samples that are actually positive among all samples predicted to be positive. The calculation formula is:

[0092]

[0093] The recall rate is the proportion of samples that are predicted to be positive among samples that are actually positive. The calculation formula is:

[0094]

[0095] F1-score is the harmonic mean of precision and recall, and its calculation formula is:

[0096]

[0097] The results of the ablation experiment are as follows:

[0098] Table 2 Ablation experiment

[0099]

[0100] The experimental results show that BERT, Bi-LSTM and attention mechanism are all key factors in improving the performance of clinical diagnosis models. When BERT, Bi-LSTM and attention mechanism are combined, the precision, recall and F1-score of the model reach the highest 0.927, 0.950 and 0.939 respectively. This shows that the combination of the three can maximize their respective advantages and jointly improve the performance of the model in clinical diagnosis tasks.

[0101] Step 9: Classify the output of the fully connected layer through the softmax activation function and output the prediction vector P = {p1, p2, ..., p i ,…,p n}, where n represents the number of text categories, p i It represents the probability that the text belongs to the i-th category, and the category with the largest probability value is taken as the final classification result.

[0102] like Figure 5 The figure shows the actual effect of the clinical data classification method based on BERT, Bi-LSTM and attention mechanism. The model accepts the patient's clinical data, and clicks the Start Prediction button to obtain the classification result of the patient's clinical data.

[0103] In addition, the present invention also provides a clinical data classification system based on BERT, Bi-LSTM and attention mechanism, including a computing background and a front-end interface mounted on a computer; the computing background is provided with a processor and a memory, and the memory stores a program. When the processor loads the program, the method steps as described above are executed to realize the prediction of the disease type; the front-end interface is used for human-computer interaction to input long corpus data, label information, model initial parameters, training parameters and related instructions for controlling the operation of the model, as well as to visually display the process data of the model calculation and the final disease type results.

[0104] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A clinical data classification method based on BERT, Bi-LSTM and attention mechanism, characterized in that: The following steps are involved: S1. Collect the long corpus clinical data of patients, perform data cleaning, label processing, and word segmentation processing to obtain the [disease type label, text corpus word segmentation] pair data set; randomly divide it into a fine-tuning data set and a training data set; S2. Construct a disease type prediction model for clinical data classification based on BERT, Bi-LSTM and attention mechanism, wherein the model includes: a BERT layer, a Bi-LSTM layer, an attention mechanism layer, a fully connected classification layer, and a softmax classifier; Based on the fine-tuning data set and the training data set, the classification disease type prediction model is trained in two ways to obtain an ideal model; S3. Collect clinical data of patients to be diagnosed, process them by word segmentation, and input them into the ideal model to automatically output the classification results of disease types.

2. A clinical data classification method based on BERT, Bi-LSTM and attention mechanism according to claim 1, characterized in that: Label the patient information; label the disease type based on the patient's diagnosis information given by the doctor.

3. A clinical data classification method based on BERT, Bi-LSTM and attention mechanism according to claim 1, characterized in that: The word segmentation process is to define a Dataset class as a word segmenter, which inherits from torch.utils.data.Dataset of PyTorch, receives a DataFrame containing text and labels, processes the text into a format readable by the model through a tokenizer, and stores the disease type label.

4. A clinical data classification method based on BERT, Bi-LSTM and attention mechanism according to claim 1, characterized in that: The two-way loop training includes: 1) Add a fully connected layer after the original BERT model for classification tasks, use the fine-tuning dataset to train it and save the network weight parameters, obtain the fine-tuned BERT model and output it to the subsequent network; 2) The fine-tuned BERT layer, as well as the Bi-LSTM layer, attention mechanism layer, fully connected classification layer, and softmax classifier connected in sequence, are trained using the training data set so that the network uses text corpus segmentation as input and disease type labels as output, and performs iterative fitting learning to predict the disease type of unknown text corpus.

5. A clinical data classification method based on BERT, Bi-LSTM and attention mechanism according to claim 4, characterized in that: The disease type prediction model for clinical data classification includes: The fine-tuned BERT layer is used to encode the data and obtain the feature representation of the data, that is, the last hidden layer state of BERT; Bi-LSTM layer: The last hidden layer state of the BERT model is used as the input of Bi-LSTM to obtain the high-level representation of the feature sequence LstmOutput i ; Attention layer: Input the high-level representation of the feature sequence into the attention layer, calculate the attention score, apply L2 regularization to it to enhance the generalization ability of the model, and use the softmax function to normalize the regularized attention score to obtain the attention weight AttenWeights; multiply the attention weight and the output of the Bi-LSTM to obtain the weighted sequence representation AttenOutput and output it to the fully connected layer; Fully connected layer: The output of the attention mechanism is used as the input of the fully connected layer, and the high-dimensional output of the attention mechanism is converted into a set of original logits vectors y. Each element in the vector y corresponds to the original score of a category. The specific calculation formula is as follows: y=W×AttenOutput+b Where W is the weight matrix and b is the bias vector; The softmax classifier normalizes the original logits vector y, maps each original score in the vector y to a probability value between [0,1], outputs the maximum value of the probability value, and obtains the disease classification result.

6. A clinical data classification method based on BERT, Bi-LSTM and attention mechanism according to claim 1, 4 or 5, characterized in that: The Bi-LSTM structure has two LSTM layers, one for processing sequences from forward to backward and the other for processing sequences from backward to forward, for processing both forward and backward context information at the same time. When processing sequences, the input of each time step is passed to the two LSTM layers respectively, and the final Bi-LSTM output is obtained by merging the forward and reverse hidden states at each time step: Input sequence X = (σ1, X2,…, X n )In Bi-LSTM, the forward calculation is The reverse calculation is Final output LstmOutput i Depend on and composition.

7. A clinical data classification method based on BERT, Bi-LSTM and attention mechanism according to claim 6, characterized in that: The LSTM layer specifically calculates the high-level representation of the feature sequence through the following formula: i t =σ(W ic X t +W ih h t-1 +b i ) f t =σ(w fx x t +W fh h t-1 +b f )~C t =tanh(W cx X t +W ch h t-1 +b c ) C t =tanh(f t ⊙C t-1 +i t ⊙~C t ) o t =σ(W ox X t +W oh h t-1 +b o ) h t =o t ⊙tanh(C t ) In the above formula, i i 、f t , o t Respectively represent input gate, forget gate, and output gate; W ix Represents the input feature X of the input gate t The weight matrix, W ih Represents the hidden layer unit h of the previous moment of the input gate t-1 The weight matrix, W fx Represents the input feature X of the forget gate t The weight matrix, W fh Represents the hidden layer unit h of the previous moment of the forget gate t-1 The weight matrix, W cx Input feature X representing the memory cell update t The weight matrix, W ch Indicates the memory unit and the latest hidden layer unit h at the previous moment t-1 The weight matrix, W ox Represents the input feature X of the output gate t The weight matrix, W oh Represents the previous hidden layer unit h of the output gate t-1 The weight matrix of i represents the bias variable of the input gate, b f represents the bias variable of the forget gate, b c represents the bias variable of the memory unit, b o Represents the bias variable of the output gate; X t Represents the input of the current time step; h t-1 represents the previous hidden layer unit; ~C t Represents the updated memory unit; C t Represents the current memory unit; σ represents the sigmoid activation function; tanh represents the hyperbolic tangent activation function; ⊙ represents the principal element multiplication.

8. A clinical data classification method based on BERT, Bi-LSTM and attention mechanism according to claim 1, characterized in that: The queries of all elements in the attention layer are calculated using the same task token, and L2 regularization is introduced into the calculation of attention weights. The specific calculation formula is as follows: AttentionWeights=softmax(Attention L2 (z)) Where z is the input sequence, the dimension is N×D, N is the sequence length, D is the embedding dimension; t represents the task token, the dimension is 1×D t , D t is the token dimension, λ is the coefficient of the L2 regularization term, which is used to control the strength of regularization; W q is the weight matrix of the query, W k is the weight matrix of the key, W v It is the weight matrix of value.

9. A clinical data classification method based on BERT, Bi-LSTM and attention mechanism according to claim 1, characterized in that: The softmax classifier is classified through the softmax activation function and outputs the prediction vector P = {p1, p2, ..., p i ,…,p n }, where n represents the number of text categories, p i It represents the probability that the text belongs to the i-th category, and the category with the largest probability value is taken as the final classification result.

10. A clinical data classification system based on BERT, Bi-LSTM and attention mechanism, characterized in that: It includes a computing background and a front-end interface mounted on a computer; the computing background stores a program, and when the program is loaded, the method steps described in any one of claims 1 to 9 are executed to realize disease type prediction; the front-end interface is used for human-computer interaction to input long corpus data, label information, model initial parameters, training parameters and related instructions for controlling model operation, as well as visually displaying the process data of model calculation and the final disease type result.

Images