A method for classifying pediatric medical text data

Abstract

The present invention discloses a method for classifying children's medical text data, comprising: (1) obtaining children's medical text data, and annotating to obtain hierarchical multi-label text; (2) processing to obtain medical text, positive sample label text, and negative sample label text, and constructing a triple data set; (3) constructing a three-tower model, the three-tower model comprising three encoder modules, a label recall module, and a label sorting module; (4) training the three-tower model based on the triple data set; (5) using the trained three-tower model, respectively processing the parent class label text and the child class label text in the hierarchical multi-label text, and saving the obtained feature vectors in an offline feature vector library; (6) for the children's medical text to be predicted, using the three-tower model for processing, and then combining with the offline feature vector library, predicting and obtaining the top-k predicted labels with a similarity higher than a threshold. The present invention can greatly improve the matching efficiency of children's medical text to labels.

Description

Technical Field

[0001] The present invention belongs to the field of medical data processing classification, and in particular relates to a children's medical text data classification method. Background Art

[0002] The classification and grading of children's medical data is to divide medical data into different categories and levels according to the characteristics, sensitivity, value and other factors of medical data, so as to effectively manage medical data. In medical scenarios, medical text data has the characteristics of huge data volume, heterogeneous and complex data, and multi-level categories. Dividing medical data into different categories is of great significance for improving the security of medical data, promoting the effective use of medical data, and realizing the assetization of medical data.

[0003] Multi-label text classification aims to select one or more categories corresponding to text from a predefined set of candidate labels. It is a basic task of language processing. The hierarchical labels are stored in a predetermined hierarchical structure. The current multi-label classification methods still combine multiple models to train and predict a data set, and statically sample negative samples when sorting labels during model training and reasoning. This method consumes a lot of computing resources. At the same time, due to the static sampling of negative samples, the model only focuses on a small number of negative sample labels when sorting labels, making it difficult for the model to converge.

[0004] A Chinese patent document with publication number CN111403028A discloses a medical text method and apparatus, a storage medium, and an electronic device, which classify the medical text to be classified according to the attention value of the medical text to be classified according to the medical feature attributes.

[0005] A Chinese patent document with publication number CN109471945A discloses a medical text classification method, device and storage medium based on deep learning, which uses a recursive neural network and a bidirectional attention mechanism to classify medical texts.

[0006] However, in medical scenarios, the classification system is large and complex. The Chinese patent document with publication number CN111403028A consumes a lot of resources and time when calculating the attention value of medical attributes to classified medical texts, and cannot quickly and accurately obtain classification results. The Chinese patent document with publication number CN109471945A also cannot efficiently process complex medical text data. Summary of the invention

[0007] The present invention provides a method for classifying children's medical text data, which can effectively handle the hierarchical multi-label text classification problem of children's medical data, has small calculation amount and fast operation speed, and greatly improves the matching efficiency of children's medical text to labels.

[0008] A method for classifying children's medical text data includes the following steps:

[0009] (1) Obtain children's medical text data and annotate them to obtain hierarchical multi-label text;

[0010] (2) Processing children's medical text data and hierarchical multi-label text into medical text, positive sample label text, and negative sample label text to obtain a triple data set; wherein each positive sample label text and each negative sample label text contains a parent class label text and a corresponding child class label text;

[0011] (3) constructing a three-tower model, wherein the three-tower model includes three encoder modules, a label recall module, and a label sorting module;

[0012] The three encoder modules share weights, where the first encoder and the second encoder are used to generate feature vectors of medical text and parent class label text respectively; the label recall module measures the similarity of the feature vectors of medical text and parent class label text, dynamically obtains the subclass positive and negative label texts corresponding to the several parent class labels with the highest similarity, and inputs them into the third encoder to generate the feature vector of the subclass label text; the medical text feature vector generated by the first encoder and the subclass label text feature vector generated by the third encoder are input into the label sorting module together, and the top-k label text is generated as the prediction output according to the similarity value;

[0013] (4) Train the three-tower model based on the triplet dataset;

[0014] (5) Using the trained three-tower model, the parent class label text and the child class label text in the hierarchical multi-label text are processed respectively, and the feature vectors corresponding to all the processed label texts are saved in the offline feature vector library;

[0015] (6) For the children's medical text to be predicted, the trained three-tower model is used for processing, and then combined with the offline feature vector library to predict the top-k prediction labels with similarity higher than the similarity threshold.

[0016] In step (3), the encoder module includes a transformer-based Bert encoding layer, a matrix decomposition part and a pooling layer;

[0017] The matrix decomposition part is used to decompose the attention matrix in the encoding process of the Bert encoding layer into multiple smaller matrices; the role of the pooling layer is to process the output of the Bert encoding layer to obtain a fixed-size sentence embedding, that is, a feature vector.

[0018] Each layer of the Bert encoding layer contains an encoder unit, and each encoder unit is composed of a multi-head attention mechanism module, a feedforward layer, and a cross-layer normalization layer.

[0019] The multi-head attention mechanism module is composed of multiple self-attention mechanism modules. The specific formula is:

[0020]

[0021] Where Q = W Q X, K = W K X、V=W V X corresponds to Query, Key, and Value respectively. In the self-attention mechanism, it is the transformation of input X itself. k is the dimension of the word vector;

[0022] The matrix decomposition part samples the rows of the Query matrix and the columns of the Key matrix according to the distribution scores, thereby decomposing them into corresponding sub-matrices. The specific formula is as follows:

[0023] A c =exp(Q(K T ) c )

[0024] A r =exp((Q) r K T )

[0025] U=W +

[0026] Among them, A c represents the submatrix after sampling the columns of the Key matrix, c represents the sampling of the columns, the number of samples is k, and the matrix A c The dimension is (k, N); A r represents the submatrix after sampling the columns of the Query matrix, r represents sampling the rows, the number of samples is k, and the matrix A r The dimension is (k, N); the intermediate matrix is ​​U = W + , where W is the matrix A c and the matrix A r Multiply to get, W + is its pseudo-inverse matrix; decompose the submatrix A obtained by Query and Key c , A r and the intermediate matrix U to obtain the output O = A c UA r .

[0027] In step (3), the label recall module includes a first similarity measurement module and a dynamic sample sampling module;

[0028] The first similarity measurement module receives the feature vectors of the medical text and the parent class label text output by the first encoder and the second encoder respectively, and performs similarity measurement;

[0029] According to the obtained similarity ranking of the parent class labels, the sub-class positive and negative label texts corresponding to the several parent class labels with the highest similarity are dynamically obtained, and the obtained sub-class positive and negative label texts are used as the input of the third encoder.

[0030] In step (3), the label sorting module includes a second similarity measurement module and a classifier module;

[0031] Among them, the structure of the second similarity measurement module is consistent with the structure of the first similarity measurement module in the label recall module; the second similarity measurement module accepts the feature vectors of the medical text and the subclass label text output by the first encoder and the third encoder respectively, and performs similarity measurement; the top-k label texts with similarity values ​​higher than the similarity threshold are selected as the prediction output.

[0032] In step (4), when training the three-tower model, the loss function is set in the label recall module and the label sorting module respectively. The formula is as follows:

[0033] Loss = α loss binary +(1-α)·loss triplet

[0034]

[0035] loss triplet =max(d(a,p)-d(a,n)+margin)

[0036] Among them, Loss represents the loss function set when distinguishing parent class label samples from child class label samples, α represents the weight for balancing the binary cross entropy loss, and loss binary Indicates the binary cross entropy loss of whether the medical text is similar to the label text. y is the true value of whether the two sample pairs in the binary classification are similar. Is the predicted value of whether two samples are similar; loss triplet represents the triple loss between medical text, positive sample label text, and negative sample label text. d(a,p) represents the distance between the feature vector of medical text and positive sample label text. d(a,n) represents the distance between the feature vector of medical text and negative sample label text. Margin represents the difference in the distance between the feature vectors of medical text and positive label text and negative label text.

[0037] The specific process of step (5) is as follows:

[0038] The encoder module is used to process all parent-class label texts and child-class text labels respectively, obtain the corresponding feature vector library and store it offline; the parent-class label offline feature vector library and the child-class label text offline feature vector library are linked according to the hierarchical correspondence, so as to quickly locate the child-class label offline feature vector library based on the parent-class label offline feature vector library.

[0039] The specific process of step (6) is as follows:

[0040] Prepare the medical text to be predicted, use the first encoder in the trained three-tower model to obtain the feature vector of the medical text to be predicted, measure its similarity with the parent class label in the parent class label offline feature vector library, obtain the probability of whether the text pairs are similar, sort them according to the similarity probability, and recall the subclass label offline feature vector library of the subclass label text corresponding to the parent class label with higher similarity;

[0041] The feature vector of the medical text to be predicted is measured again with the offline feature vector library of the subclass label text to obtain the probability of whether the text pairs are similar. Then, the texts are sorted according to the similarity probability to obtain the top-k predicted label texts whose similarity values ​​are higher than the similarity threshold.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] 1. The present invention proposes a new deep learning method, which is trained through an end-to-end method, gives full play to the advantages of the transformer neural network, and uses the dynamic negative sample sampling method to recall and sort the label text, so that the model learns from simple samples to difficult samples. At the same time, the method of the present invention has a small amount of calculation and a fast operation speed, which greatly improves the matching efficiency of text to label.

[0044] 2. The present invention constructs a three-tower model to obtain high-quality text features in the encoder module and perform multi-level feature extraction, which significantly improves classification accuracy; the label recall module effectively reduces the resource consumption of online computing and improves the efficiency of medical text classification through dynamic negative sample sampling and the use of an offline feature vector library of label text; the label sorting module distinguishes the correct label from the positive and negative labels to ensure that the classification results are accurate and reliable. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A flowchart of a method for classifying children's medical text data according to an embodiment of the present invention;

[0046] Figure 2 A network structure diagram of a three-tower model in an embodiment of the present invention;

[0047] Figure 3Schematic diagram of the coding layer structure in an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be pointed out that the embodiments described below are intended to facilitate the understanding of the present invention and do not have any limiting effect on the present invention.

[0049] like Figure 1 As shown, a method for classifying children's medical text data includes the following steps:

[0050] 1) Obtain children's medical text data, annotate it to obtain hierarchical multi-label text, and construct a triple dataset.

[0051] The acquired medical text and the annotated hierarchical multi-label text are processed into text data, positive sample label text, and negative sample label text to form a triplet data set. In the process of processing label text, taking the three-level multi-label text as an example, the first-level label and all the second-level labels under this first-level label are spliced ​​as the parent label input of the positive and negative label text, for example: first-level label#second-level label1#second-level label2#..., and the hierarchical labels are spliced ​​in order according to the level to form the sub-class labels of the positive and negative label text, for example: first-level label#second-level label#third-level label,..., in the label recall process, the sub-class label text is recalled according to the matching of the text encoding and the parent label text encoding.

[0052] 2) Build a three-tower model and train it.

[0053] The three-tower model includes an encoder module, a label recall module, and a label sorting module. Figure 2 As shown, the three encoder modules share weights, wherein the first encoder and the second encoder are used to generate feature vectors of medical text and parent class label text, respectively; the label recall module measures the similarity of the feature vectors of medical text and parent class label text, dynamically obtains the sub-class positive and negative label texts corresponding to the parent class labels with the highest similarity, and generates the feature vector of the sub-class label text after inputting it into the third encoder; the medical text feature vector generated by the first encoder and the sub-class label text feature vector generated by the third encoder are input into the label sorting module together, and the top-k label text is generated as the prediction output according to the similarity value.

[0054] In the present invention, the encoder module generates feature vectors of medical text and label text, including a transformer-based Bert encoder, a matrix decomposition part, and a pooling layer, such as Figure 3 shown.

[0055] The Bert encoder is a type of Transformer structure, in which the encoding module is the encoder unit of the standard transformer structure. A transformer encoder unit is generated by the superposition of a multi-head-attention module, a feed forward layer, and a cross-layer normalization (Add&Norm) layer. Each layer of Bert is composed of such an encoder unit.

[0056] The Multi-head-Attention module consists of multiple Self-Attention modules, and its specific formula is:

[0057]

[0058] Where Q = W Q X, K = W K X, V = W V X corresponds to Query, Key, and Value respectively. In self-attention, it is the transformation of input X itself. k is the dimension of the word vector.

[0059] The cross-layer normalization (Add&Norm) layer includes residual connection and normalization. The residual connection is used to prevent network degradation. Normalization normalizes the activation values ​​of each layer, which can be expressed as follows:

[0060] LayerNorm(X+Sublayer(X))

[0061] The Feed Forward layer is a fully connected structure.

[0062] FFN(X)=max(0,XW1+b1)W2+b2

[0063] like Figure 3 As shown, in the matrix decomposition part, the rows of the Query matrix and the columns of the Key matrix are sampled according to the distribution scores, so as to be decomposed into corresponding sub-matrices. The specific formula is as follows:

[0064] A c =exp(Q(K T ) c )

[0065] A r =exp((Q) r K T )

[0066] U=W +

[0067] Among them, A c represents the submatrix after sampling the columns of the Key matrix, c represents the sampling of the columns, the number of samples is k, and the matrix A c The dimension is (k, N), A r represents the submatrix after sampling the columns of the Query matrix, r represents sampling the rows, the number of samples is k, and the matrix A r The dimension is (k, N), and the intermediate matrix is ​​U = W + , where W is the matrix A c and the matrix A r Multiply to get, W + is its pseudo-inverse matrix. The submatrix A obtained by decomposing Query and Key c , A r and the intermediate matrix U to obtain the output O = A c UA r Through matrix decomposition, the attention matrix is ​​decomposed into multiple smaller matrices, which reduces the data dimension while retaining the main features of the data, can greatly reduce the computational complexity and improve the effectiveness and efficiency of the model.

[0068] The role of the pooling layer is to process the output of the Bert encoding layer to obtain a fixed-size sentence embedding. In the specific implementation, three pooling strategies were tried: CLS-labeling strategy: using the output of the CLS-labeling (the original Bert vector for classification, taking the first token vector of the sentence), MEAN strategy: calculating the average of all output word vectors, MAX strategy: calculating the maximum value of the output word vector, and the MEAN strategy had the best effect.

[0069] When training the three-tower model, the loss functions are set in the label recall module and the label sorting module respectively. The loss function is set by the binary cross entropy loss of whether the text is similar to the positive sample label and the triplet loss based on the distance measurement between feature vectors. The formula is as follows:

[0070] Loss = α loss binary +(1-α)·loss triplet

[0071]

[0072] loss triplet =max(d(a,p)-d(a,n)+margin)

[0073] Among them, Loss represents the loss function set when distinguishing parent class label samples from child class label samples, α represents the weight for balancing the binary cross entropy loss, and lossbinary Indicates the binary cross entropy loss of whether the text to be predicted is similar to the label text. y is the true value of whether the two sample pairs in the binary classification are similar. Is the predicted value of whether two samples are similar, loss triplet represents the triple loss between the text to be predicted, the positive sample label text, and the negative sample label text, d(a,p) represents the distance between the feature vectors of the text to be predicted and the positive sample label text, d(a,n) represents the distance between the feature vectors of the text to be predicted and the negative sample label text, and margin represents the difference in the distance between the feature vectors of the text to be predicted and the positive label text and the negative label text.

[0074] The concept of Triplet loss is to make the text feature vector closer to the positive sample label text feature vector and farther away from the negative sample label text feature vector. In order to ensure that there is a certain distance difference between the feature vector of the positive sample label text and the feature vector of the negative sample label text to avoid the situation where the distance is equal, a hyperparameter margin is given to ensure that the feature vectors of the positive sample label text and the negative sample label text will not be equal.

[0075] The label recall module dynamically generates positive sample labels and negative sample labels, including a similarity measurement module and a dynamic negative sample sampling module. The similarity measurement module accepts the feature vectors of the medical text and the parent class label text output by the first encoder and the second encoder, and performs similarity measurement, wherein the cosine similarity is used for measurement. The cosine similarity formula is as follows:

[0076]

[0077] Among them, u and v represent two matrix vectors. According to the obtained similarity ranking of the parent class labels, the sub-class positive and negative label texts corresponding to the several parent class labels with the highest similarity are dynamically obtained, and the obtained sub-class positive and negative label texts are used as the input of the third encoder, and the third encoder outputs the feature vector of the sub-class label text.

[0078] The label sorting module distinguishes the positive label samples from the generated positive and negative label samples, which includes a similarity measurement module and a classifier module. The structure of the similarity measurement module is consistent with that of the similarity measurement module in the label recall module. It accepts the feature vectors of the medical text and the subclass label text output by the first encoder and the third encoder respectively, and performs similarity measurement, and selects the top-k label texts with similarity values ​​higher than the similarity threshold as the prediction output.

[0079] 3) Use the trained three-tower model to process the medical text to be predicted and the label text respectively.

[0080] The encoder module is used to process all parent label texts and child text labels respectively, obtain the corresponding feature vector library and store it offline. The parent label offline feature vector library and the child label text offline feature vector library are linked according to the hierarchical correspondence, so as to quickly locate the child label offline feature vector library based on the parent label offline feature vector library.

[0081] 4) Prepare the medical text to be predicted, use the first encoder in the trained three-tower model to obtain the feature vector of the medical text to be predicted, measure the similarity between it and the parent class label in the parent class label offline feature vector library, obtain the probability of whether the text pairs are similar, sort them according to the similarity probability, and recall the subclass label offline feature vector library of the subclass label text corresponding to the parent class label with higher similarity; again measure the similarity between the feature vector of the medical text to be predicted and the offline feature vector library of the subclass label text, obtain the probability of whether the text pairs are similar, and then sort them according to the similarity probability to obtain the top-k predicted label texts whose similarity values ​​are higher than the similarity threshold.

[0082] The embodiments described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for classifying children's medical text data, characterized in that: The following steps are involved: (1) Obtain children's medical text data and annotate them to obtain hierarchical multi-label text; (2) Processing children's medical text data and hierarchical multi-label text into medical text, positive sample label text, and negative sample label text to obtain a triple data set; wherein each positive sample label text and each negative sample label text contains a parent class label text and a corresponding child class label text; (3) constructing a three-tower model, wherein the three-tower model includes three encoder modules, a label recall module, and a label sorting module; The three encoder modules share weights, where the first encoder and the second encoder are used to generate feature vectors of medical text and parent class label text respectively; the label recall module measures the similarity of the feature vectors of medical text and parent class label text, dynamically obtains the subclass positive and negative label texts corresponding to the several parent class labels with the highest similarity, and inputs them into the third encoder to generate the feature vector of the subclass label text; the medical text feature vector generated by the first encoder and the subclass label text feature vector generated by the third encoder are input into the label sorting module together, and the top-k label text is generated as the prediction output according to the similarity value; (4) Train the three-tower model based on the triplet dataset; (5) Using the trained three-tower model, the parent class label text and the child class label text in the hierarchical multi-label text are processed respectively, and the feature vectors corresponding to all the processed label texts are saved in the offline feature vector library; (6) For the children's medical text to be predicted, the trained three-tower model is used for processing, and then combined with the offline feature vector library to predict the top-k prediction labels with similarity higher than the similarity threshold.

2. The method for classifying children's medical text data according to claim 1, characterized in that: In step (3), the encoder module includes a transformer-based Bert encoding layer, a matrix decomposition part and a pooling layer; The matrix decomposition part is used to decompose the attention matrix in the encoding process of the Bert encoding layer into multiple smaller matrices; the role of the pooling layer is to process the output of the Bert encoding layer to obtain a fixed-size sentence embedding, that is, a feature vector.

3. The method for classifying children's medical text data according to claim 2, characterized in that: Each layer of the Bert encoding layer contains an encoder unit, and each encoder unit is composed of a multi-head attention mechanism module, a feedforward layer, and a cross-layer normalization layer. The multi-head attention mechanism module is composed of multiple self-attention mechanism modules. The specific formula is: Where Q = W Q X, K = W K X、V=W V X corresponds to Query, Key, and Value respectively. In the self-attention mechanism, it is the transformation of input X itself. k is the dimension of the word vector; The matrix decomposition part samples the rows of the Query matrix and the columns of the Key matrix according to the distribution scores, thereby decomposing them into corresponding sub-matrices. The specific formula is as follows: A c =exp(Q(K T ) c ) A r =exp((Q) r K T ) U=W + Among them, A c represents the submatrix after sampling the columns of the Key matrix, c represents the sampling of the columns, the number of samples is k, and the matrix A c The dimension is (k, N); A r represents the submatrix after sampling the rows of the Query matrix, r represents the sampling of the rows, the number of samples is k, and the matrix A r The dimension is (k, N); the intermediate matrix is ​​U = W + , where W is the matrix A c and the matrix A r Multiply to get, W + is its pseudo-inverse matrix; decompose the submatrix A obtained by Query and Key c , A r and the intermediate matrix U to obtain the output O = A c UA r .

4. The method for classifying children's medical text data according to claim 1, characterized in that: In step (3), the label recall module includes a first similarity measurement module and a dynamic sample sampling module; The first similarity measurement module receives the feature vectors of the medical text and the parent class label text output by the first encoder and the second encoder respectively, and performs similarity measurement; According to the obtained similarity ranking of the parent class labels, the sub-class positive and negative label texts corresponding to the several parent class labels with the highest similarity are dynamically obtained, and the obtained sub-class positive and negative label texts are used as the input of the third encoder.

5. The method for classifying children's medical text data according to claim 4, characterized in that: In step (3), the label sorting module includes a second similarity measurement module and a classifier module; Among them, the structure of the second similarity measurement module is consistent with the structure of the first similarity measurement module in the label recall module; the second similarity measurement module accepts the feature vectors of the medical text and the subclass label text output by the first encoder and the third encoder respectively, and performs similarity measurement; the top-k label texts with similarity values ​​higher than the similarity threshold are selected as the prediction output.

6. The method for classifying children's medical text data according to claim 1, characterized in that: In step (4), when training the three-tower model, the loss function is set in the label recall module and the label sorting module respectively. The formula is as follows: Loss=α·loss binary +(1-α)·loss triplet loss triplet =max(d(a,p)-d(a,n)+margin) Among them, Loss represents the loss function set when distinguishing parent class label samples from child class label samples, α represents the weight for balancing the binary cross entropy loss, and loss binary Indicates the binary cross entropy loss of whether the medical text is similar to the label text. y is the true value of whether the two sample pairs in the binary classification are similar. Is the predicted value of whether two samples are similar; loss triplet represents the triple loss between medical text, positive sample label text, and negative sample label text. d(a,p) represents the distance between the feature vector of medical text and positive sample label text. d(a,n) represents the distance between the feature vector of medical text and negative sample label text. Margin represents the difference in the distance between the feature vectors of medical text and positive label text and negative label text.

7. The method for classifying children's medical text data according to claim 1, characterized in that: The specific process of step (5) is as follows: The encoder module is used to process all parent-class label texts and child-class text labels respectively, obtain the corresponding feature vector library and store it offline; the parent-class label offline feature vector library and the child-class label text offline feature vector library are linked according to the hierarchical correspondence, so as to quickly locate the child-class label offline feature vector library based on the parent-class label offline feature vector library.

8. The method for classifying children's medical text data according to claim 1, characterized in that: The specific process of step (6) is as follows: Prepare the medical text to be predicted, use the first encoder in the trained three-tower model to obtain the feature vector of the medical text to be predicted, measure its similarity with the parent class label in the parent class label offline feature vector library, obtain the probability of whether the text pairs are similar, sort them according to the similarity probability, and recall the subclass label offline feature vector library of the subclass label text corresponding to the parent class label with higher similarity; The feature vector of the medical text to be predicted is measured again with the offline feature vector library of the subclass label text to obtain the probability of whether the text pairs are similar. Then, the texts are sorted according to the similarity probability to obtain the top-k predicted label texts whose similarity values ​​are higher than the similarity threshold.

Images