A medical text professional classification method based on adversarial data enhancement

Abstract

The present invention relates to the field of medical text professional classification, and discloses a medical text professional classification method based on adversarial data enhancement, which preprocesses medical text data; improves the quality of the medical text data set based on the adversarial attack data enhancement method; constructs the preprocessed medical text into a word vector matrix, which serves as the input of a convolutional neural network or is adjusted to a format required by a training model as the input of a pre-training model; and based on a classification layer of key text probability information, the medical text professional classification method based on adversarial data enhancement effectively alleviates the problems of insufficient data volume, poor quality and category imbalance faced by the medical text professional classification data set. Since the present invention adopts multi-step adversarial attacks to generate enhanced samples, the generated enhanced data is not only simply similar to the original data, but also has the characteristic of covering a larger model decision space.

Description

Technical Field

[0001] The present invention relates to the field of medical text professional classification, and specifically to a medical text professional classification method based on adversarial data enhancement. Background Art

[0002] In recent years, deep neural networks (DNNs) have achieved significant success in classification tasks across various fields, such as computer vision, network anomalies, medical text, and images. DNNs have also been applied to electronic health record (EHR) systems. Their widespread adoption has made medical specialty classification even more crucial in modern healthcare. Categorizing clinical notes into medical specialty fields, such as orthopedics, ophthalmology, and endocrinology, can improve EHR retrieval efficiency, enabling physicians to quickly locate targeted information. Furthermore, automatically categorizing clinical notes into the correct medical subfield can effectively expand downstream applications of medical specialty. For example, clarifying medical subfields facilitates the extraction of medical knowledge and effectively supports intelligent medical decision-making systems. However, acquiring and labeling medical text stored in unstructured text is challenging. For medical specialty classification tasks, the unique characteristics of different medical subfields, such as diverse physician writing styles, the sensitivity of textual information, and the probability of disease outbreaks, lead to insufficient, poor, and imbalanced data sets in medical text. These dataset issues further contribute to low classification accuracy.

[0003] The following are existing patents and solutions:

[0004] For example, the patent "A hierarchical multi-label medical problem classification method based on CNN-DBN" (application number 201910490855.X) constructs a hierarchical multi-label medical problem classification method by adding a label classifier DBN to CNN (convolutional neural network);

[0005] In the paper “Medical subdomain classification of clinical notes using a machine learning-based natural language processing approach” (Weng WH, Wagholikar KB, Mccray AT, et al. Medical subdomain classification of clinicalnotes using a machine learning-based natural language processing approach[J]. Bmc Medical Informatics&Decision Making, 2017, 17(1): 155.), the authors built a machine learning-based natural language processing (NLP) pipeline and developed a medical subdomain classifier based on medical record content; other works focused on improving the problems existing in the dataset.

[0006] The patent "A method and device for data enhancement" (application number CN202010648077.5) uses the GT2 model to generate corpus in a specific medical field for data expansion to solve the problems of insufficient data volume, poor quality and imbalance in medical data sets.

[0007] Among the above technologies for solving medical professional classification:

[0008] While the first technique improved the accuracy of using a CNN as the classification model alone, the classification results of the CNN-DBN hierarchical classifier, which integrates the label classifier, were not ideal. This is because medical texts contain specialized vocabulary that only exists in specialized fields, which is not fully utilized by the CNN-DBN hierarchical classifier.

[0009] The second technology combines multiple machine learning models, and the constructed classifier is not limited to electronic medical records. However, it still faces problems such as insufficient data volume, poor quality and class imbalance during training.

[0010] The third technology is data augmentation technology for medical texts. Although the advanced GT2 model is used as a corpus generator to enrich the training set by generating new data close to the original data, the newly added corpus, although similar to the original sample, ignores the distribution of the data in the model decision space.

[0011] To this end, we propose a medical text professional classification method based on adversarial data enhancement. Summary of the Invention

[0012] (1) Technical problems solved

[0013] In response to the shortcomings of the existing technology, the present invention provides a medical text professional classification method based on adversarial data enhancement, which solves the problems of insufficient data volume, poor quality and category imbalance in medical datasets, and improves the accuracy of medical text professional classification tasks.

[0014] (2) Technical solution

[0015] To achieve the above-mentioned objectives, the present invention provides the following technical solution: a medical text professional classification method based on adversarial data enhancement, comprising the following steps:

[0016] Step 1: Preprocess medical text data;

[0017] Step 2: Improve the quality of medical text datasets based on adversarial attack data augmentation methods;

[0018] Step 3: Construct the pre-processed medical text into a word embedding matrix as the input of the convolutional neural network or adjust it to the format required by the training model as the input of the pre-training model;

[0019] Step 4: Based on the classification layer of key text probability information, the appearance of professional words in medical texts is often associated with the characteristics of professional fields, and the classification results of the neural network are further adjusted.

[0020] Preferably, the anti-attack data enhancement method in the second step includes the following steps:

[0021] Ranking of word importance;

[0022] Combine candidate words;

[0023] Build a data augmentation example.

[0024] Preferably, the importance ranking of the words includes the following:

[0025] The embedding vector difference of the replacement word is calculated as [MASK], and the importance of the word is measured by projecting the vector difference in the gradient direction. For each word w i The importance in x is calculated as I(w i ,x);

[0026]

[0027] Among them, V [MASK] For the embedding of [MASK], For word w i The embedding of , J is the loss function of model F.

[0028] Preferably, the combination candidate words includes the following steps:

[0029] Using the pre-trained word vector model Glove, for each w in the corpus i Determine the synonym set Syn wi , based on the cosine similarity between w and each word in the synset, w from WordNet i Select the k closest words from the synonym set, and after determining the candidate words for each word, initialize a replacement set CandiSet for the word to be disturbed. For each attack action, select the first t words from the sorted set W as the disturbed word set PerSet = w i ,...,w i+t , where i is the i-th attack action.

[0030] Preferably, the constructing data enhancement example includes the following:

[0031] The words in PerSet are replaced sequentially with the candidate word combinations in CandiSet to generate the disturbed examples x′.

[0032] Preferably, in the classification layer based on the probability information of the key text, a method for recalculating the probability distribution is designed, comprising the following steps:

[0033] Calculate part-of-speech importance;

[0034] Estimate the category tendency of the input sample;

[0035] Recalculation of probability distributions.

[0036] Preferably, the calculation of word class importance includes the following:

[0037] The calculation formula of WCI is set as:

[0038]

[0039] where |D y | represents the total number of examples with label y in the dataset, is the average amount of data for all categories k, b is the coordination factor, which is a constant. IDF′ is a variant of inverse document frequency and is expressed as:

[0040]

[0041] The control coefficient a is a constant between 0 and 1, which is used to limit The contribution of |D|, and |D y | are the number of all texts in the dataset, including wi The number of texts containing w i The number of texts with the label y and the number of texts containing the label y. TF′ is the category frequency of a word, expressed as:

[0042]

[0043] in, is the total number of times word wi appears in all texts with label y, C y is the total number of words in all examples with label y.

[0044] Preferably, estimating the category tendency of the input sample includes the following:

[0045] For any input x, we measure whether it belongs to any category y based on the category importance of all words in x i The tendency of is expressed as:

[0046]

[0047] Preferably, the recalculation of the probability distribution includes the following:

[0048] The Softmax output of the target model is a normalized logit distribution, represented as {z i |i∈c,∑z i =1}, where z i represents the output of the i-th node, c represents the number of categories, and after the probability information layer, the output of each node is:

[0049]

[0050] Where M = max(z i Score(x,y j )).

[0051] (3) Beneficial effects

[0052] Compared with the existing technology, the present invention provides a medical text professional classification method based on adversarial data enhancement, which has the following beneficial effects:

[0053] 1. This medical text professional classification method based on adversarial data enhancement effectively alleviates the problems of insufficient data volume, poor quality and category imbalance faced by medical text professional classification datasets. Since the present invention uses multi-step adversarial attacks to generate enhanced samples, the generated enhanced data is not only simply similar to the original data, but also covers a larger model decision space.

[0054] 2. This medical text professional classification method based on adversarial data enhancement can improve the performance of medical text professional classification tasks. Due to the data enhancement method adopted in the present invention and the classification layer based on key text probability information, the output of the neural network can be more reasonably adjusted according to the characteristics of the medical text, thereby improving the performance of the classification task. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 It is a schematic diagram of the process of the present invention;

[0056] Figure 2 This is a flowchart of the data enhancement part;

[0057] Figure 3 The figure is a schematic diagram of the principle flow of an example of a multi-step attack method using BioBERT as a model. DETAILED DESCRIPTION

[0058] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0059] See also Figure 1-3 ,A medical text professional classification method based on adversarial data enhancement, comprising the following steps:

[0060] Step 1: First, preprocess the medical text. Preprocessing operations generally include: cleaning the medical text, segmenting the words, removing special symbols, modifying the text format, and processing stop words. If it is a pre-trained model such as BERT, it is also necessary to add operations such as padding data and add methods to enhance data against attacks to improve the quality of the medical text dataset, thereby increasing the number of training sets and the coverage of data in the model decision space.

[0061] During the data augmentation process, samples for data augmentation are determined through a multi-step adversarial attack method.

[0062] Specifically, it includes three steps: ranking the importance of words, combining candidate words, and constructing data enhancement examples.

[0063] 1. Order of importance of words. For any input x=[w1,w2,...,w m], since each word has a different impact on the final prediction result. Therefore, we sort the importance of all words and prioritize attacking words with high importance. Calculating the difference in confidence resulting from deleting a word is a common method for comparing word importance. This method requires accessing the target model m times, which is computationally expensive. To improve computational efficiency, we calculate the embedding vector difference of the replaced word as [MASK] and measure the importance of the word by projecting the vector difference in the gradient direction. Each word w i The importance in x is calculated as I(w i ,x).

[0064]

[0065] Among them, V [MASK] For the embedding of [MASK], For word w i where is the embedding of and J is the loss function of model F. Only a single query to the model is required to obtain the scores for all words, greatly improving efficiency. Further filtering out stop words from the NLTK and Spacy libraries, such as "the," "then," and "...", yields a sorted and filtered set of important words, W.

[0066] 2. Combine candidate words. Using the pre-trained word vector model Glove, we combine candidate words for each word in the corpus. i Determine the synonym set Syn wi , based on the cosine similarity between w and each word in the synset, w from WordNet i Select the k closest words from the synonym set. WordNet is a semantically oriented English dictionary with 155,287 words and 117,659 synonyms. Human-written medical notes are not perfect and contain some syntactic errors, so we do not need to require the generated enhanced samples to be perfect. After determining the candidate words for each word, initialize a replacement set CandiSet for the word to be disturbed. For each attack action, select the first t words from the sorted set W as the disturbed word set PerSet = w i ,...,w i+t , where i is the i-th attack action. Each disrupted word and each word has k candidate words, so all possible combinations have k n To save the cost, we randomly take r = k × t examples, reducing the number of combinations with exponential complexity to a constant.

[0067] 3. Construct data augmentation examples. We sequentially replace words in the PerSet with candidate word combinations in the CandiSet to generate perturbed examples x′. If the predicted probability of x′ for the original label y decreases, we add x′ to the final augmentation set. It is important to note that the prediction result of example x′ cannot change; if it does, it will not be added to the augmentation set.

[0068] Step 2: Construct the preprocessed medical text into a word vector matrix as the input of the convolutional neural network or adjust it to the format required by the training model as the input of the pre-training model. Add a classification layer based on key text probability information after the neural network. By taking advantage of the fact that the appearance of professional words in medical texts is often associated with the characteristics of professional fields, the classification results of the neural network can be further adjusted to improve the accuracy of professional classification of medical texts.

[0069] In the classification layer based on probabilistic information about key text, we leveraged the fact that specialized vocabulary in medical text often corresponds to specific fields. We designed a layer that recalculates the probability distribution and incorporates knowledge from probabilistic statistics to improve model prediction accuracy. This layer, added after the Softmax layer, essentially involves three steps: calculating word class importance, estimating the category bias of input samples, and recalculating the probability distribution.

[0070] 1. We proposed the concept of word class importance (WCI) to indicate the relevance of different nouns to different medical specialties. Referring to the BM25 algorithm in information retrieval, we set the calculation formula of WCI as

[0071]

[0072] where |D y | represents the total number of examples with label y in the dataset. is the average data size of all categories k, b is the coordination factor, which is a constant. IDF′ is a variant of inverse document frequency and is expressed as

[0073]

[0074] The control coefficient a is a constant between 0 and 1, which is used to limit Contribution. |D|, and |D y | are the number of all texts in the dataset, including w i The number of texts containing w i The number of texts with label y and the number of texts containing label y. TF′ is the category frequency of a word, expressed as

[0075]

[0076] in, is the total number of times word wi appears in all texts with label y, C y is the total number of words in all examples with label y.

[0077] 2. Estimate the category tendency of the input sample. For any input x, we measure whether it belongs to any category y based on the category importance of all words in x i The tendency of

[0078]

[0079] 3. Recalculation of probability distribution. The Softmax output of the target model is the normalized logit distribution, expressed as {z i |i∈c,∑z i =1}, where z i represents the output of the i-th node, and c represents the number of categories. After the probability information layer, the output of each node is

[0080]

[0081] Where M = max(z i Score(x,y j )), used to prevent numerical overflow.

[0082] Step 3: The output of the neural network is the confidence level of the medical text corresponding to different fields, where the label with the highest confidence level can be considered as the professional field of the medical text.

[0083] In the figure, 101 is the medical text data preprocessing stage, which removes special symbols from unprocessed medical electronic medical records and other contents, and performs operations such as paragraph merging to obtain pure text content; 102 is the adversarial attack data enhancement part in the present invention, which obtains samples that can cover more model decision spaces through multi-step adversarial attacks; 103 is the combination of original data and enhanced samples into a new training set, and the training set is adjusted to the format required for neural network input; 104 is the neural network model used in the present invention, which can be a deep convolutional neural network CNN, a recurrent neural network LSTM, a pre-trained model BERT and other modules; 105 is the classification layer based on key text probability information in the present invention, which can adjust the output probability distribution of the neural network model according to information such as the frequency of professional vocabulary in the text; 106 is the output medical professional category after adjustment.

[0084] Figure 2 This is a flow chart of the data enhancement part of the present invention.

[0085] In the figure, 201 represents the importance ranking of words in the text. When perturbing an attack, the most important words are selected for the most likely outcome of high-quality augmented samples, so it is necessary to rank the words in the text by importance. 202 represents candidate combinations for important words. When selecting important words, more than one word replacement is performed, so words that could potentially cause perturbations are randomly combined. 303 represents a constructed data augmentation example. This constructed perturbation text is fed into the model. If the score decreases compared to the original label but the output remains unchanged, it is considered a valid data augmentation example.

[0086] Figure 3 This is a principle flow chart of an example of the present invention using BioBERT as a model and multi-step attack as a counter-attack method.

[0087] In the figure, 301 is the medical text data cleaning, which removes special symbols from the unprocessed medical electronic medical records and performs operations such as paragraph merging to obtain pure text content; 302 is the use of multi-step attack as an attack method, selecting

[0088] Interference text that meets the requirements is used as enhanced data; 303 is the operation of segmenting the new text training set and adjusting it to the input format required by BioBERT; 304 is the most commonly used pre-training model BioBERT in the medical field used in this example; 305 is the output result of the BioBERT pre-training model directly to the fully connected layer and then the initial probability distribution of the input sample is obtained through the Softmax layer; 306 is the classification layer based on key text probability information proposed in the present invention, which uses the output of the Softmax layer as the input of this layer, and recalculates the initial probability distribution according to information such as the frequency of professional vocabulary to obtain the final probability distribution.

[0089] Specifically, taking a medical professional classification dataset from Kaggle as an example, after removing special symbols and other processing, the original data in the dataset is used as the original data for data augmentation. For example, an ophthalmology text note "In external blepharitis, the incision should be on the skin surface, parallel to the lid margin; in internal blepharitis, the incision should be on the conjunctival surface, perpendicular to the margin" is labeled as ophthalmology with a confidence level of 92%. By calculating the I(w i,x) score to obtain several important words in the text as candidate attack words, such as skin, external and incision, and use the Glove model to calculate the synonyms of these words with smaller cosine similarity in the embedding space. According to the synonyms of these words, a candidate word dictionary is generated, such as the dictionary [surface, face...][outer,ectropion...][cut,incident...]. These words are randomly combined and replaced in the original sample as candidate interference samples, such as interference sample 1 "In outer blepharitis, the incision should be on the sueface surface, parallel to the lid margin; in internal blepharitis, the incision should be on the conjunctival surface, perpendicular to the margin" and interference sample 2 "In ectropionblepharitis, the cut should be on the sueface surface, parallel to the lidmargin; in internal blepharitis, the cut should be on the conjunctival surface, perpendicular to the margin". When perturbation sample 1 is fed into the model, the output is still ophthalmology with a confidence level of 71%. Therefore, perturbation sample 1 can be used as augmentation data. When perturbation sample 2 is fed into the model, the output is nursing with a confidence level of 53%. Therefore, perturbation sample 2 is not used as augmentation data. By randomly combining the perturbations, a large number of augmentation samples can be obtained to expand the original training set. Taking BioBERT as an example, the classification model was trained with dropout set to 0.1, epochs set to 10, maximum sequence length to 512, batch size to 16, and learning rate to 0.00001. The trained model was used for the medical text classification task.

[0090] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A medical text professional classification method based on adversarial data enhancement, characterized by: The following steps are involved: Step 1: Preprocess medical text data; Step 2: Improve the quality of medical text datasets based on adversarial attack data augmentation methods; Step 3: Construct the pre-processed medical text into a word embedding matrix as the input of the convolutional neural network or adjust it to the format required by the training model as the input of the pre-training model; Step 4: Based on the classification layer of key text probability information, the occurrence of professional terms in medical texts is used to associate with the characteristics of professional fields to further adjust the classification results of the neural network; The second step of the data enhancement method based on the anti-attack method includes the following steps: Ranking of word importance; Combine candidate words; Constructing data augmentation examples; The importance ranking of the words includes the following: The embedding vector difference of the replacement word is calculated as [MASK], and the importance of the word is measured by projecting the vector difference in the gradient direction. The importance in x is calculated as ; ； in, For the embedding of [MASK], For words The embedding of , x represents any input.

2. The method for medical text professional classification based on adversarial data enhancement according to claim 1, characterized in that: The combination candidate words comprises the following steps: Using the pre-trained word vector model Glove, for each w in the corpus i Determine the synonym set Syn wi , based on the cosine similarity between w and each word in the synset, w from WordNet i Select the k closest words from the synonym set, and after determining the candidate words for each word, initialize a replacement set CandiSet for the word to be disturbed. For each attack action, select the first t words from the sorted set W as the disturbed word set PerSet = w i ,...,w i+t , where i is the i-th attack action.

3. The method for professional medical text classification based on adversarial data enhancement according to claim 1, characterized in that: The data augmentation example includes the following: The words in PerSet are replaced sequentially with the candidate word combinations in CandiSet to generate the disturbed examples x′.

4. The method for professional medical text classification based on adversarial data enhancement according to claim 1, characterized in that: In the classification layer based on the probability information of key texts, a method for recalculating the probability distribution is designed, which includes the following steps: Calculate part-of-speech importance; Estimate the category tendency of the input sample; Recalculation of probability distributions.

5. The method for professional medical text classification based on adversarial data enhancement according to claim 4, characterized in that: The calculation of word class importance includes the following: The calculation formula of WCI is set as: ； in represents the total number of examples with label y in the dataset, is the average amount of data for all categories k, b is the coordination factor, which is a constant. IDF′ is a variant of inverse document frequency and is expressed as: ； The control coefficient a is a constant between 0 and 1, which is used to limit Contribution, 、 、 and The number of all texts in the dataset, The number of texts containing w i The number of texts with label y and the number of texts containing label y, TF′ is the category frequency of a word, expressed as: ； in, is the total number of times word wi appears in all texts with label y, is the total number of words in all examples with label y.

6. The method for professional medical text classification based on adversarial data enhancement according to claim 4, characterized in that: The estimation of the category tendency of the input sample includes the following: For any input x, we measure whether it belongs to any category based on the category importance of all words in x The tendency of is expressed as: 。 7. The method for professional medical text classification based on adversarial data enhancement according to claim 6, characterized in that: The recalculation of the probability distribution includes the following: The softmax output of the target model is a normalized logit distribution, expressed as ,in represents the output of the i-th node, c represents the number of categories, and after the probability information layer, the output of each node is: ； in .

Images