Chinese medicine named entity recognition method based on MMoE multiple tasks

Abstract

The invention discloses a Chinese medicine named entity recognition method based on MMoE multiple tasks. Specifically, a text is input into a BERT model to obtain sequence features and overall semantic features; subsequently determining the number of tasks according to the number of entity categories and setting a plurality of experts; and finally, inputting the features into an MMoE-NER module to obtain a probability output matrix of each task. The model provided by the invention has the advantages that a plurality of expert models independent of a single model are provided, so that the problem caused by supervision of the single model is remarkably reduced, and the problem of fuzzy entity boundary information is solved by utilizing the ResNet-CNN; and subsequently, a negative sampling training learning mode is adopted for the entity to relieve the problem caused by data set label missing, and finally, the identification effect of the model on the entity is improved.

Description

Technical Field

[0001] The present application relates to the technical field of natural language processing and named entity recognition, and in particular to a method for Chinese medical named entity recognition based on MMoE multi-task. Background Art

[0002] As the digitization of the medical field continues to advance, the number and complexity of Chinese medical texts are also increasing. These texts describe human diseases, symptoms, treatments and prevention in detail and have a wide range of application value. They not only provide important learning and research materials for doctors, help formulate treatment plans and judge the severity of diseases, but also provide support for patients and improve their self-care and management capabilities.

[0003] Named entity recognition is an important cornerstone in the field of natural language processing. In modern medical work, extracting medical-related entities from medical texts is a key link. The entities involved in the text include but are not limited to human organs, disease names, symptom descriptions, medical examination indicators and treatment plans. By extracting these entities, a rich medical knowledge base and diagnosis and treatment database can be constructed to provide a basis and support for medical decision-making. Therefore, in order to effectively identify specific entities (such as symptoms, diseases, drugs and surgeries) from Chinese medical texts, Chinese medical named entity recognition (NER) came into being.

[0004] Although many supervised models have achieved promising performance, the models inevitably introduce incomplete and noisy annotations, which may mislead the model training process. Named entity recognition tasks usually contain multiple categories. It is difficult for a single model to identify the differences between multiple categories and the commonalities within the categories, which leads to the model failing to achieve good results. At the same time, the named entity recognition network based on the pointer network tends to ignore boundary information, resulting in repeated deviations in the extraction of nested entity scenarios. For example, when identifying the "symptom" category of "most coughs are coughs with sputum, and a few are dry coughs", the first "cough" and the second "cough" will be identified, but the second "cough" is not a separate entity, but a "cough with sputum" as a whole entity. Finally, the task of labeling named entity recognition datasets is difficult, and some datasets often have missing labels.

[0005] In order to solve the above problems, a Chinese medical named entity recognition method based on MMoE multi-task is proposed. The main contributions are summarized as follows:

[0006] 1) To address the problem that a single model cannot effectively identify inter-class and intra-class entities of various categories, a method is proposed that does not rely on a single model for NER prediction, but instead trains and integrates multiple expert models under the expectation maximization framework, and splits the entity recognition task model into a recognition multi-task model, thereby significantly alleviating the problems caused by single model supervision.

[0007] 2) In order to solve the problem of repeated deviation in the extraction of nested entity scenes in model entity recognition, the ResNet-CNN convolutional neural network is applied to solve the problem of blurred entity boundary information.

[0008] 3) In view of the problem of missing labels in named entity recognition datasets, a training and learning method of negative sampling of entities is proposed to alleviate the problem caused by missing labels in the dataset. Summary of the invention

[0009] The purpose of the present invention is to improve the effect of identifying words with specific meanings in Chinese medical texts, and to provide a basis and support for the patient's subsequent medical decision-making based on the identified medical entities.

[0010] In order to achieve the above objectives, a method for Chinese medical named entity recognition based on MMoE multi-task was invented. The specific model training steps are as follows:

[0011] 1) Obtain three public Chinese medical named entity recognition datasets. To ensure the rationality of experimental training data, use their respective official training sets and test sets;

[0012] 2) Preprocess the dataset for input into the BERT model to obtain CLS features and sequence features.

[0013] 3) Set the number of tasks m (the type of entity categories in the dataset), and according to the specific number of tasks, pass the CLS features of step 2) into the MMoE-NER module.

[0014] 4) The sequence features of step 2) are passed into the MMoE-NER module, and expert network training is performed according to the set number of experts n, so that each expert can learn the content of their own field to solve the problem that a single network cannot distinguish between classes and within classes well.

[0015] 5) In the expert network in step 4), the sequence features are passed through the ResNet network to obtain expert features to solve the problem of blurred boundary information.

[0016] 6) The CLS features in step 3) are passed through the expert weight gate allocation network to assign weights to each expert feature, so that each expert can dynamically contribute his or her understanding of different tasks.

[0017] 7) According to the number of tasks in step 3), the weighted features assigned by each expert obtained in step 6) are multiplied by the expert features obtained in step 5) and the multiple expert features after weight assignment are accumulated and passed into the corresponding number of tasks (number of categories).

[0018] 8) For each task, the features obtained in step 7) are learned using a bidirectional GRU to obtain weight matrices in two different directions.

[0019] 9) For each task, the inner product of the bidirectional matrix obtained in step 8) is calculated to finally obtain the upper triangular prediction entity (logits>0) of each task (category).

[0020] The overall training framework is shown in the attached figure of the specification. Figure 1 As shown. Specifically, the datasets in step 1) are CMeEE, CMeEE-V2 and CCKS2019, which are from the evaluation datasets of Task 1 of the 2022 and 2023 Chinese Medical Information Processing Challenge List and the evaluation dataset of Task 1 of the 2019 National Knowledge Graph and Semantic Conference, respectively. CMeEE has 15,000 training sets, 5,000 validation sets and 3,000 test sets, including 9 predefined entity types such as dis and bod. Compared with CMeEE, CMeEE-V2 fixes some annotation errors, increases the number of entities, and improves the quality of the corpus. CMeEE and CMeEE-V2 contain nested entities. CCKS2019 has 1,000 training sets and 379 test sets, including 6 predefined entity types such as disease and diagnosis, surgery, etc. CCKS2019 does not contain nested entities.

[0021] Step 2) Pass the text X of the training data into the BERT model to obtain the sequence feature H sent And the overall semantic label feature H cls The steps are as follows:

[0022] X=[x1,...,x i ,...,x n ]

[0023] H sent =[h1,...h i ,...,h n ]=BERT(X)[1:]

[0024] H cls =BERT(X)[0]

[0025] Where X represents the original text sequence of the input; BERT represents the general pre-trained model to be trained; H sent H represents the sequence features of the last layer output after the text X passes through BERT; clsRepresents the overall semantic label features of the last layer output after the text X passes through BERT;

[0026] Step 3) Set m tasks and obtain the sequence features H sent And the overall semantic label feature H cls The module structure is shown in the figure of the specification. Figure 2 shown.

[0027] Step 4) Design n expert networks in the MMoE-NER module, and let multiple different expert networks learn the differences between different categories and within categories respectively, so that each expert can learn their own field of focus, so as to solve the problem that a single network cannot distinguish between categories and within categories well.

[0028] Step 5) In a single expert network, the sequence feature H sent Pass into the i-th CNN convolutional network, where the height of the convolution kernel is equal to the sequence feature H sent The hidden layer dimension is 1000, so as to obtain the sequence features after convolution, so as to solve the problem of blurred boundary information; after obtaining the sequence features after convolution, the residual structure is introduced to convert the original sequence features H sent The expert output features are accumulated with the sequence features to prevent the problem of gradient backpropagation disappearing. Multiple expert networks will repeat this operation, the steps are as follows:

[0029] H cnn i =CNN i (H sent )

[0030] H resnet i =Add(H sent ,H cnn i )

[0031] Among them, i represents the i-th expert network; CNN i represents the convolutional neural network in the i-th expert network, H cnn i represents the H obtained in step 2) sent After the CNN in the i-th expert network i Output features of the module; Add means adding features; H resnet i Represents the final output features of the i-th expert network.

[0032] Step 6) In the MMoE-NER module, the H clsThe overall semantic features of the text are obtained through the jth expert linear network. The output dimension is kept consistent with the number of expert networks. The obtained features are passed through the SoftMax function to obtain the weight of each expert network's understanding of the jth task. This operation is repeated for multiple tasks. The steps are as follows:

[0033] H gate j =Dnn j (H cls )

[0034] Weight j =SoftMax(H gate j )

[0035] Among them, Dnn j represents the jth expert weight allocation network, H gate j Indicates H cls The overall semantic features of the text are obtained through the j-th expert linear network, Weight j Represents the weight of each expert network's understanding of the jth task.

[0036] Step 7) During the interaction, the expert output feature H obtained in step 5) is resnet i Multiply it with the weight distribution feature obtained in step 6) to get the feature output by the ith expert for the jth task, and then add up the features obtained by each expert to get the overall understanding semantic feature H of each expert for the jth task. mmoe j This operation is repeated for multiple tasks, and the steps are as follows:

[0037] H mmoe ij =Weight ij ·H resnet i

[0038]

[0039] Among them, Weight ij represents the weight of the ith expert’s understanding of the jth task; H mmoe ij H represents the specific output features of the i-th expert for the j-th task; mmoe j Represents the overall understanding semantic features of each expert on the jth task;

[0040] Step 8) In the jth task, H obtained in step 7 is mmoej Perform bidirectional semantic understanding to obtain two output features. This operation is repeated for multiple tasks. The steps are as follows:

[0041] H left j =GRU left j (H mmoe j )

[0042] H right j =GRU right j (H mmoe j )

[0043] Among them, GRU left j The temporal GRU model representing the jth task performs semantic understanding from left to right; GRU right j The temporal GRU model of the jth task performs semantic understanding from right to left; H left j Represents the output features obtained from left to right, H left j Represents the output features obtained from right to left;

[0044] Step 9) For task j, the output feature H obtained in step 8) is left j With H left j Perform matrix inner product to obtain a second-order matrix, where the value of the matrix is ​​the probability that a certain word and other words can form an entity; in the training phase, by randomly sampling negative samples, some missing entities will not be forced to be negative samples, thereby alleviating the problem of missing labels; in the inference phase, a matrix value greater than 0 indicates that the corresponding word-word is an entity. This operation will be repeated for multiple tasks, and the steps are as follows:

[0045] L(x,y)=max(0,-x)+max(0,y)

[0046] Out j =H left j ·H right j

[0047] Among them, x belongs only to the positive sample label and represents the output probability value corresponding to the positive sample; y belongs only to the negative sample label and represents the output probability value corresponding to the sampled negative sample; L(x,y) represents the loss generated between any two positive and negative sample pairs. The purpose of this loss is to make the output probability of the positive sample entity greater than 0 and the output probability of the negative sample entity less than 0; Out j The output matrix represents the probability that an entity composed of a certain word and other words belongs to category j, where a value greater than 0 indicates that the entity belongs to category j. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is the overall framework diagram of the network model of the present invention;

[0049] Figure 2 This is the multi-expert multi-task MMoE-NER training flow chart of the present invention.

Claims

1. A method for Chinese medical named entity recognition based on MMoE multi-task. The overall training framework is shown in Figure 1 of the accompanying drawings of the specification. The model training method includes the following steps: (1) Obtain three public Chinese medical named entity recognition datasets. To ensure the rationality of experimental training data, the official training and test sets of each dataset were used; (2) Preprocess the dataset for input into the BERT model to obtain CLS features and sequence features. (3) Set the number of tasks m (the number of entity categories in the dataset), and according to the specific number of tasks, pass the CLS features of step (2) into the MMoE-NER module. (4) The sequence features of step (2) are passed into the MMoE-NER module. According to the set number of experts n, the expert network training is performed so that each expert can learn the content of their own field to solve the problem that a single network cannot distinguish between classes and within classes well. (5) In the expert network in step (4), the sequence features are passed through the ResNet network to obtain expert features to solve the problem of blurred boundary information. (6) The CLS features in step (3) are passed through the expert weight gate allocation network to assign weights to each expert feature, so that each expert can dynamically contribute his or her understanding of different tasks. (7) According to the number of tasks in step (3), the weighted features assigned by each expert obtained in step (6) are multiplied by the expert features obtained in step (5), and the multiple expert features after weight assignment are accumulated and passed into the corresponding number of tasks (number of categories). (8) For each task, the features obtained in step (7) are learned using a bidirectional GRU to obtain weight matrices in two different directions. (9) For each task, the inner product of the bidirectional matrix obtained in step (8) is calculated to finally obtain the upper triangular prediction entity (logits>0) of each task (category).

2. According to the method of Chinese medical named entity recognition based on MMoE multi-task according to claim 1, it is characterized in that: The data sets in the step 1) are CMeEE, CMeEE-V2 and CCKS2019, which are respectively from the evaluation data sets of Task 1 of the Chinese Medical Information Processing Challenge List in 2022 and 2023 and the evaluation data sets of Task 1 of the National Knowledge Graph and Semantic Conference in 2019. CMeEE has 15,000 training sets, 5,000 validation sets and 3,000 test sets, including 9 predefined entity types such as dis and bod. Compared with CMeEE, CMeEE-V2 fixes some annotation errors, increases the number of entities, and improves the quality of the corpus. CMeEE and CMeEE-V2 contain nested entities. CCKS2019 has 1,000 training sets and 379 test sets, including 6 predefined entity types such as disease and diagnosis, surgery, etc. CCKS2019 does not contain nested entities.

3. According to the method of Chinese medical named entity recognition based on MMoE multi-task according to claim 1, it is characterized in that: In step 2), the text X of the training data is passed into the BERT model to obtain the sequence feature H sent And the overall semantic label feature H cls The steps are as follows: X=[x1,...,x i ,...,x n ] H sent =[h1,...h i ,...,h n ]=BERT(X)[1:] H cls =BERT(X)[0] Where X represents the original text sequence of the input; BERT represents the general pre-trained model to be trained; H sent H represents the sequence features of the last layer output after the text X passes through BERT; cls Represents the overall semantic label features of the last layer output after the text X passes through BERT.

4. According to the method of Chinese medical named entity recognition based on MMoE multi-task in claim 1, it is characterized in that: In step 3), m tasks are set, and the obtained sequence features H sent And the overall semantic label feature H cls The data is passed to the MMoE-NER module, and the structure of the module is shown in Figure 2 of the attached drawing of the specification. In the step 4), n expert networks are designed in the MMoE-NER module, and multiple different expert networks are allowed to learn the differences between different categories and within categories respectively, so that each expert learns the field he or she focuses on, so as to solve the problem that a single network cannot distinguish the differences between categories and within categories well.

5. The method for Chinese medical named entity recognition based on MMoE multi-task according to claim 1, characterized in that: In step 5), in a single expert network, the sequence feature H sent Pass into the i-th CNN convolutional network, where the height of the convolution kernel is equal to the sequence feature H sent The hidden layer dimension is 1000, so as to obtain the sequence features after convolution, so as to solve the problem of blurred boundary information; after obtaining the sequence features after convolution, the residual structure is introduced to convert the original sequence features H sent The expert output features are accumulated with the sequence features to prevent the problem of gradient backpropagation disappearing. Multiple expert networks will repeat this operation, the steps are as follows: H cnn i =CNN i (H sent ) H resnet i =Add(H sent ,H cnn i ) Among them, i represents the i-th expert network; CNN i represents the convolutional neural network in the i-th expert network, H cnn i represents the H obtained in step 2) sent After the CNN in the i-th expert network i Output features of the module; Add means adding features; H resnet i Represents the final output features of the i-th expert network.

6. The method for Chinese medical named entity recognition based on MMoE multi-task according to claim 1, characterized in that: In the step 6), in the MMoE-NER module, the H cls The overall semantic features of the text are obtained through the jth expert linear network. The output dimension is kept consistent with the number of expert networks. The obtained features are passed through the SoftMax function to obtain the weight of each expert network's understanding of the jth task. This operation is repeated for multiple tasks. The steps are as follows: H gate j =Dnn j (H cls ) Weight j =SoftMax(H gate j ) Among them, Dnn j represents the jth expert weight allocation network, H gate j Indicates H cls The overall semantic features of the text are obtained through the j-th expert linear network, Weight j Represents the weight of each expert network's understanding of the jth task.

7. The method for Chinese medical named entity recognition based on MMoE multi-task according to claim 1, characterized in that: In the step 7), during the interaction process, the expert output feature H obtained in step 5) is resnet i Multiply it with the weight distribution feature obtained in step 6) to get the feature output by the ith expert for the jth task, and then add up the features obtained by each expert to get the overall understanding semantic feature H of each expert for the jth task. mmoe j This operation is repeated for multiple tasks, and the steps are as follows: H mmoe ij =Weight ij ·H resnet i Among them, Weight ij represents the weight of the ith expert’s understanding of the jth task; H mmoe ij H represents the specific output features of the i-th expert for the j-th task; mmoe j Represents the overall understanding semantic features of each expert on the jth task.

8. The method for Chinese medical named entity recognition based on MMoE multi-task according to claim 1, characterized in that: In step 8), in the jth task, H obtained in step 7 is mmoe j Perform bidirectional semantic understanding to obtain two output features. This operation is repeated for multiple tasks. The steps are as follows: H left j =GRU left j (H mmoe j ) H right j =GRU right j (H mmoe j ) Among them, GRU left j The temporal GRU model representing the jth task performs semantic understanding from left to right; GRU right j The temporal GRU model of the jth task performs semantic understanding from right to left; H left j Represents the output features obtained from left to right, H left j Represents the output features obtained from right to left.

9. The method for Chinese medical named entity recognition based on MMoE multi-task according to claim 1, characterized in that: In step 9), for task j, the output feature H obtained in step 8) is left j With H left j Perform matrix inner product to obtain a second-order matrix, where the value of the matrix is ​​the probability that a certain word and other words can form an entity; In the training phase, by randomly sampling negative samples, some entities with missing labels will not be forced to be negative samples, thus alleviating the problem of missing labels; in the inference phase, the value of the matrix greater than 0 indicates that the corresponding word-word is an entity. This operation will be repeated for multiple tasks, and the steps are as follows: L(x,y)=max(0,-x)+max(0,y) Out j =H left j ·H right j Among them, x belongs only to the positive sample label and represents the output probability value corresponding to the positive sample; y belongs only to the negative sample label and represents the output probability value corresponding to the sampled negative sample; L(x,y) represents the loss generated between any two positive and negative sample pairs. The purpose of this loss is to make the output probability of the positive sample entity greater than 0 and the output probability of the negative sample entity less than 0; Out j The output matrix represents the probability that an entity composed of a certain word and other words belongs to category j, where a value greater than 0 indicates that the entity belongs to category j.

Images