Method, system, device and storage medium for establishing named entity recognition model

Abstract

The present disclosure provides a method for establishing a named entity recognition model, comprising the following steps: S1: using a training set and a development set to perform model training, generating a benchmark model, and using the benchmark model to label unlabeled data to obtain labeled data, and using the labeled data as a standard answer; S2: based on an acquisition queue system, distributing the unlabeled data in a queue space in sequence; S3: adding pseudo data other than the labeled data in the queue space to the training data used for model training to perform model training; selecting the selected queue data that has the most obvious performance improvement on the benchmark model as the optimal data to add to the training set, and removing the selected queue data from the unlabeled data; S4: looping steps S1 to S3 until the model performance is no longer improved or the unlabeled data is used up.

Description

Technical Field

[0001] The present disclosure relates to the technical field of entity recognition, and in particular to a method for establishing a named entity recognition model, a system, a recognition method, an electronic device, and a storage medium. Background Art

[0002] One of the difficulties in identifying named entities in electronic medical records is the scarcity of labeled data. Among these data, there are very few labeled data, and more are large amounts of unlabeled data.

[0003] In the prior art, in named entity recognition tasks, methods such as replacing words with the same label, replacing synonyms, and randomly injecting noise are usually used to enhance the data of the annotated text. However, in the medical field, electronic medical records contain the patient's real disease information and the doctor's professional diagnosis and treatment process, which are extremely rigorous and professional. The use of traditional data enhancement methods will lead to errors or loss of medical information, and it is impossible to accurately perform medical named entity recognition. Summary of the invention

[0004] In order to solve at least one of the above technical problems, the present disclosure proposes a medical named entity recognition method based on a method, which makes full use of unlabeled data to generate high-quality annotated data, improves the utilization rate of plain text of electronic medical records, expands the training set data and thereby improves the performance of the medical named entity recognition model.

[0005] According to one aspect of the present disclosure, a method for establishing a named entity recognition model is provided, comprising the following steps:

[0006] S1: Use the training set and the development set to train the model, generate a benchmark model, and use the benchmark model to annotate the unlabeled data to obtain the annotated data, which is used as the standard answer;

[0007] S2: Based on the acquisition queue system, the unlabeled data is distributed in the queue space in sequence;

[0008] S3: Add the pseudo data in the queue space to the training data used for model training to perform model training; select the selected queue data that has the most obvious improvement on model performance as the optimal data to be added to the training set, and remove the selected queue data from the unlabeled data;

[0009] S4: Repeat steps S1 to S3 until the model performance no longer improves or the unlabeled data is used up.

[0010] According to at least one embodiment of the present disclosure, the method for establishing a named entity recognition model includes:

[0011] S21: Put all untagged data into queue 0, and set the other queues to empty;

[0012] S22: Select a specific number n of single-round models in the model training process of step S1;

[0013] S23: For each single-round model, the annotation result of the single-round model is compared with the annotation answer of the benchmark model, and the parameter difference between the annotation data of the single-round model and the standard answer is compared in each cycle, and the parameters include entity boundaries, entity categories and the number of entities; the queue data is placed at intervals, and if they are completely consistent, they are promoted to a higher queue, otherwise they are demoted to a lower queue, and returned to the queue system after the cycle is completed.

[0014] According to the method for establishing a named entity recognition model of at least one embodiment of the present disclosure, in step S23, an absolute mode is used to judge the entities annotated by the model. When the queue data is used to compare the annotation results with the optimal model, only when the entities annotated by the model in two sentences are completely consistent in number, boundaries and types, are they judged as data with the same annotations and can be promoted to a higher-level queue, otherwise they are demoted to a lower-level queue.

[0015] According to the method for establishing a named entity recognition model of at least one embodiment of the present disclosure, in steps S3 and S4, whether the model performance is improved is determined by comparing the harmonic mean F value:

[0016] F=(2×P×R)×100% / (P+R);

[0017] P is the accuracy:

[0018] P = number of correctly identified entities × 100% / number of identified entities;

[0019] R is the recall rate:

[0020] R = number of correctly identified entities x 100% / number of correct entities.

[0021] According to the method for establishing a named entity recognition model of at least one embodiment of the present disclosure, a higher harmonic mean F value indicates better model performance.

[0022] According to a method for establishing a named entity recognition model according to at least one embodiment of the present disclosure, the named entity recognition model is used for medical electronic medical record entity recognition.

[0023] According to the method for establishing a named entity recognition model of at least one embodiment of the present disclosure, the named entity recognition model uses a WC-LSTM model.

[0024] According to another aspect of the present disclosure, a system for establishing a named entity recognition model is provided, comprising:

[0025] A benchmark model generation module, wherein the benchmark model generation module generates a benchmark model using a training set and a development set;

[0026] A standard answer generation module, wherein the standard answer generation module uses the benchmark model generated by the benchmark model generation module to annotate the unlabeled data, obtain the annotated data, and generate a standard answer;

[0027] The queue system acquisition module distributes the unlabeled data in the queue space in sequence based on the acquisition queue system;

[0028] The optimal data selection module, the queue system acquisition module adds the pseudo data in the queue space to the training data for model training to perform model training; selects the selected queue data that has the most obvious improvement on model performance as the optimal data to be added to the training set, and removes the selected queue data from the unlabeled data;

[0029] An operation control module controls the benchmark model generation module, the standard answer generation module, the queue system acquisition module and the optimal data selection module to run in a loop until the model performance is no longer improved or the unlabeled data is used up.

[0030] According to another aspect of the present disclosure, a method for named entity recognition is provided, comprising the following steps:

[0031] S100: Obtain training set, development set, unlabeled data, and data to be labeled;

[0032] S200: Using the training set, the development set, and the unlabeled data, an entity recognition model is established by using the method for establishing a named entity recognition model of any embodiment of the present disclosure;

[0033] S300: Input the data to be annotated into the entity recognition model to obtain annotated data.

[0034] According to another aspect of the present disclosure, there is provided an electronic device, including:

[0035] A memory storing execution instructions;

[0036] A processor, wherein the processor executes the execution instructions stored in the memory, so that the processor executes the method for establishing a named entity recognition model as in any embodiment of the present disclosure and / or the method for named entity recognition as in any embodiment of the present disclosure.

[0037] According to another aspect of the present disclosure, a computer storage medium is provided, which stores computer execution instructions, wherein the computer execution instructions are used to execute the method for establishing a named entity recognition model of any embodiment of the present disclosure and / or the named entity recognition method of any embodiment of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings illustrate exemplary embodiments of the present disclosure and together with the description serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.

[0039] Figure 1 It is a flowchart of a method for establishing a named entity recognition model according to an embodiment of the present invention.

[0040] Figure 2 It is a schematic diagram of a processing flow based on an acquisition queue system according to an embodiment of the present disclosure.

[0041] Figure 3 It is a flowchart of a method for named entity recognition according to a disclosed embodiment.

[0042] Figure 4 The invention is a block diagram showing the structure of an apparatus for generating a named entity recognition model using a hardware implementation of a processing system.

[0043] Description of Reference Numerals

[0044] 1000 Device for generating named entity recognition model

[0045] 1001 Benchmark Model Generation Module

[0046] 1002 Standard answer generation module

[0047] 1003 Queue system acquisition module

[0048] 1004 Optimal Data Selection Module

[0049] 1005 Operation Control Module

[0050] 1100 Bus

[0051] 1200 processor

[0052] 1300 Memory

[0053] 1400 Other circuits. DETAILED DESCRIPTION

[0054] The present disclosure is further described in detail below in conjunction with the accompanying drawings and implementations. It is understood that the specific implementations described herein are only used to explain the relevant content, rather than to limit the present disclosure. It should also be noted that, for ease of description, only the parts related to the present disclosure are shown in the accompanying drawings.

[0055] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in the present disclosure can be combined with each other. The technical solution of the present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0056] Unless otherwise specified, the exemplary embodiments / embodiments shown will be understood as providing exemplary features of various details of some ways in which the technical concept of the present disclosure can be implemented in practice. Therefore, unless otherwise specified, the features of the various embodiments / embodiments can be combined, separated, interchanged and / or rearranged without departing from the technical concept of the present disclosure.

[0057] The use of cross-hatching and / or shading in the accompanying drawings is generally used to make the boundaries between adjacent components clear. As such, unless otherwise specified, the presence or absence of cross-hatching or shading does not convey or indicate any preference or requirement for the specific materials, material properties, dimensions, proportions, commonalities between the components shown, and / or any other characteristics, attributes, properties, etc. of the components. In addition, in the accompanying drawings, the sizes and relative sizes of the components may be exaggerated for clarity and / or descriptive purposes. When the exemplary embodiments can be implemented differently, the specific process sequence can be performed in a different order than described. For example, two successively described processes can be performed substantially simultaneously or in an order opposite to the described order. In addition, the same figure numbers represent the same components.

[0058] When a component is referred to as being "on" or "over," "connected to," or "coupled to" another component, the component may be directly on, directly connected to, or directly coupled to the other component, or intervening components may be present. However, when a component is referred to as being "directly on," "directly connected to," or "directly coupled to" another component, there are no intervening components. For this purpose, the term "connected" may refer to a physical connection, an electrical connection, etc., with or without intervening components.

[0059] The terms used herein are for the purpose of describing specific embodiments, and are not intended to be restrictive. As used herein, unless the context clearly indicates otherwise, the singular forms "one (kind, person)" and "said (the)" are also intended to include plural forms. In addition, when the terms "comprise" and / or "include" and their variations are used in this specification, it is explained that there are stated features, integral bodies, steps, operations, parts, assemblies and / or their groups, but it is not excluded that there are or add one or more other features, integral bodies, steps, operations, parts, assemblies and / or their groups. It should also be noted that, as used herein, the terms "substantially", "approximately" and other similar terms are used as approximate terms and not as degree terms, so that they are used to explain the inherent deviations of the measured values, calculated values ​​and / or the values ​​provided that will be recognized by those of ordinary skill in the art.

[0060] Figure 1 It is a flowchart of a method for establishing a named entity recognition model according to an embodiment of the present disclosure.

[0061] The named entity recognition model disclosed in the present invention can make full use of unlabeled data to generate high-quality annotated data, improve the utilization rate of electronic medical record plain text, expand the data of the training set and thus improve the performance of the medical named entity recognition model.

[0062] like Figure 1 As shown, the method for establishing the named entity recognition model of the present disclosure includes:

[0063] S1: Use the training set and the development set to train the model, generate a benchmark model, and use the benchmark model to annotate the unlabeled data to obtain the annotated data, which is used as the standard answer;

[0064] S2: Based on the acquisition queue system, the unlabeled data is distributed in the queue space in sequence;

[0065] S3: Add the pseudo data other than the labeled data in the queue space to the training data used for model training to perform model training; select the selected queue data that has the most obvious improvement on the model performance as the optimal data to be added to the training set, and remove the selected queue data from the unlabeled data;

[0066] S4: Repeat steps S1 to S3 until the model performance no longer improves or the unlabeled data is used up.

[0067] The training set and the development set are used to perform model training to generate a benchmark model, and a neural network model can be used for model training, such as a WC-LSTM model in the neural network model.

[0068] Select the model that performs best on the development set bestAs a baseline model, all unlabeled data q0 are labeled and the labeled data is used as the standard answer.

[0069] Figure 2 is a process flow chart based on an acquisition queue system according to one embodiment of the present disclosure.

[0070] As an example, see Figure 2 , based on the acquisition queue system, where the unlabeled data is distributed in the queue space in sequence, including:

[0071] S21: Put all untagged data into queue 0, and set the other queues to empty;

[0072] S22: Select a specific number n of single-round models in the model training process of step S1;

[0073] S23: For each single-round model, the annotation results of the single-round model are compared with those of the benchmark model. In each cycle, the parameter differences between the annotation data of the single-round model and the standard answer are compared. The parameters include entity boundaries, entity categories, and the number of entities. The queue data are placed at intervals. If they are completely consistent, they are promoted to a higher queue, otherwise they are demoted to a lower queue. After the cycle is completed, they are returned to the queue system.

[0074] Specifically, according to the growth trend of the model F value during the training process, n models obtained from a single round of training are selected, and these n models are used to repeatedly annotate all unlabeled data. Model0 to model3 are selected as single-round models as examples. First, model0 is used to annotate all unlabeled data. After comparing with the standard answer obtained in step S1, the completely consistent data samples are moved to q1, and the inconsistent ones are moved to q0; then model1 is used to annotate the data in the two groups of queues respectively, and the queues are continued to be divided after comparison with the standard answer; then model1 and model1 are continued to be used for queue division; finally, queues with the same serial number are continued to be merged until the single-round models are exhausted, and queue systems q0, q1, q2, and q3 are obtained.

[0075] Among them, when comparing with the standard answer, the absolute mode is used to judge the entities annotated by the model. When the queue data is used to compare the annotation results with the optimal model, only when the entities annotated by the model in the two sentences are completely consistent in number, boundaries and types, they are judged to be annotated with the same data and can be promoted to a higher-level queue, otherwise they will be demoted to a lower-level queue.

[0076] Step S3: Add the pseudo data other than the labeled data in the queue space to the training data for model training to perform model training; select the selected queue data that has the most obvious improvement in model performance as the optimal data to be added to the training set, and remove the selected queue data from the unlabeled data. Among them, the pseudo data of each queue in the queue system generated in step S2 is added to the training set respectively, and the model is trained through the training set and the development set. The performance of the trained model is compared, and the selected queue data that has the most obvious improvement in model performance is selected as the optimal data to be added to the training set.

[0077] Among them, the improvement of model performance is determined by comparing the harmonic mean F value:

[0078] F=(2×P×R)×100% / (P+R);

[0079] P is the accuracy:

[0080] P = number of correctly identified entities × 100% / number of identified entities;

[0081] R is the recall rate:

[0082] R = number of correctly identified entities x 100% / number of correct entities.

[0083] After the selected queue data with the most obvious improvement in model performance is added as the optimal data to the training set, the selected queue data is removed from the unlabeled data to generate a new training set and unlabeled data.

[0084] Step S4, for the newly generated training set and unlabeled data in step S3, repeat the cycle of steps S1 to S3 until the model performance is no longer improved or the unlabeled data is used up.

[0085] Among them, whether the model performance is improved can be determined by comparing the harmonic mean F value.

[0086] Figure 3 It is a flowchart of a method for named entity recognition according to a disclosed embodiment.

[0087] As an example, see Figure 3 , the named entity recognition method of this embodiment includes:

[0088] S100: Obtain training set, development set, unlabeled data, and data to be labeled;

[0089] S200: Using the training set, the development set, and the unlabeled data, an entity recognition model is established by using the method for establishing a named entity recognition model of any of the above embodiments;

[0090] S300: Input the data to be annotated into the entity recognition model to obtain annotated data.

[0091] Figure 4 It is a schematic block diagram of the structure of an apparatus for named entity recognition model implemented by hardware of a processing system according to an embodiment of the present disclosure.

[0092] like Figure 4 As shown, the device 1000 for establishing a named entity recognition model of the present disclosure may include corresponding modules for executing each or several steps in the above flowchart. Therefore, each step or several steps in the above flowchart can be executed by the corresponding module, and the device may include one or more modules in these modules. The module can be one or more hardware modules specially configured to execute the corresponding steps, or implemented by a processor configured to execute the corresponding steps, or stored in a computer-readable medium for implementation by a processor, or implemented by some combination.

[0093] Specifically, the apparatus 1000 for establishing a named entity recognition model disclosed herein includes:

[0094] A benchmark model generation module 1001, the benchmark model generation module 1001 generates a benchmark model using a training set and a development set;

[0095] The standard answer generation module 1002 uses the benchmark model generated by the benchmark model generation module 1001 to annotate the unlabeled data, obtain the annotated data, and generate a standard answer;

[0096] The queue system acquisition module 1003 distributes the unlabeled data in the queue space in sequence based on the acquired queue system;

[0097] The optimal data selection module 1004 and the queue system acquisition module 1003 respectively add the pseudo data in the queue space to the training data for model training to perform model training; select the queue data that has the most obvious improvement on the model performance as the optimal data to be added to the training set, and remove this batch of data from the unlabeled data;

[0098] The operation control module 1005 controls the benchmark model generation module 1001, the standard answer generation module 1002, the queue system acquisition module 1003 and the optimal data selection module 1004 to run in a loop until the model performance is no longer improved or the unlabeled data is used up.

[0099] The hardware structure of the device can be implemented using a bus architecture. The bus architecture can include any number of interconnected buses and bridges, depending on the specific application and overall design constraints of the hardware. The bus 1100 connects various circuits including one or more processors 1200, memory 1300 and / or hardware modules together. The bus 1100 can also connect various other circuits 1400 such as peripherals, voltage regulators, power management circuits, external antennas, etc.

[0100] The bus 1100 may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the figure only uses one connecting line, but does not mean that there is only one bus or one type of bus.

[0101] Any process or method description in the flowchart or otherwise described herein can be understood as a module, fragment or portion of a code representing one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present disclosure includes other implementations, in which the functions may not be performed in the order shown or discussed, including performing the functions in a substantially simultaneous manner or in a reverse order according to the functions involved, which should be understood by a person skilled in the art of the present disclosure. The processor performs the various methods and processes described above. For example, the method implementation in the present disclosure can be implemented as a software program, which is tangibly contained in a machine-readable medium, such as a memory. In some embodiments, part or all of the software program can be loaded and / or installed via a memory and / or a communication interface. When the software program is loaded into the memory and executed by the processor, one or more steps in the method described above can be performed. Alternatively, in other embodiments, the processor can be configured to perform one of the above methods in any other appropriate manner (e.g., by means of firmware).

[0102] The logic and / or steps represented in the flowchart or otherwise described herein may be embodied in any readable storage medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other system that can fetch instructions from and execute instructions on an instruction execution system, apparatus, or device), or in combination with such instruction execution systems, apparatuses, or devices.

[0103] For the purposes of this specification, a "readable storage medium" may be any device that can contain, store, communicate, propagate or transmit a program for use with or in conjunction with an instruction execution system, device or apparatus. More specific examples of readable storage media (a non-exhaustive list) include the following: an electrical connection with one or more wirings (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and editable read-only memory (EPROM or flash memory), a fiber optic device, and a portable read-only memory (CDROM). In addition, the readable storage medium may even be paper or other suitable medium on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a memory.

[0104] It should be understood that the various parts of the present disclosure can be implemented in hardware, software, or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented in software stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0105] A person skilled in the art may understand that all or part of the steps of implementing the above-mentioned implementation method may be completed by instructing related hardware through a program, and the program may be stored in a readable storage medium, which, when executed, includes one or a combination of the steps of the implementation method.

[0106] In addition, each functional unit in each embodiment of the present disclosure may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a readable storage medium. The storage medium may be a read-only memory, a disk or an optical disk, etc.

[0107] In the description of this specification, the description with reference to the terms "one embodiment / method", "some embodiments / methods", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment / method or example are included in at least one embodiment / method or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment / method or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments / methods or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments / methods or examples described in this specification and the features of the different embodiments / methods or examples, unless they are contradictory.

[0108] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0109] Those skilled in the art should understand that the above embodiments are only for the purpose of clearly illustrating the present disclosure, and are not intended to limit the scope of the present disclosure. For those skilled in the art, other changes or modifications may be made based on the above disclosure, and these changes or modifications are still within the scope of the present disclosure.

Claims

1. A method for establishing a named entity recognition model, characterized in that: The steps include: S1: Use the training set and the development set to train the model, generate a benchmark model, and use the benchmark model to annotate the unlabeled data to obtain the annotated data, which is used as the standard answer; S2: Based on the acquisition queue system, the unlabeled data is distributed in the queue space in sequence; S3: adding the pseudo data other than the labeled data in the queue space to the training data for model training to perform model training; selecting the selected queue data that has the most obvious improvement on the model performance as the optimal data to be added to the training set, and removing the selected queue data from the unlabeled data; as well as S4: Repeat steps S1 to S3 until the model performance no longer improves or the unlabeled data is used up; Step S2 includes: S21: Put all untagged data into queue 0, and set the other queues to empty; S22: Select a specific number n of single-round models in the model training process of step S1; S23: for each single-round model, the annotation result of the single-round model is compared with the standard answer, and the parameter difference between the annotation data of the single-round model and the standard answer is compared in each cycle, wherein the parameters include entity boundaries, entity categories and the number of entities; the queue data is placed at intervals, and if they are completely consistent, they are promoted to a higher queue, otherwise they are demoted to a lower queue, and after the cycle is completed, they are returned to the queue system; In step S23, the absolute mode is used to judge the entities annotated by the model. When the queue data is used to compare the annotation results with the optimal model, only when the entities annotated by the model in the two sentences meet the requirements of complete consistency in number, boundaries and types, are they judged as annotated data and can be promoted to a higher-level queue, otherwise they are demoted to a lower-level queue.

2. The method for establishing a named entity recognition model according to claim 1, characterized in that: In steps S3 and S4, whether the model performance is improved is determined by comparing the harmonic mean F value, wherein: F=(2×P×R)×100% / (P+R); P is the accuracy: P = number of correctly identified entities × 100% / number of identified entities; R is the recall rate: R = number of correctly identified entities x 100% / number of correct entities.

3. The method for establishing a named entity recognition model according to claim 2, characterized in that: The higher the harmonic mean F value, the better the model performance; The named entity recognition model is used for entity recognition of medical electronic medical records.

4. The method for establishing a named entity recognition model according to claim 1, characterized in that: The named entity recognition model uses the WC-LSTM model.

5. A method for named entity recognition, comprising the following steps: S100: Obtain training set, development set, unlabeled data, and data to be labeled; S200: Using the training set, the development set, and the unlabeled data, an entity recognition model is established by the method for establishing a named entity recognition model according to any one of claims 1 to 4; and S300: Input the data to be annotated into the entity recognition model to obtain annotated data.

6. An electronic device, characterized in that: include: A memory storing execution instructions; as well as A processor, wherein the processor executes the execution instructions stored in the memory, so that the processor executes the method for establishing a named entity recognition model as described in any one of claims 1 to 4 and / or the named entity recognition method as described in claim 5.

7. A computer storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute the method for establishing a named entity recognition model according to any one of claims 1 to 4 and / or the named entity recognition method according to claim 5.

Images