Trigger word quality assessment method and assessment device

Abstract

The present disclosure provides a method for evaluating the quality of trigger words, including: segmenting the sentences containing entities in the training set, and determining a candidate trigger word set according to the context information of the entities after segmentation; using the trigger word P in the candidate trigger word set i Mining the training set to obtain N i entities; determining the number F of entities that already exist in the entity dictionary of the training set among the N i entities i ; if F i is not equal to 0, calculate the value of F i / N i , and use this value as the quality score of the trigger word P i ; if F i is equal to 0, obtain the annotation result y of each entity among the N i entities, calculate the similarity of each entity among the N i entities, sum up the similarities of all entities, and use this summation value as the quality score of the trigger word P i ; determine the quality of the pair of trigger words P i according to the quality score of the trigger word P i . The present disclosure also provides a device for evaluating the quality of trigger words, a readable storage medium, and an electronic device.

Description

Technical Field

[0001] The present disclosure relates to the field of language processing technology, and in particular to a method, an evaluation device, a readable storage medium, and an electronic device for evaluating the quality of trigger words in medical record entity recognition. Background Art

[0002] Early named entity recognition systems generally adopted a rule-based recognition method, in which linguists formulated a series of entity recognition rules for specific tasks to guide computers to perform entity recognition on text content. In recent years, with the construction of large-scale named entity recognition corpora, data-driven named entity recognition models have become the main method for scholars to study and implement named entity recognition tasks.

[0003] Compared with general domain entity recognition, medical entities are extremely professional and specific. The core of machine learning methods lies in feature extraction. The premise for the model to achieve good results is to first invest a lot of manual feature analysis. Due to the strong professionalism of medical field texts, the difficulty in collecting external dictionaries, and too little labeled data, model training has been hindered. Summary of the invention

[0004] In order to solve at least one of the above technical problems, the present disclosure proposes a trigger word quality assessment method, an assessment device, a readable storage medium, and an electronic device.

[0005] According to one aspect of the present disclosure, a trigger word quality assessment method is provided, comprising the following steps:

[0006] S1: Segment the sentences containing entities in the training set, and determine the candidate trigger word set according to the context information of the entities after segmentation;

[0007] S2: Use the trigger word P in the candidate trigger word set i Mining the training set to obtain N i Entity;

[0008] S3: Determine the N i The number of entities that already exist in the training set entity dictionary F i ; if F i Not equal to 0, go to step S4; if F i Equal to 0, go to step S5;

[0009] S4: Calculate F i / N i The value of the trigger word P i quality score; proceed to step S7;

[0010] S5: Get model pair N i Each entity ymatch The labeling result y;

[0011] S6: Calculate N i The similarity of each entity in the entities is calculated, and the similarity of all entities is summed up, and the sum value is used as the trigger word P i The quality score of

[0012] S7: For each pair of trigger words P in the candidate trigger word set i Steps S2 to S6 are executed repeatedly to determine the trigger word pair P in the candidate trigger word set. i The quality score of

[0013] S8: According to each pair of trigger words P in the candidate trigger word set i The quality score of the trigger word P i quality.

[0014] According to the trigger word quality assessment method of at least one embodiment of the present disclosure, the entity is a medical named entity.

[0015] According to the trigger word quality assessment method of at least one embodiment of the present disclosure, the medical named entity types include examination and inspection, symptoms and signs, disease and diagnosis, treatment, and site.

[0016] According to the trigger word quality assessment method of at least one embodiment of the present disclosure, the trigger word has a preceding word and a succeeding word.

[0017] According to the trigger word quality assessment method of at least one embodiment of the present disclosure, in step S2, the trigger word P in the candidate trigger word set is used. i Mining the training set to obtain N i Entities include: Find N in the training set i Trigger word P i , the trigger word P i The entities between the preceding word and the succeeding word are considered as mined entities.

[0018] According to the trigger word quality assessment method of at least one embodiment of the present disclosure, in step S5, the model is a model trained by a training set and a development set.

[0019] According to the trigger word quality assessment method of at least one embodiment of the present disclosure, in step S6, the similarity sim(y match , y).

[0020] According to the trigger word quality assessment method of at least one embodiment of the present disclosure, in step S8, the quality score value indicates the quality of the trigger word.

[0021] According to another aspect of the present disclosure, a trigger word quality assessment device is provided, comprising:

[0022] A candidate trigger word set determination module, which performs word segmentation on sentences containing entities in a training set and determines a candidate trigger word set according to context information of the entities after word segmentation;

[0023] A trigger word mining module uses a trigger word P in a candidate trigger word set. i Mining the training set to obtain N i Entity;

[0024] An entity number comparison module is used to determine the N given by the trigger word mining module. i The number of entities that already exist in the training set entity dictionary F i , according to F i The value of the module is operated;

[0025] The quality score determines the first module, wherein the quality score determines the first module in F i When it is not equal to 0, calculate F i / N i The value of the trigger word P i The quality score of

[0026] A model annotation module, wherein the model annotation module obtains the model pair N i Each entity y match The labeling result y;

[0027] The quality score determines the second module, and the quality score determines the second module to calculate N i The similarity of each entity in the entities is calculated, and the similarity of all entities is summed up, and the sum value is used as the trigger word P i The quality score of

[0028] A loop execution module is used to calculate the quality score of each trigger word in the candidate trigger word set, and determine the quality score of each pair of trigger words P in the candidate trigger word set. i The quality score of

[0029] A trigger word quality assessment module is configured to assess the quality of each trigger word P in the candidate trigger word set. i The quality score of each pair of trigger words P i quality.

[0030] According to another aspect of the present disclosure, a readable storage medium is provided, wherein the readable storage medium stores a computer program, wherein the computer program is used for a processor to execute the trigger word quality assessment method of any embodiment of the present disclosure.

[0031] According to another aspect of the present disclosure, an electronic device is provided, which includes a processor and a readable storage medium, wherein the readable storage medium stores execution instructions, and the processor executes the execution instructions in the readable storage medium, so that the processor executes the trigger word quality assessment method of any embodiment of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The accompanying drawings illustrate exemplary embodiments of the present disclosure and together with the description serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.

[0033] Figure 1 It is a flowchart of a trigger word quality assessment method according to an embodiment of the present disclosure.

[0034] Figure 2 It is a flowchart of constructing a candidate trigger word set according to an embodiment of the present disclosure.

[0035] Figure 3 It is a schematic block diagram of the structure of a trigger word quality assessment device according to an embodiment of the present disclosure.

[0036] Figure 4 It is a schematic block diagram of the structure of an electronic device with a trigger word quality assessment device according to an embodiment of the present disclosure.

[0037] Description of Reference Numerals

[0038] 1000 Electronic equipment

[0039] 1001 Candidate trigger word set determination module

[0040] 1002 Trigger word mining module

[0041] 1003 Entity Number Comparison Module

[0042] 1004 Quality Score Determination Module 1

[0043] 1005 Model Annotation Module

[0044] 1006 Quality Score Determination Module 2

[0045] 1007 Loop execution module

[0046] 1008 Trigger word quality assessment module

[0047] 1100 Bus

[0048] 1200 processor

[0049] 1300 Memory

[0050] 1400 Other circuits. DETAILED DESCRIPTION

[0051] The present disclosure is further described in detail below in conjunction with the accompanying drawings and implementations. It is understood that the specific implementations described herein are only used to explain the relevant content, rather than to limit the present disclosure. It should also be noted that, for ease of description, only the parts related to the present disclosure are shown in the accompanying drawings.

[0052] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in the present disclosure can be combined with each other. The technical solution of the present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0053] Unless otherwise specified, the exemplary embodiments / embodiments shown will be understood as providing exemplary features of various details of some ways in which the technical concept of the present disclosure can be implemented in practice. Therefore, unless otherwise specified, the features of the various embodiments / embodiments can be combined, separated, interchanged and / or rearranged without departing from the technical concept of the present disclosure.

[0054] The use of cross-hatching and / or shading in the accompanying drawings is generally used to make the boundaries between adjacent components clear. As such, unless otherwise specified, the presence or absence of cross-hatching or shading does not convey or indicate any preference or requirement for the specific materials, material properties, dimensions, proportions, commonalities between the components shown, and / or any other characteristics, attributes, properties, etc. of the components. In addition, in the accompanying drawings, the sizes and relative sizes of the components may be exaggerated for clarity and / or descriptive purposes. When the exemplary embodiments can be implemented differently, the specific process sequence can be performed in a different order than described. For example, two successively described processes can be performed substantially simultaneously or in an order opposite to the described order. In addition, the same figure numbers represent the same components.

[0055] When a component is referred to as being "on," "over," "connected to," or "coupled to" another component, the component may be directly on, directly connected to, or directly coupled to the other component, or intervening components may be present. However, when a component is referred to as being "directly on," "directly connected to," or "directly coupled to" another component, there are no intervening components. For this purpose, the term "connected" may refer to a physical connection, an electrical connection, etc., with or without intervening components.

[0056] The terms used herein are for the purpose of describing specific embodiments, and are not intended to be restrictive. As used herein, unless the context clearly indicates otherwise, the singular forms "one (kind, person)" and "said (the)" are also intended to include plural forms. In addition, when the terms "comprise" and / or "include" and their variations are used in this specification, it is explained that there are stated features, integral bodies, steps, operations, parts, assemblies and / or their groups, but it is not excluded that there are or add one or more other features, integral bodies, steps, operations, parts, assemblies and / or their groups. It should also be noted that, as used herein, the terms "substantially", "approximately" and other similar terms are used as approximate terms and not as degree terms, so that they are used to explain the inherent deviations of the measured values, calculated values ​​and / or the values ​​provided that will be recognized by those of ordinary skill in the art.

[0057] For medical named entities, it is crucial to improve the quality of newly annotated data expanded using pre-models. From the data samples, it can be found that the semantic information of the entity's trigger word can accurately lock the entity's position, that is, the entity boundary required for named entity recognition, and the entity type can be determined by the trigger word and the professional knowledge of the annotator. Therefore, extracting high-quality trigger words to trigger the constraints of semantic information can effectively identify and denoise potential noise samples, effectively improving model performance.

[0058] Figure 1 : is a flow chart of a trigger word quality assessment method according to an embodiment of the present disclosure, comprising the following steps:

[0059] S1: Segment the sentences containing entities in the training set, and determine the candidate trigger word set according to the context information of the entities after segmentation;

[0060] S2: Use the trigger word P in the candidate trigger word set i Mining the training set to obtain N i entities, where the subscript i is the trigger word number;

[0061] S3: Determine the N i The number of entities that already exist in the training set entity dictionary Fi ; if F i Not equal to 0, go to step S4; if F i Equal to 0, go to step S5;

[0062] S4: Calculate F i / N i The value of the trigger word P i quality score; proceed to step S7;

[0063] S5: Get model pair N i Each entity y match The labeling result y;

[0064] S6: Calculate N i The similarity sim(y match , y), sum the similarities of all entities and use the sum as the trigger word P i The quality score of

[0065] S7: For each pair of trigger words P in the candidate trigger word set i Steps S2 to S6 are executed repeatedly to determine the trigger word pair P in the candidate trigger word set. i The quality score of

[0066] S8: According to each pair of trigger words P in the candidate trigger word set i The quality score of the trigger word P i quality.

[0067] In step S1, entities in the training set are selected to construct a trigger vocabulary, where the entities are medical named entities, and the medical named entity types include examination and testing (B-CHECK, I-CHECK), symptoms and signs (B-SIGNS, I-SIGNS), disease and diagnosis (B-DISEASE, I-DISEASE), treatment (B-TREATMENT, I-TREATMENT), and body part (B-BODY, I-BODY).

[0068] Specifically, Figure 2 The flowchart of constructing a candidate trigger word set according to an embodiment of the present disclosure includes:

[0069] S11. Add the entity dictionary in the training set to the custom dictionary in the word segmentation tool, and collect the sentences containing entities in the training set.

[0070] S12. Segment the sentences containing entities using a segmentation tool.

[0071] The word segmentation tool may use an existing word segmentation tool, such as the open source tool Jieba, add professional dictionaries to the Jieba dictionary library, and use Jieba to perform word segmentation on the text.

[0072] S13. Construct a set of candidate trigger words according to the context information of the entity after word segmentation.

[0073] Statistical analysis shows that in a training set containing 6,000 sentences, more than 700 trigger word pairs can be obtained for the disease type alone. The quality of these word pairs varies greatly. Using all trigger words for entity mining will not only consume a lot of time, but is also likely to introduce more noise data.

[0074] In step S2, after determining the candidate trigger word set, use the trigger word P in the candidate trigger word set i Mining the training set to obtain N i entity.

[0075] Preferably, the trigger word has a preceding word and a succeeding word, and the trigger word P in the candidate trigger word set is used. i Mining the training set to obtain N i Entities include: Find N in the training set i Trigger word P i , the trigger word P i The entities between the preceding word and the succeeding word are considered as mined entities.

[0076] In step S3, determine N i The number of entities that already exist in the training set entity dictionary F i ; if F i Not equal to 0, go to step S4; if F i Equal to 0, go to step S5.

[0077] For example, the trigger word pair "to, admitted to hospital" mines four entities: "hypertension, diabetes, heart disease, hypoglycemia". i =4, go to step S4; for example, for the sentence "For further treatment, the emergency department was admitted to the hospital with left orbital skin soft tissue laceration, multiple soft tissue injuries, head trauma and neurological reaction introduction", "admitted to the hospital with xxx" mined the entity "left orbital skin soft tissue laceration, multiple soft tissue injuries, head trauma and neurological reaction introduction", at this time F i Equal to 0, go to step 5.

[0078] In step S4, calculate F i / N i The value of the trigger word P i The quality score Score (P i); for example, the trigger word pair "to be hospitalized" mines four entities, namely, "hypertension, diabetes, heart disease, and hypoglycemia", among which "hypertension, diabetes, and heart disease" have appeared in the training set entities of the original diseases, indicating that its accuracy rate is 3 / 4*100%=75%.

[0079] In steps S5 to S6, for F i is equal to 0, indicating that the trigger word P i The mined entity has never appeared in the entity dictionary. Get the model for N i Each entity y match The annotation result y, calculate N i The similarity sim(y match , y). Among them, the model is a model trained by the training set and the development set. For example, for the sentence "For further treatment, the emergency department was admitted to the hospital with the introduction of left orbital skin and soft tissue lacerations, multiple soft tissue injuries, and neurological reactions after head trauma", "admitted to the hospital with xxx" mined the entity "introduction to neurological reactions after left orbital skin and soft tissue lacerations, multiple soft tissue injuries, and head trauma". This entity is very long, and the entity annotated by the model is "left orbital skin and soft tissue lacerations, multiple soft tissue injuries, and neurological reactions after head trauma". Calculate the similarity between the two and calculate the score. Among them, the similarity can be calculated by the edit distance. For N i The similarity of each entity in the entities is summed up, and the sum value is used as the trigger word P i The quality score Score (P i ).

[0080] For each pair of trigger words P in the candidate trigger word set i Steps S2 to S5 are executed repeatedly to determine the trigger word pair P in the candidate trigger word set. i The quality score of each pair of trigger words P in the candidate trigger word set i The quality score of the trigger word Pi is used to determine the quality of the pair of trigger words Pi. The value of the quality score Score (Pi) of the trigger word Pi indicates the quality of the trigger word. The higher the quality of the corresponding trigger word, the higher the accuracy of the entity word selected based on it. The trigger words whose quality score Score (Pi) value is higher than the quality threshold can be selected to form the trigger word library.

[0081] Figure 3 A trigger word quality assessment device according to an embodiment of the present disclosure is shown, comprising:

[0082] A candidate trigger word set determination module 1001, which performs word segmentation on sentences containing entities in a training set and determines a candidate trigger word set according to context information of the entities after word segmentation;

[0083] The trigger word mining module 1002 uses the trigger word P in the candidate trigger word set to i Mining the training set to obtain N i Entity;

[0084] Entity number comparison module 1003, the entity number comparison module 1003 determines the N given by the trigger word mining module i The number of entities that already exist in the training set entity dictionary F i , according to F i The value of the module is operated;

[0085] The quality score determines the first module 1004, the quality score determines the first module 1004 in F i When it is not equal to 0, calculate F i / N i The value of the trigger word P i The quality score of

[0086] Model annotation module 1005, the model annotation module obtains the model 1005 for N i Each entity y match The labeling result y; the quality score determines the second module 1006, the quality score determines the second module 1006 calculates N i The similarity sim(y match , y), sum the similarities of all entities and use the sum as the trigger word P i The quality score of

[0087] The loop execution module 1007 performs a quality score calculation operation on each trigger word in the candidate trigger word set to determine the quality score of each pair of trigger words P in the candidate trigger word set. i The quality score of

[0088] The trigger word quality evaluation module 1008 is used to evaluate the quality of each trigger word P in the candidate trigger word set. i The quality score of each pair of trigger words P i quality.

[0089] Figure 4 It is a schematic block diagram of the structure of an electronic device with a trigger word quality assessment device according to an embodiment of the present disclosure.

[0090] like Figure 4As shown, the electronic device 1000 may include a corresponding module for executing each or several steps in the above method. Therefore, each step or several steps in the above method may be executed by a corresponding module, and the electronic device 1000 may include one or more modules in these modules. The module may be one or more hardware modules specially configured to execute the corresponding steps, or implemented by a processor configured to execute the corresponding steps, or stored in a computer-readable medium for implementation by a processor, or implemented by some combination.

[0091] The hardware structure of the electronic device 1000 can be implemented using a bus architecture. The bus architecture can include any number of interconnected buses and bridges, depending on the specific application and overall design constraints of the hardware. The bus 1100 connects various circuits including one or more processors 1200, memory 1300 and / or hardware modules together. The bus 1100 can also connect various other circuits 1400 such as peripherals, voltage regulators, power management circuits, external antennas, etc.

[0092] The bus 1100 may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the figure only uses one connecting line, but does not mean that there is only one bus or one type of bus.

[0093] In the description of this specification, the description with reference to the terms "one embodiment / method", "some embodiments / methods", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment / method or example are included in at least one embodiment / method or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment / method or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments / methods or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments / methods or examples described in this specification and the features of the different embodiments / methods or examples, unless they are contradictory.

[0094] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0095] Those skilled in the art should understand that the above embodiments are only for the purpose of clearly illustrating the present disclosure, and are not intended to limit the scope of the present disclosure. For those skilled in the art, other changes or modifications may be made based on the above disclosure, and these changes or modifications are still within the scope of the present disclosure.

Claims

1. A trigger word quality assessment method, It is characterized in that The steps include: S1: Segment the sentences containing entities in the training set, and determine the candidate trigger word set according to the context information of the entities after segmentation; S2: Use the trigger word Pi in the candidate trigger word set to mine the training set to obtain Ni entities; The trigger word has a preceding word and a succeeding word. The trigger word Pi in the candidate trigger word set is used to mine the training set to obtain Ni entities including: Find Ni trigger words Pi in the training set, and take the entity between the preceding word and the succeeding word of the trigger word Pi as the mined entity; S3: Determine the number Fi of entities that already exist in the training set entity dictionary among the Ni entities; if Fi is not equal to 0, proceed to step S4; if Fi is equal to 0, proceed to step S5; S4: Calculate the value of Fi / Ni and use the value as the quality score of the trigger word Pi; proceed to step S7; S5: Obtain the model's annotation result y for each of the Ni entities; S6: Calculate the similarity of each entity in the Ni entities, sum the similarities of all entities, and use the sum as the quality score of the trigger word Pi; S7: looping through steps S2 to S6 for each pair of trigger words P i in the candidate trigger word set to determine the quality score of each pair of trigger words P i in the candidate trigger word set; and S8: Determine the quality of each pair of trigger words Pi in the candidate trigger word set according to the quality score of the pair of trigger words Pi.

2. The trigger word quality assessment method according to claim 1, Features: The entity is a medical named entity.

3. The trigger word quality assessment method according to claim 2, Features: The entities of the medical named entity type include examination and test, symptoms and signs, disease and diagnosis, treatment, and part.

4. The trigger word quality assessment method according to claim 1, Features: The trigger word has a preceding word and a succeeding word.

5. The trigger word quality assessment method according to claim 1, Features: In the step S2, using the trigger word Pi in the candidate trigger word set to mine the training set to obtain Ni entities includes: searching for Ni trigger words Pi in the training set, and taking the entity located between the preceding word and the succeeding word of the trigger word Pi as the mined entity.

6. The trigger word quality assessment method according to claim 1, Features: In step S5, the model is a model trained by a training set and a development set.

7. The trigger word quality assessment method according to claim 1, Features: In step S8, the quality score value indicates the quality of the trigger word.

8. A trigger word quality assessment device, It is characterized in that include: A candidate trigger word set determination module, which performs word segmentation on sentences containing entities in a training set and determines a candidate trigger word set according to context information of the entities after word segmentation; A trigger word mining module, wherein the trigger word mining module uses the trigger word Pi in the candidate trigger word set to mine the training set to obtain Ni entities, wherein the trigger word has a preceding word and a succeeding word, and uses the trigger word Pi in the candidate trigger word set to mine the training set to obtain Ni entities, including: Find Ni trigger words Pi in the training set, and take the entity between the preceding word and the succeeding word of the trigger word Pi as the mined entity; An entity number comparison module, which determines the number of entities Fi that already exist in the training set entity dictionary among the Ni entities given by the trigger word mining module, and performs module operations according to the value of Fi; A first quality score determination module, wherein when Fi is not equal to 0, the first quality score determination module calculates a value of Fi / Ni and uses the value as the quality score of the trigger word Pi; A model annotation module, wherein the model annotation module obtains the annotation result y of the model for each entity in the Ni entities; The second module for determining the quality score calculates the similarity of each entity in the Ni entities, sums the similarities of all entities, and uses the sum as the quality score of the trigger word Pi; A loop execution module, wherein the loop execution module performs an operation of calculating a quality score for each trigger word in the candidate trigger word set to determine a quality score for each pair of trigger words P i in the candidate trigger word set; and A trigger word quality assessment module is used to determine the quality of each pair of trigger words Pi according to the quality score of each pair of trigger words Pi in the candidate trigger word set.

9. A readable storage medium, It is characterized in that The readable storage medium stores a computer program, and the computer program is used for a processor to execute the trigger word quality assessment method according to any one of claims 1 to 7.

10. An electronic device, It is characterized in that It includes a processor and a readable storage medium, the readable storage medium stores execution instructions, and the processor executes the execution instructions in the readable storage medium, so that the processor executes the trigger word quality assessment method according to one of claims 1-7.

Images