Method, device, computing equipment and medium for assisting in labeling medical named entities

Abstract

A method, apparatus, computing device and medium for assisting in labeling medical named entities are provided. The method may include: using at least a first model and a second model to respectively recognize medical text to determine a first score corresponding to each medical named entity recognized by the first model; using a third model to recognize the medical text to obtain a reference recognition result, the third model having a higher recognition accuracy than the at least first model and second model; determining an evaluation score corresponding to each medical named entity recognized by the first model based on the first recognition result of the first model, the reference recognition result and the first score corresponding to each medical named entity recognized by the first model; and outputting a labeling result of each medical named entity included in the recognition result of the first model based on the evaluation score corresponding to each medical named entity recognized by the first model.

Description

Technical Field

[0001] The present application relates to the field of data processing, and more specifically, to a method, apparatus, computing device, and medium for assisting in labeling medical named entities in medical texts based on multiple models. Background Art

[0002] As an important part of the modern medical system, smart healthcare faces the challenge of limited high-quality medical resources. Especially in the field of oncology, the workload of specialists is huge, and there is an urgent need to reduce the workload of doctors through artificial intelligence technology, so that medical resources can be allocated more reasonably and efficiently, thereby alleviating the problem of tight medical resources. In this context, the structuring and standardization of medical texts have become key tasks, especially in named entity recognition. Named entity recognition is generally considered to be the process of identifying meaningful entities or entity referents from the original text, that is, identifying named entities in the text and dividing them into corresponding entity categories (i.e., annotations), etc.

[0003] Medical named entity recognition is an important application of natural language processing in the medical field, and is of great significance for improving the efficiency of information extraction and the quality of knowledge discovery in medical texts. However, traditional machine learning models face high annotation costs when processing massive amounts of unformatted medical data. The complexity of tumor medical texts further increases this challenge, mainly due to the following reasons: the numerous entity concepts in the medical field, the diverse entity context constraints, the huge differences in entity lengths, and the existence of inclusion and intersection relationships between entities. These characteristics make the named entity recognition and annotation of medical texts extremely difficult and time-consuming.

[0004] Therefore, there is a need for an effective auxiliary annotation system that can assist doctors and researchers to quickly and accurately extract key information from complex medical texts, reduce the cost of manual annotation and improve the efficiency of data processing. Summary of the invention

[0005] According to one aspect of the present application, a method for assisting in labeling medical named entities is provided, the method comprising: using at least a first model and a second model to respectively recognize medical text to obtain at least one recognition result to determine a first score corresponding to each medical named entity recognized by the first model; using a third model having a higher recognition accuracy than the at least first model and second model to recognize the medical text to obtain a reference recognition result; based on the first recognition result of the first model, the reference recognition result and the first score corresponding to each medical named entity recognized by the first model, determining an evaluation score corresponding to each medical named entity recognized by the first model; and outputting a labeling result for each medical named entity included in the recognition result of the first model based on the evaluation score corresponding to each medical named entity recognized by the first model.

[0006] According to another aspect of the present application, a device for assisting in labeling medical named entities is also provided, the device comprising: a first recognition module, for respectively recognizing medical text using at least a first model and a second model to determine a first score corresponding to each medical named entity recognized by the first model; a second recognition module, for recognizing the medical text using a third model having a higher recognition accuracy than the at least first model and the second model to obtain a reference recognition result; an evaluation module, for determining an evaluation score corresponding to each medical named entity recognized by the first model based on the first recognition result of the first model, the reference recognition result and the first score corresponding to each medical named entity recognized by the first model; and an output module, for outputting a labeling result of each medical named entity included in the recognition result of the first model based on the evaluation score corresponding to the second score of each medical named entity recognized by the first model.

[0007] According to another aspect of the present application, a computing device is also provided, which includes: one or more processors; and one or more memories, coupled to the one or more processors and storing a computer program, and when the computer program is executed by the one or more processors, the one or more processors execute the method as described above.

[0008] According to another aspect of the present application, a non-transitory computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by the one or more processors, the one or more processors execute the method as described above.

[0009] In an embodiment of the present application, by adopting models of different scales to work together and integrating the recognition results of medical texts by models of different scales, entity recognition information from multiple angles can be comprehensively considered, thereby improving the overall accuracy and reliability. In addition, the powerful summarization, logical understanding and reasoning capabilities of the large model are utilized to evaluate the medical named entities identified by the small model to determine the final output evaluation score, which can be used to determine whether the corresponding identified medical named entity requires further manual review. For the identified medical named entities that require manual review, after manual review, they can have more accurate labeling results; for the identified medical named entities that do not require manual review, it is determined that the labeling results of the medical named entities identified by the model are accurate (reliable), thereby improving labeling efficiency and saving manpower. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings illustrate various embodiments of various aspects of the present application, and together with the specification, they are used to explain the principles of the present application. It is understood by those skilled in the art that the specific embodiments shown in the accompanying drawings are only exemplary, and they are not intended to limit the scope of the present application. In the accompanying drawings:

[0011] Figure 1 An example architecture diagram of an auxiliary annotation solution for medical named entities according to an embodiment of the present application is shown.

[0012] Figure 2 A flow chart of a method for assisting in labeling medical named entities according to an embodiment of the present application is shown.

[0013] Figure 3 Example reference recognition results of the large model on example medical text are shown.

[0014] Figure 4 A schematic diagram of a process for assisting in labeling medical named entities is shown.

[0015] Figure 5 A structural block diagram of a device for assisting in labeling medical named entities is shown.

[0016] Figure 6 A structural diagram of a computing device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution of the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings of the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the described embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0018] Unless otherwise defined, the technical terms or scientific terms used in the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. "First", "second" and similar words used in the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. In order to keep the following description of the embodiments of the present disclosure clear and concise, the present disclosure omits the detailed description of some known functions and known components.

[0019] Flowcharts are used in the present disclosure to illustrate the steps of the method according to the embodiments of the present disclosure. It should be understood that the preceding or following steps are not necessarily performed accurately in order. On the contrary, various steps may be processed in reverse order or simultaneously. At the same time, other operations may also be added to these processes, or one or more steps may be removed from these processes.

[0020] As mentioned earlier, current machine learning models face high annotation costs when processing massive amounts of unformatted medical text data. Therefore, an effective auxiliary annotation system is needed to assist doctors and researchers in quickly and accurately extracting key information from complex medical texts, while also reducing the cost of manual annotation and improving data processing efficiency.

[0021] Therefore, the embodiment of the present application proposes a method for assisting the labeling of medical named entities based on model integration and large model review, which can assist the recognition and labeling process of each medical named entity in the medical text. Specifically, it can be achieved by using models of different scales to work together to identify the medical named entities, and combine each recognition result for comprehensive evaluation and optionally further combine confidence evaluation, so as to determine whether further manual review is required based on the evaluation results. For medical named entities identified by the model that require manual review, after manual review, they can have more accurate labeling results; for medical named entities identified by the model that do not require manual review, they are directly labeled based on the recognition results of the model and the labeling results are output, thereby improving labeling efficiency and saving manpower.

[0022] Figure 1 An example architecture diagram of an auxiliary annotation solution for medical named entities according to an embodiment of the present application is shown.

[0023] like Figure 1 As shown, the original medical text is input into multiple models to realize annotation (model labeling) using the multiple models, and the final annotation result is output by fusing the recognition results (annotation results) of each model. The multiple models or other models mentioned in this article refer to various models that have been trained to perform named entity recognition to realize annotation, and they can have different scales and different functions and capabilities. For example, multiple models can include large models, medium models, small models and regular word list models. The large model can be a model with a parameter amount greater than 1 billion, which is a large language model based on the standard Transformer architecture, such as the LLAMA-70b model, which can analyze the medical text in detail and output the entity recognition result in JSON format. The medium model can be a trained model with a parameter amount of 100 million to 10 million levels, which can perform entity recognition and annotation on medical texts, for example, the Bert architecture can be used. The small model can be a model with a parameter amount of less than 10 million levels, for example, the conditional random field (CRF) architecture can be used, which can perform entity recognition and annotation on medical texts. In addition, the rule-based vocabulary model is a trained model based on vocabulary matching or rule implementation, which can also perform named entity recognition and labeling on medical texts.

[0024] In addition, a classification model can be used to improve the accuracy of recognition, as will be further described below. A classification model is a model that can classify a specific text to determine whether its content belongs to a corresponding category, and it can use classification algorithms such as logistic regression, naive Bayes, decision tree, support vector machine, random forest, gradient boosting tree, etc.

[0025] Each model mentioned in this application can be stored in the form of computer executable instructions or computer programs, and when called and executed by a processor, it can realize its corresponding functions, such as recognition and classification.

[0026] exist Figure 1 In the architecture diagram shown, by integrating the recognition results of models of different sizes, it is possible to comprehensively consider entity recognition information from multiple angles, improve the overall accuracy and reliability, and use the powerful summarization, logical understanding and reasoning capabilities of the large model to evaluate the medical named entities recognized by the small model, and determine whether annotators are needed for further review based on the evaluation results. Therefore, it can provide efficient assistance to annotators to generate high-quality training data. In addition, compared with large models, small models have a larger throughput and less demanding hardware requirements, so entity annotations can be generated mainly based on the recognition results of small models (the computing power and hardware requirements required for the large model review process are smaller than those required for the annotation generation process).

[0027] The following is combined with Figure 2-6 The solution of the embodiment of the present application based on the example architecture diagram is introduced in detail.

[0028] Figure 2 A flow chart of a method for assisting in labeling medical named entities according to an embodiment of the present application is shown.

[0029] like Figure 2 As shown, in step S210, at least a first model and a second model are used to respectively recognize the medical text to determine a first score corresponding to each medical named entity recognized by the first model.

[0030] According to an example of the present invention, the medical text associated with the user may be text data describing the user's medical treatment, disease diagnosis, examination, medical history, etc. In one example, the medical text associated with the patient may be the user's medical history, prescription, examination report, etc. For example, a medical text may include the following content: After the patient was admitted to the hospital and completed the preoperative examination, he underwent right upper lobectomy plus partial right middle lobe and fifth posterior rib resection plus lymph node dissection under general anesthesia on July 17, 2021. ........ PTNM: PT3N1 (considering pulmonary primary, please combine clinical). Immunohistochemistry results showed: CK5 / 6 (3+), P40 (3+), P63 (3+), TTF-1 (-), CDX-2 (-), SatB2 (focal weak +), P16 (-), Ki-67 (+80%), EBER-.

[0031] In some cases, only the first model and the second model may be used to identify the medical text, thereby obtaining a first score corresponding to each medical named entity. Alternatively, in other cases, a model other than the first model and the second model may be used to identify the medical text, thereby obtaining a first score corresponding to each medical named entity identified by the first model (i.e., the number of models is greater than or equal to 3).

[0032] Optionally, the first model can be a small model, and the second model can be a medium model. For example, the first model can be used to identify medical text to obtain a first set of medical named entities. Generally, the number of medical named entities identified by the small model is the highest, the range of identified medical named entities is also the widest, and therefore the recognition accuracy is relatively low. In addition, the second model can also be used to identify medical text to obtain a second set of medical named entities. Then, the medical named entities in the first set that are also included in the second set (that is, the medical named entities included in both the first set and the second set) are determined as having the highest first score (for example, 100 points), and the medical named entities included in the first set but not included in the second set are determined as having a lower first score (for example, 50 points).

[0033] As an example, each medical named entity identified by the first model is associated with a corresponding category for the medical named entity identified by the first model. For example, the medical named entity can be divided into a surgical name category (which may include multiple subcategories) and an immunohistochemistry result category (which may include multiple subcategories), and so on.

[0034] For example, for the above-mentioned medical text, a part of the first set of medical named entities identified by the first model and their corresponding first scores can be shown as follows, where each piece of data includes the content of the identified medical named entity (i.e., the corresponding text, for example, right upper lobectomy), its position in the medical text, a category label indicating the corresponding category, and other information.

[0035] {"end":"36","entity_id":"2","label":"L0577","start":"30","word":"Right upper lobectomy"},

[0036] {"end":"36","entity_id":"2","label":"L0577","start":"30","word":"all numb"},

[0037] {"end":"50","entity_id":"4","label":"L0577","start":"37","word":"Partial resection of the right middle lobe and the fifth posterior rib"},

[0038] {"end":"56","entity_id":"8","label":"L0577","start":"51","word":"Lymph node dissection"},

[0039] {"end":"504","entity_id":"84","label":"L0573","start":"499","word":"ck5 / 6"},

[0040] {"end":"507","entity_id":"85","label":"L0574","start":"505","word":"3+"},

[0041] {"end":"512","entity_id":"86","label":"L0573","start":"509","word":"p40"},

[0042] …

[0043] {"end":"530","entity_id":"102","label":"L0573","start":"525","word":" pt3n1"},

[0044] {"end":"532","entity_id":"91","label":"L0574","start":"531","word":"-"},

[0045] …

[0046] For example, for the above medical text, a part of the second set of medical named entities identified by the second model may be, for example:

[0047] {"end":"36","entity_id":"2","label":"L0577","start":"30","word":"Right upper lobectomy"},

[0048] {"end":"50","entity_id":"4","label":"L0577","start":"37","word":"Partial resection of the right middle lobe and the fifth posterior rib"},

[0049] {"end":"56","entity_id":"8","label":"L0577","start":"51","word":"Lymph node dissection"},

[0050] {"end":"504","entity_id":"84","label":"L0573","start":"499","word":"ck5 / 6"},

[0051] {"end":"507","entity_id":"85","label":"L0574","start":"505","word":"3+"},

[0052] {"end":"512","entity_id":"86","label":"L0573","start":"509","word":"p40"},

[0053] …

[0054] {"end":"532","entity_id":"91","label":"L0574","start":"531","word":"-"},

[0055] …

[0056] In the above, for example, the medical named entity "general anesthesia" labeled as the surgical name category (label code L0577) and the medical named entity "pt3n1" labeled as the gene name category (which can be regarded as a subcategory of the immunohistochemistry result category) in the first set of medical named entities are only included in the first recognition result of the small model-the first model, so it has a lower first score; in addition, the remaining medical named entities in the first set are also included in the second set, so the remaining medical named entities have a higher first score.

[0057] In addition, when more (greater than or equal to 3) models including the first model and the second model are used to recognize medical texts respectively, multiple sets (greater than or equal to 3 sets) of medical named entities can be obtained, that is, multiple models each correspond to a set of medical named entities recognized. Then, the medical named entities in the first set that are also included in all other sets (that is, all sets) are determined as having the highest level of first score (for example, 100 points), the medical named entities in the first set that are included in sets whose number is greater than or equal to the first threshold number and less than the total number of multiple sets are determined as having the second highest level of first score (for example, 80 points), and the medical named entities in the first set that are included in sets whose number is less than the first threshold number are determined as having the lowest level of first score (for example, 30 points).

[0058] For example, when the number of the multiple models is 3, the first threshold number may be 2; when the number of the multiple models is 4, the first threshold number may be 2 or 3; or when the number of the multiple models is 5, the first threshold number may be 3, etc. The present disclosure does not limit the specific values ​​of the number of the multiple models and the first threshold number.

[0059] For example, in addition to the first model and the second model, a rule vocabulary model can also be used to identify medical named entities in medical texts to obtain a third set of medical named entities. The rule vocabulary model is a trained model based on vocabulary matching or rule implementation. Similarly, the first score of each medical named entity in the first set can be determined based on the intersection of the medical named entities in the first set, the second set, and the third set. In addition, since there may be certain relationships between the various medical named entities, a relationship model for determining the relationship between entities can also be used to identify and annotate medical named entities in medical texts.

[0060] In step S220, the medical text is recognized using a third model having a higher recognition accuracy than at least the first model and the second model to obtain a reference recognition result.

[0061] For example, as mentioned above, the large model has stronger summarization, logical understanding and reasoning capabilities than the small model and the medium model, and can review the medical named entities identified by the small model, thereby providing efficient assistance to the annotators. Therefore, the third model can be such a large model.

[0062] Optionally, the reference recognition result of the large model for the medical text may include multiple events for the medical text and corresponding multiple key entity information.

[0063] Optional, for example, Figure 3 The reference recognition results of the large model for the above-mentioned medical text are shown in FIG, for example, text output in JSON format.

[0064] exist Figure 3 In the data, "operation date", "operation name", "anesthesia method" and "immunohistochemistry results" are events, and the corresponding "July 17, 2021", "right upper lobectomy plus partial right middle lobe and fifth posterior rib resection plus lymph node dissection" and "general anesthesia" are the specific contents corresponding to the corresponding events, which can be stored in a format with thick entities so that they can be used for display. They are all information closely related to the medical named entities in the medical text, and are recognition results with relatively high recognition accuracy, so they can be used as a reference for further evaluation of the recognition results of the small model.

[0065] Back to Figure 2In step S230, based on the first recognition result of the first model, the reference recognition result, and the first score corresponding to each medical named entity recognized by the first model, an evaluation score corresponding to each medical named entity recognized by the first model is determined.

[0066] For example, an additional score for each medical named entity identified by the first model can be determined based on the first recognition result and the reference recognition result, and then the corresponding first score and the additional score of each medical named entity identified by the first model are added to obtain a corresponding evaluation score.

[0067] In other examples, considering that the medical named entity may be a medical named entity in the text of efficacy evaluation, previous radiotherapy, etc., and the classification model can determine the category of the content according to the content to be classified and its context, the classification model can be introduced to determine the confidence of each medical named entity identified by the first model belonging to the corresponding category it is identified. In this way, the classification model can be used in combination with the small model to determine the category of the medical named entity whose recognition accuracy is not high enough, so as to determine the confidence that the medical named entity belongs to the corresponding category identified by the small model.

[0068] In this way, for each medical named entity identified by the first model, the text corresponding to the medical named entity can be classified using the classification model (for example, in combination with the context of the text content corresponding to the medical named entity in the medical text) to determine the confidence that the medical named entity belongs to the corresponding category identified by the first model. Then, based on the corresponding first score of each medical named entity identified by the first model and the confidence of belonging to the corresponding category, the corresponding confidence score of each medical named entity identified by the first model is obtained, wherein the confidence score is used to adjust the corresponding first score for subsequent determination of the corresponding evaluation score.

[0069] For example, the confidence is a value between 0 and 1, and the corresponding confidence score of each medical named entity identified by the first model is obtained by multiplying the corresponding first score and the confidence of the corresponding category. The confidence score can be regarded as a more accurate version of the first score.

[0070] Therefore, in this case, an additional score for each medical named entity identified by the first model can be calculated based on the first recognition result and the reference recognition result, and then the corresponding confidence score of each medical named entity identified by the first model is added to the additional score to obtain a corresponding evaluation score.

[0071] Optionally, when calculating the additional score for each medical named entity identified by the first model based on the first recognition result and the reference recognition result, the following operations can be performed: for each medical named entity identified by the first model, based on the multiple events for the medical text included in the reference recognition result and the corresponding event specific content, determine the inclusion relationship between the reference recognition result and the medical named entity; and based on the inclusion relationship, determine the corresponding additional score of the medical named entity.

[0072] For example, for each medical named entity identified by the first model, when the reference recognition result completely includes the medical named entity, the corresponding additional score of the medical named entity is the highest additional score; when the reference recognition result partially includes the medical named entity, the corresponding additional score of the medical named entity is the second highest additional score; and when the reference recognition result does not include the medical named entity, the corresponding additional score of the medical named entity is the lowest additional score.

[0073] For example, as mentioned above, the reference recognition results may include multiple events associated with each medical named entity and their respective specific contents (also called event attributes), each event may be a collection of entities (the number of entities may be one or more, and each entity represents an attribute of the event), for example, a drug treatment event may include multiple entities (drug name, drug category, frequency of use, dosage, etc.). Figure 3 As shown, "anesthesia method" is an event, and its specific content "general anesthesia" is an event attribute, which can be regarded as an entity; "immunohistochemistry results" can be an event, and the specific content of the event includes multiple event attributes, each of which can be considered as an identified entity. The entities included in the reference recognition results (entities recognized by the large model) are coarse entities, and their granularity is larger than that of the small model. For example, a coarse entity recognized by the large model can correspond to multiple fine-grained entities of the small model. As an example, the entity "breast cancer" corresponding to the "diagnosis" event recognized by the large model can correspond to two entities identified by the small model indicating the location in the breast and the diagnosis of cancer.

[0074] If the reference recognition result completely includes a medical named entity recognized by the first model, for example, the medical named entity "right upper lobectomy" is extracted and recognized as the category of "surgery name (label L0577)", Figure 3The reference recognition result shown includes the event of the surgery name and the corresponding specific content (i.e., event attribute, entity) is "right upper lobectomy plus partial right middle lobe and fifth posterior rib resection plus lymph node dissection", that is, the reference recognition result completely includes "right upper lobectomy", so the corresponding additional score of the medical named entity is the highest additional score, such as 100 points. For another example, if the medical named entity "general anesthesia" is extracted and identified as the category of "surgery name (label L0577)", but Figure 3 The reference recognition results shown include that the specific content corresponding to the event of the surgery name does not include "general anesthesia", so the corresponding additional score of the medical named entity is the lowest additional score, such as 0 points; or the medical named entity "pt3n1" is extracted and identified as the category of "gene name (label)", but Figure 3 The reference recognition results shown do not include the event of gene name, so it is also considered that the reference recognition results do not include the medical named entity. In addition, in some other cases, Figure 3 The specific content corresponding to the event of the surgical name included in the reference recognition result shown may only include a part of the extracted and identified medical named entity. For example, the medical named entity "right upper lobectomy" is extracted and identified as the category of "surgical name (label L0577)", and the reference recognition result includes the event of the surgical name and the corresponding specific content (i.e., event attributes, entity) is "upper lobectomy", then the corresponding additional score of the medical named entity is the second highest additional score, for example 80 points.

[0075] In this way, an additional score for each medical named entity identified by the first model can be obtained, which can be added to the first score or the confidence score to determine the evaluation score for each medical named entity identified by the first model.

[0076] In step S240, the labeling result of each medical named entity included in the recognition result of the first model is output based on the corresponding evaluation score of each medical named entity recognized by the first model.

[0077] For example, for each identified medical named entity, the annotation result of the medical named entity is displayed in one of a plurality of display formats based on the corresponding evaluation score of the medical named entity, wherein different display formats correspond to evaluation scores in different score ranges.

[0078] For example, medical named entities with different evaluation scores can be displayed in different colors, different shapes and / or different font sizes, such as using gray to display medical named entities with high evaluation scores, using red to display medical named entities with low evaluation scores, and using black to display text that is not recognized by the model.

[0079] In addition, for each identified medical named entity, if the corresponding evaluation score of the medical named entity is less than the score threshold, the labeling result indicating the medical named entity and the indication that manual review is required are output. On the contrary, if the corresponding evaluation score of the medical named entity is greater than or equal to the score threshold, the labeling result is determined directly based on the recognition result of the identified medical named entity, for example, its position or category in the text, etc. The specific content that needs to be labeled for the medical named entity can be set according to actual needs and the information contained in the recognition result.

[0080] Optionally, when the evaluation scores of the medical named entities identified by the first model are generally higher than the set score threshold, it can be considered that no manual review is required, and the data can be directly used as a training sample for the first model to achieve automatic replacement of manual annotations.

[0081] The annotated medical named entities obtained by the above method can be used to train various related models, such as the above-mentioned first model, second model, rule vocabulary model and third model, as well as other related relational models such as those used to determine the relationship between entities.

[0082] It can be seen that in the reference Figure 2-3 In the method described for assisting the annotation of medical named entities, by using models of different scales to work together and integrating the recognition results of models of different scales, it is possible to comprehensively consider entity recognition information from multiple angles, thereby improving the overall accuracy and reliability, and utilizing the powerful summarization, logical understanding and reasoning capabilities of the large model to score the medical named entities identified by the small model to determine the final output evaluation score, which can be used to determine whether the corresponding identified medical named entity requires further manual review. For the identified medical named entities that require manual review, after manual review, they can have more accurate annotation results; for the identified medical named entities that do not require manual review, it is determined that the annotation results of the medical named entities identified by the model are accurate (credible). Therefore, the workload of medical text annotation can be significantly reduced, and the efficiency and quality of data processing can be improved. Optionally, when the evaluation score of the medical named entity identified by the first model is generally higher than the set score threshold, it can be considered that no manual review is required, and the data can be directly used as a training sample for the first model to achieve automatic replacement of manual annotation.

[0083] Figure 4 A schematic diagram of a process for assisting in labeling medical named entities is shown. Figure 2-3 The method described corresponds to .

[0084] like Figure 4As shown, the small model (first model), the medium model (second model) and the rule vocabulary model respectively recognize the input original medical text to obtain the first recognition result Z1, the second recognition result Z2 and the third recognition result Z3. The first recognition result Z1 and the second recognition result Z2 can be as shown in the previous reference Figure 2 The storage and / or display form shown as an example in the description of step S210. The storage and / or display form of the third recognition result Z3 may be the same as that of the first recognition result Z1 and the second recognition result Z2, or may be other forms.

[0085] Then, according to the output results of the middle model, the small model and the regular vocabulary model, each medical named entity identified by the small model is scored (first score). The scoring strategy is: for each medical named entity identified by the small model, if the medical named entity is recognized by all three models, it is given 100 points, if it is recognized by two models, it is given 80 points, and if it is recognized by only one model, it is given 30 points.

[0086] In addition, the classification model can classify each medical named entity in the first recognition result Z1, that is, to judge whether the medical named entity belongs to a specific text. The classification model will judge whether the content recognized by the small model is accurate in combination with the context, and give a confidence level of 0 to 1 for each medical named entity. The classification model can be a 0-1 classification model. For example, the small model recognizes that the category of the medical named entity "general anesthesia" is the name of a surgery. The classification model can give the probability of judging that the text content "general anesthesia" belongs to the category of the surgery name based on the content of the medical named entity "general anesthesia" and its context and this recognition result. For example, the probability value is 0.5, that is, the confidence level is 0.5.

[0087] Then, for each medical named entity identified by the small model, the confidence associated with the category (e.g., surgery name) corresponding to the identified medical named entity output by the classification model can be combined with the corresponding first score to form a confidence score (second score). The calculation formula is: confidence score = first score * confidence.

[0088] In addition, as mentioned above, the large model (third model) with higher recognition accuracy is also used to recognize the original medical text to obtain Figure 3 The reference recognition result ZR shown. On the other hand, the reference recognition result ZR of the large model can also be used to score the recognition result (first recognition result Z1) of the small model to obtain an additional score using the large model.

[0089] Therefore, if the improvement effect of the classification model is not considered, for each medical named entity recognized by the small model, the first score corresponding to the medical named entity and the additional score obtained using the large model are added together to obtain the final evaluation score; if the improvement effect of the classification model is considered, for each medical named entity recognized by the small model, the confidence score corresponding to the medical named entity (as above, obtained from the first score and the corresponding confidence) and the additional score obtained using the large model are added together to obtain the final evaluation score.

[0090] In this way, for each medical named entity identified by the small model, if the final evaluation score of the medical named entity is greater than or equal to the score threshold, it means that the accuracy of the medical named entity identified by the first model (small model) is high enough, so its labeling result can be obtained based on the recognition result of the medical named entity. On the contrary, if the final evaluation score of the medical named entity is less than the score threshold, it means that the accuracy of the medical named entity identified by the first model (small model) is not enough and manual labeling is required, so the manual labeling result for the medical named entity can be obtained and stored.

[0091] According to another aspect of the present application, a device for assisting in labeling medical named entities is also provided.

[0092] Figure 5 A structural block diagram of a device for assisting in labeling medical named entities is shown.

[0093] like Figure 5 As shown, the apparatus 500 may include a first recognition module 510 , a second recognition module 520 , an evaluation module 530 , and an output module 540 .

[0094] The first recognition module 510 can be used to recognize medical text using at least a first model and a second model respectively to determine a first score corresponding to each medical named entity recognized by the first model.

[0095] For example, the recognition results of the first model include a first set of medical named entities, and the recognition results of the second model include a second set of medical named entities. In this case, the medical named entities in the first set that are also included in the second set can be determined as having the highest level of first scores, and the medical named entities included in the first set but not included in the second set can be determined as having a lower level of first scores.

[0096] Alternatively, when more than three models (medium model and small model) are used to recognize medical text, the medical named entities in the first set recognized by the first model and included in the sets corresponding to multiple models are determined as having the highest first score, the medical named entities in the first set included in the sets whose number is greater than or equal to the first threshold number and less than the total number of multiple sets are determined as having the second highest first score, and the medical named entities recognized by the first model and included in the sets whose number is less than the first threshold number are determined as having the lowest first score.

[0097] The second recognition module 520 can be used to recognize the medical text using a third model having a higher recognition accuracy than at least the first model and the second model to obtain a reference recognition result.

[0098] For example, the third model may be a large model.

[0099] For example, the reference recognition result of the third model includes multiple events for medical texts and corresponding event specific contents, such as Figure 3 shown.

[0100] The evaluation module 530 may be configured to determine an evaluation score corresponding to each medical named entity identified by the first model based on the first recognition result of the first model, the reference recognition result, and the first score corresponding to each medical named entity identified by the first model.

[0101] Optionally, each medical named entity identified by the first model is associated with a corresponding category, and the device 500 may also include a classification module for classifying the text corresponding to each medical named entity identified by the first model using the classification model to determine the confidence that the medical named entity belongs to the corresponding category identified by the first model, and based on the corresponding first score of each medical named entity identified by the first model and the confidence that it belongs to the corresponding category, obtain the corresponding confidence score of each medical named entity identified by the first model.

[0102] For example, when determining the evaluation score, the evaluation module can be configured to determine an additional score (i.e., the judgment score of the large model) for each medical named entity identified by the first model based on the first recognition result and the reference recognition result, and then add the corresponding first score or confidence score (considering the classification model) of each medical named entity identified by the first model to the additional score to obtain the corresponding evaluation score.

[0103] The output module 540 may be configured to output a labeling result of each medical named entity included in the recognition result of the first model based on the second score evaluation score corresponding to each medical named entity recognized by the first model.

[0104] For example, for each identified medical named entity, the annotation result of the medical named entity is displayed in one of a plurality of display formats based on the corresponding evaluation score of the medical named entity, wherein different display formats correspond to evaluation scores in different score ranges.

[0105] Additionally or alternatively, for each identified medical named entity, when the corresponding evaluation score of the medical named entity is less than a score threshold, a labeling result indicating the medical named entity and an indication that manual review is required are output.

[0106] More details of the operations performed by the various modules in the apparatus 500 have been previously described in reference to Figure 1-4 The contents of the description are described in , so they will not be repeated here.

[0107] In addition, although Figure 5 The above modules and submodules are shown in an exemplary manner, but it should be understood that the device 500 can be divided into more or fewer modules according to different functions, or each module can be divided into further more or fewer submodules. For example, the first identification module 510 can be the same module as the second identification module 520, and / or the evaluation module 530 can also include a determination submodule and a scoring submodule, etc. In some example embodiments, the module or its submodule can be implemented with electronic hardware (e.g., a general-purpose processor, DSP, ASIC, FPGA or other programmable logic device, discrete gate or transistor logic, discrete hardware components, etc.), computer software (e.g., can be stored in random access memory (RAM), flash memory, read-only memory (ROM), erasable programmable ROM (EPROM), etc.), or a combination of the two.

[0108] In reference Figure 5 In the device described for assisting in the labeling of medical named entities, by using models of different scales to work together and integrating the recognition results of models of different scales, it is possible to comprehensively consider entity recognition information from multiple angles, thereby improving the overall accuracy and reliability, and utilizing the powerful summarization, logical understanding and reasoning capabilities of the large model to evaluate the medical named entities identified by the small model, for determining the final output evaluation score, which can be used to determine whether the corresponding identified medical named entity requires further manual review. For the identified medical named entities that require manual review, after manual review, they can have more accurate labeling results; for the identified medical named entities that do not require manual review, it is determined that the labeling results of the medical named entities identified by the model are accurate (credible). Therefore, the workload of medical text labeling can be significantly reduced, and the efficiency and quality of data processing can be improved.

[0109] Figure 6 A structural diagram of a computing device according to an embodiment of the present application is shown.

[0110] See also Figure 6 , the computing device 600 may include one or more processors 601 and one or more memories 602 connected to the one or more processors 601. The processors 601 and the memories 602 may be connected via a bus 603. The computing device 600 may be any type of portable device (such as a smart camera, a smart phone, a tablet computer, etc.) or any type of fixed device (such as a desktop computer, a server, etc.).

[0111] The processor 601 can perform various actions and processes according to the computer program stored in the memory 602. Specifically, the processor 601 can be an integrated circuit chip with signal processing capabilities. The above processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The disclosed methods, steps and logic block diagrams in the embodiments of the present invention can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc., which can be an X86 architecture or an ARM architecture.

[0112] The memory 602 stores computer executable instructions, and when the computer executable instructions are executed by the processor 601, the above-mentioned method for assisting the annotation of medical named entities is implemented. The memory 602 can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. The non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM) or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of exemplary but not limiting explanation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous connection dynamic random access memory (SLDRAM) and direct memory bus random access memory (DR RAM). It should be noted that the memory of the method described herein is intended to include but is not limited to these and any other suitable categories of memory.

[0113] In addition, the method for identifying medical named entities according to the present invention can be recorded in a non-transitory computer-readable recording medium. Specifically, according to the present invention, a non-transitory computer-readable recording medium storing computer-executable instructions or computer programs can be provided, and when the computer-executable instructions or computer programs are executed by a processor, the processor can be prompted to execute the method for assisting in labeling medical named entities as described above.

[0114] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment, or a part of a code, and the module, program segment, or a part of a code contains at least one executable instruction for realizing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0115] In general, various example embodiments of the present invention may be implemented in hardware or dedicated circuits, software, firmware, logic, or any combination thereof. Certain aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device. When various aspects of embodiments of the present invention are illustrated or described as block diagrams, flow charts, or using some other graphical representation, it will be understood that the blocks, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general purpose hardware or controllers or other computing devices, or some combination thereof as non-limiting examples.

[0116] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the invention belongs. It should also be understood that terms such as those defined in common dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology and should not be interpreted in an idealized or extremely formal sense, unless explicitly defined as such herein.

[0117] The above is an explanation of the present invention and should not be considered as a limitation thereof. Although several exemplary embodiments of the present invention have been described, it will be readily understood by those skilled in the art that many modifications may be made to the exemplary embodiments without departing from the novel teachings and advantages of the present invention. Therefore, all such modifications are intended to be included within the scope of the present invention as defined in the claims. It should be understood that the above is an explanation of the present invention and should not be considered as being limited to the specific embodiments disclosed, and modifications to the disclosed embodiments and other embodiments are intended to be included within the scope of the appended claims. The present invention is defined by the claims and their equivalents.

Claims

1. A method for assisting in labeling medical named entities, comprising: Recognize the medical text using at least a first model and a second model respectively to determine a first score corresponding to each medical named entity recognized by the first model; Recognize the medical text using a third model to obtain a reference recognition result, wherein the third model has a higher recognition accuracy than the at least first model and second model; Determine an evaluation score corresponding to each medical named entity identified by the first model based on the first recognition result of the first model, the reference recognition result, and a first score corresponding to each medical named entity identified by the first model; as well as outputting the labeling result of each medical named entity included in the recognition result of the first model based on the corresponding evaluation score of each medical named entity recognized by the first model, Wherein, determining the corresponding evaluation score of each medical named entity identified by the first model comprises: determining an additional score for each medical named entity recognized by the first model based on the first recognition result and the reference recognition result; and Based on the corresponding first score and the additional score of each medical named entity identified by the first model, a corresponding evaluation score is obtained. The reference recognition result of the third model includes multiple events and corresponding event specific contents for the medical text. The step of determining an additional score for each medical named entity identified by the first model comprises: for each medical named entity identified by the first model, Based on the multiple events and the corresponding event specific contents, determining the inclusion relationship between the reference recognition result and the medical named entity; and Based on the inclusion relationship, an additional score corresponding to the medical named entity is determined.

2. The method according to claim 1, wherein: Each medical named entity identified by the first model is associated with a corresponding category identified by the first model, The method further comprises: For each medical named entity identified by the first model, classify the text corresponding to the medical named entity using the classification model to determine the confidence that the medical named entity belongs to the corresponding category identified by the first model; and Based on the first score corresponding to each medical named entity identified by the first model and the confidence of belonging to the corresponding category, determine the confidence score corresponding to each medical named entity identified by the first model, wherein the confidence score is used to adjust the corresponding first score to determine the corresponding evaluation score.

3. The method according to claim 1 or 2, wherein: The first recognition result of the first model includes a first set of medical named entities, and the recognition result of the second model includes a second set of medical named entities; Wherein, determining a first score corresponding to each medical named entity identified by the first model comprises: determining the medical named entities included in both the first set and the second set as having a first score with the highest rank, and Medical named entities included by the first set but not included by the second set are determined to have a first score of a lower rank.

4. The method according to claim 1 or 2, wherein: The at least first model and the second model are a plurality of models whose number is greater than or equal to 3, and the plurality of models each correspond to a set of identified medical named entities, thereby corresponding one-to-one with the plurality of sets; Wherein, determining a first score corresponding to each medical named entity identified by the first model comprises: Determine the medical named entities in the first set identified by the first model and included in the sets corresponding to the multiple models as having a first score with the highest level; determining, in the first set, medical named entities included in sets whose number is greater than or equal to a first threshold number and less than a total number of the plurality of sets as having a second highest first score; and Medical named entities identified by the first model and included in the set whose number is less than the first threshold number are determined as having a first score with a lowest rank.

5. The method according to claim 2, wherein: Using a classification model to classify the text corresponding to the medical named entity to determine the confidence that the medical named entity belongs to the corresponding category, including: The classification model is used to determine the confidence that the medical named entity belongs to its corresponding category according to the context of the text corresponding to the medical named entity.

6. The method according to claim 2, wherein: The confidence level is a value between 0 and 1. The corresponding first score of each medical named entity identified by the first model and the confidence score belonging to the corresponding category are multiplied to obtain the corresponding confidence score of the medical named entity.

7. The method according to claim 1, wherein: Based on the corresponding first score and the additional score of each medical named entity identified by the first model, a corresponding evaluation score is obtained, including: The corresponding first score and the additional score of each medical named entity identified by the first model are added together to obtain a corresponding evaluation score.

8. The method according to claim 2, wherein: Based on the corresponding first score and the additional score of each medical named entity identified by the first model, a corresponding evaluation score is obtained, including: The corresponding confidence score and the additional score of each medical named entity identified by the first model are added together to obtain a corresponding evaluation score.

9. The method according to claim 1, wherein: Based on the inclusion relationship, determining the corresponding additional score of the medical named entity includes: In the case where the reference recognition result completely includes the medical named entity, the additional score corresponding to the medical named entity is the highest additional score; In the case where the reference recognition result partially includes the medical named entity, the additional score corresponding to the medical named entity is the second highest additional score; and In the case that the reference recognition result does not include the medical named entity, the additional score corresponding to the medical named entity is the lowest additional score.

10. The method according to claim 1, wherein: Outputting the labeling result of each medical named entity included in the recognition result of the first model based on the corresponding evaluation score of each recognized medical named entity includes: For each medical named entity identified by the first model, displaying the annotation result of the medical named entity in one of a plurality of display formats based on the corresponding evaluation score of the medical named entity, Different display formats correspond to evaluation scores in different score ranges.

11. The method according to claim 1, outputting the labeling result of each medical named entity included in the recognition result of the first model based on the corresponding evaluation score of each recognized medical named entity, comprising: For each medical named entity identified by the first model, if the corresponding evaluation score of the medical named entity is less than a score threshold, output a labeling result indicating the medical named entity and an indication that manual review is required.

12. The method according to claim 1, wherein: The first model includes models with a parameter amount within 10 million or implemented based on rule matching, the second model includes models with a parameter amount between 100 million and 10 million, and / or the third model includes models with a parameter amount exceeding 1 billion.

13. A device for assisting in labeling medical named entities, comprising: A first recognition module, configured to recognize medical text using at least a first model and a second model respectively, to determine a first score corresponding to each medical named entity recognized by the first model; A second recognition module, configured to recognize the medical text using a third model having a higher recognition accuracy than the at least first model and the second model, to obtain a reference recognition result; an evaluation module, configured to determine an evaluation score corresponding to each medical named entity identified by the first model based on the first recognition result of the first model, the reference recognition result, and a first score corresponding to each medical named entity identified by the first model; as well as an output module, configured to output a labeling result of each medical named entity based on a second score evaluation score corresponding to each medical named entity identified by the first model, Wherein, when determining the evaluation score corresponding to each medical named entity identified by the first model, the determination module is configured as follows: determining an additional score for each medical named entity recognized by the first model based on the first recognition result and the reference recognition result; and Based on the corresponding first score and the additional score of each medical named entity identified by the first model, a corresponding evaluation score is obtained. The reference recognition result of the third model includes multiple events and corresponding event specific contents for the medical text. The step of determining an additional score for each medical named entity identified by the first model comprises: for each medical named entity identified by the first model, Based on the multiple events and the corresponding event specific contents, determining the inclusion relationship between the reference recognition result and the medical named entity; and Based on the inclusion relationship, an additional score corresponding to the medical named entity is determined.

14. A computing device comprising: one or more processors; as well as One or more memories are coupled to the one or more processors and store a computer program, which, when executed by the one or more processors, causes the one or more processors to perform the method according to any one of claims 1 to 12.

15. A non-transitory computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by one or more processors, the one or more processors are caused to perform the method according to any one of claims 1 to 12.

Images