A training method and device of a recognition model, an electronic device, and a storage medium
By filtering and correcting the medical order data of the target hospital, and utilizing multiple recognition models and medical knowledge graphs, the problems of target domain labeled data selection and labeling errors were solved, thereby improving the model's generalization ability and accuracy.
Patent Information
- Application Number
- CN202211146072.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-09-20
- Publication Date
- 2025-12-30
- Estimated Expiration
- 2042-09-20
AI Technical Summary
Existing technologies do not address the problem of how to select valuable target domain labeled data. Labeling errors lead to a decrease in model accuracy, and the lack of diversity in training data results in a lack of model generalization ability.
By acquiring the initial unlabeled dataset of the target hospital, named entity recognition is performed using multiple original recognition models. Medical order data with high confidence or consistent entity names are selected to generate the target unlabeled dataset. Error data analysis and medical knowledge graphs are used to correct the data, and finally the recognition model is iteratively trained.
It improved the model's generalization ability, corrected labeling errors, enhanced the diversity of training data, and improved the model's accuracy.
Smart Images
Figure CN115456199B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing technology, and in particular to a method, apparatus, electronic device and storage medium for training a recognition model. Background Technology
[0002] It does not address how to select valuable target domain labeled data; it does not address the issues of labeling errors in labeled data leading to decreased model accuracy, and the lack of training data diversity leading to a lack of model generalization ability. Summary of the Invention
[0003] In order to solve the above-mentioned technical problems, or at least partially solve the above-mentioned technical problems, this application provides a training method, apparatus, electronic device and storage medium for a recognition model.
[0004] According to one aspect of the embodiments of this application, a method for training a recognition model is provided, characterized in that it includes:
[0005] Obtain the initial unlabeled dataset corresponding to the target hospital, wherein the target hospital is the hospital for which knowledge transfer is to be performed, and the initial unlabeled dataset includes at least one medical order data;
[0006] Obtain the recognition results obtained by performing named entity recognition on the medical order data in the initial dataset to be labeled, and use the target medical order data that meets the preset conditions to generate the target dataset to be labeled.
[0007] The detection process applies to the annotation operation corresponding to the target dataset to be annotated, obtains the target annotation data corresponding to each target medical order data in the target dataset to be annotated, and generates the test dataset corresponding to the target hospital based on the target medical order data and the target annotation data;
[0008] The test dataset is predicted using the recognition model to be trained to obtain the prediction result. If the prediction result does not meet the preset conditions, iterative training is performed based on the test dataset and the recognition model to be trained until the trained recognition model meets the preset conditions.
[0009] Furthermore, the step of obtaining the recognition result obtained by performing named entity recognition on the medical order data in the initial dataset to be labeled, and generating the target dataset to be labeled using the target medical order data that meets the preset conditions based on the recognition result, includes:
[0010] Two original recognition models corresponding to the original hospital are obtained, wherein the original recognition models are trained using the original medical order data of the original hospital;
[0011] The original recognition model is used to perform named entity recognition on the medical order data in the initial dataset to be labeled, and the recognition results are obtained.
[0012] The medical order data whose recognition results meet the preset conditions are identified as the target medical order data, and data to be labeled is generated based on the target medical order.
[0013] Furthermore, the original recognition model includes: a first original recognition model and a second original recognition model;
[0014] The process of using the original recognition model to perform named entity recognition on the medical order data in the initial dataset to be labeled, and obtaining the recognition results, includes:
[0015] Using the first original recognition model, the medical order data in the initial dataset to be labeled are used to obtain the first entity name corresponding to each medical order data and the confidence level corresponding to the first entity name;
[0016] Using the second original recognition model, the medical order data in the initial dataset to be labeled is used to obtain the second entity name corresponding to each medical order data.
[0017] Furthermore, the step of determining the medical order data whose identification results meet preset conditions as the target medical order data, and generating data to be labeled based on the target medical order, includes:
[0018] The medical order data with a confidence level lower than the preset confidence level is determined as the first target medical order data, and the medical order data with the second entity name inconsistent with the first entity name is determined as the second target medical order data;
[0019] The data to be labeled is generated based on the first target medical order data and the second target medical order data.
[0020] Furthermore, the detection is applied to the annotation operation corresponding to the target dataset to be annotated, to obtain the target annotation data corresponding to each target medical order data in the target dataset to be annotated, including:
[0021] The detection process applies to the annotation operations corresponding to each target medical order in the target dataset to be annotated, and determines the initial annotation data based on the annotation operations.
[0022] The initial annotation data is used to detect first annotation data whose boundaries are inconsistent with the original annotation data of the original hospital using a detection tool for identifying named entity boundaries;
[0023] The target medical order data is predicted using a medical order classification model to obtain the prediction result, and the initial annotation information that is inconsistent with the prediction result is used as the second annotation data.
[0024] Based on the first annotation data and the second annotation data, the erroneous annotation data in the initial annotation data is obtained, and the correct annotation data corresponding to the erroneous annotation data is obtained;
[0025] The incorrectly labeled data is replaced with the correctly labeled data to obtain the target labeled data.
[0026] Furthermore, the step of iteratively training the recognition model based on the test dataset and the model to be trained until the trained recognition model meets the preset conditions when the prediction result does not meet the preset conditions includes:
[0027] Based on the prediction results, obtain the model prediction error dataset and suspicious error annotation data;
[0028] Obtain corrected data by performing a correction operation on the predicted error dataset and the suspected error labeled data;
[0029] The test dataset is updated using the corrected data to obtain the updated test dataset. The recognition model to be trained is then iteratively trained using the updated test dataset until the recognition model meets the preset conditions.
[0030] Furthermore, the corrected data obtained by performing a correction operation on the predicted error dataset and the suspected error-labeled data includes:
[0031] Detect annotation operations applied to suspected erroneous labeled data, and obtain corrected suspected erroneous labeled data based on the annotation operations;
[0032] Based on the medical knowledge graph, the synonyms and subordinate entities of the erroneous entities in the prediction error dataset are obtained. The prediction error dataset is expanded using the synonyms and subordinate entities of each erroneous entity to obtain the target prediction error dataset. The target error dataset is then corrected to obtain the corrected dataset.
[0033] The longest common subsequence algorithm is used to select multiple candidate medical order data from the unlabeled medical order data corresponding to the target hospital. The similarity between these candidate medical order data and the erroneous medical order data in the predicted error dataset is greater than a preset threshold.
[0034] If the candidate medical order data meets the preset requirements, the candidate medical order data will be added to the correction data set as pseudo-label data.
[0035] According to another aspect of the embodiments of this application, a training apparatus for a recognition model is also provided, comprising:
[0036] The first acquisition module is used to acquire the initial unlabeled dataset corresponding to the target hospital, wherein the target hospital is the hospital for which knowledge transfer is to be performed, and the initial unlabeled dataset includes at least one medical order data.
[0037] The second acquisition module is used to acquire the recognition result obtained by performing named entity recognition on the medical order data in the initial dataset to be labeled, and to generate the target dataset to be labeled using the target medical order data that meets the preset conditions based on the recognition result.
[0038] The generation module is used to detect the annotation operations applied to the target dataset to be annotated, obtain the target annotation data corresponding to each target medical order data in the target dataset to be annotated, and generate the test dataset corresponding to the target hospital based on the target medical order data and the target annotation data;
[0039] The training module is used to predict the test dataset using the recognition model to be trained, obtain the prediction result, and perform iterative training based on the test dataset and the recognition model to be trained if the prediction result does not meet the preset conditions, until the trained recognition model meets the preset conditions.
[0040] According to another aspect of the embodiments of this application, a storage medium is also provided, the storage medium including a stored program that executes the above steps when the program is run.
[0041] According to another aspect of the embodiments of this application, an electronic device is also provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; wherein: the memory is used to store computer programs; and the processor is used to execute the steps in the above method by running the programs stored in the memory.
[0042] This application also provides a computer program product containing instructions that, when run on a computer, cause the computer to perform the steps in the above-described method.
[0043] Compared with the prior art, the technical solution provided in this application has the following advantages: This application obtains the annotation information of medical order data, and through error data analysis, corrects the error data by using model back-labeling, and finally uses the corrected data to train the model, thereby improving the generalization ability of the model. Attached Figure Description
[0044] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0045] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0046] Figure 1 A flowchart illustrating a training method for a recognition model provided in an embodiment of this application;
[0047] Figure 2 A block diagram of a training device for a recognition model provided in an embodiment of this application;
[0048] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0049] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. The illustrative embodiments of this application and their descriptions are used to explain this application and do not constitute an improper limitation of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0050] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another similar entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0051] This application provides a training method, apparatus, electronic device, and storage medium for a recognition model. The method provided by this invention can be applied to any electronic device as needed, such as a server, terminal, or other electronic device. No specific limitation is made here, and for ease of description, it will be referred to as an electronic device below.
[0052] According to one aspect of the embodiments of this application, a method embodiment for training a recognition model is provided. Figure 1 A flowchart illustrating a training method for a recognition model provided in this application embodiment is shown below. Figure 1 As shown, the method includes:
[0053] Step S11: Obtain the initial unlabeled dataset corresponding to the target hospital, wherein the target hospital is the hospital for which knowledge transfer is to be performed, and the initial unlabeled dataset includes at least one medical order data.
[0054] The method provided in this application embodiment is applied to a smart terminal capable of data processing, which may be a computer, smartphone, tablet computer, etc.
[0055] Step S12: Obtain the recognition results obtained by performing named entity recognition on the medical order data in the initial dataset to be labeled, and use the target medical order data that meets the preset conditions to generate the target dataset to be labeled.
[0056] In this embodiment of the application, the recognition results obtained by performing named entity recognition on the medical order data in the initial dataset to be labeled are obtained, and the target medical order data that meets the preset conditions is used to generate the target dataset to be labeled, including the following steps A1-A3:
[0057] Step A1: Obtain two original recognition models corresponding to the original hospital. The original recognition models are trained using the original medical order data of the original hospital.
[0058] Step A2: Use the original recognition model to perform named entity recognition on the medical order data in the initial dataset to be labeled, and obtain the recognition results.
[0059] In this embodiment of the application, the original recognition model includes: a first original recognition model and a second original recognition model.
[0060] Step A2 involves using the original recognition model to perform named entity recognition on the medical order data in the initial dataset to be labeled, and obtaining the recognition results. This includes the following steps: using the first original recognition model to obtain the first entity name corresponding to each medical order data and the confidence level corresponding to the first entity name on the medical order data in the initial dataset to be labeled; and using the second original recognition model to obtain the second entity name corresponding to each medical order data on the medical order data in the initial dataset to be labeled.
[0061] In this embodiment, the original recognition model corresponding to the original hospital is obtained. The original recognition model includes a first recognition model Model1 and a second recognition model Model2. Specifically, the original hospital's medical order data is used as the original medical order data, and Model1 and Model2 are trained based on the original medical order data using the recognition model to be trained + CRF and the recognition model to be trained + SPAN methods, respectively.
[0062] In this embodiment, named entity recognition is performed on the medical order data in the initial dataset to be labeled based on the first recognition model Model1 and the second recognition model Model2, respectively, to obtain recognition results Result1 and Result2. Specifically, the first entity name predicted by the first recognition model Model1 from the medical order data, and the confidence level of the first entity name are obtained.
[0063] Step A3: Identify the medical order data that meet the preset conditions as the target medical order data, and generate the data to be labeled based on the target medical order.
[0064] In this embodiment of the application, medical order data whose identification results meet preset conditions are identified as target medical order data, and data to be labeled is generated based on the target medical order data, including: identifying medical order data with a confidence level lower than a preset confidence level as first target medical order data, and identifying medical order data with a second entity name inconsistent with the first entity name as second target medical order data; generating data to be labeled based on the first target medical order data and the second target medical order data.
[0065] In this embodiment, if the confidence level output by the first recognition model based on the medical order data is lower than the threshold ζ, then the medical order data is the first target medical order data. For example, comparing the softmax score of "chest CT plain scan" with the threshold ζ, the softmax values are: 0.99, 0.99, 0.99, 0.99, 0.94, 0.94, and 0.12. In this case, the medical order data with a value lower than the threshold ζ = 0.94 is selected as the first target medical order data. Then, the second entity name of the medical order data predicted by the second recognition model Model2 is obtained. The medical order data whose second entity name is inconsistent with the first entity name is determined as the second target medical order data. Finally, the first target medical order data and the second target medical order data are used to generate the data to be labeled.
[0066] It should be noted that by using a pre-trained model to predict unlabeled data from new hospitals, and then filtering the data samples based on the prediction results, multiple datasets to be labeled are obtained. These datasets can then be given to doctors for labeling.
[0067] Step S13: Detect the annotation operation applied to the target dataset to be annotated, obtain the target annotation data corresponding to each target medical order data in the target dataset to be annotated, and generate the test dataset corresponding to the target hospital based on the target medical order data and the target annotation data.
[0068] In this embodiment of the application, the detection of the annotation operation applied to the target dataset to be annotated, and the obtaining of the target annotation data corresponding to each target medical order data in the target dataset to be annotated, includes the following steps B1-B5:
[0069] Step B1: Detect the annotation operation applied to each target medical order data in the target dataset to be annotated, and determine the initial annotation data based on the annotation operation.
[0070] Step B2: Use a detection tool for identifying named entity boundaries to detect the first labeled data whose boundaries are inconsistent with the original labeled data of the original hospital.
[0071] Step B3: Use the medical order classification model to predict the target medical order data, obtain the prediction results, and use the initial annotation information that is inconsistent with the prediction results as the second annotation data.
[0072] Step B4: Based on the first annotation data and the second annotation data, obtain the erroneous annotation data in the initial annotation data, and obtain the correct annotation data corresponding to the erroneous annotation data;
[0073] Step B5: Replace the incorrectly labeled data with the correctly labeled data to obtain the target labeled data.
[0074] In this embodiment of the application, (1) the Named Entity Recognition Boundary Inconsistency Detection Tool is used to detect the annotation information Dataset of the original hospital medical order NER. labeled_old Dataset with annotation information of target hospital labeled_new (2) Using a medical order classification model to analyze the boundary inconsistencies of the data set, the first labeled data with inconsistent boundaries between the initial labeled data and the original labeled data of the original hospital was obtained. labeled_new The medical order content is used to predict the order category, and the prediction result is used to indicate whether the predicted category matches the label of the medical order entity. If they do not match, the initial labeling information that is inconsistent with the prediction result is used as the second labeling data. Finally, based on the first and second labeling data, the erroneous labeling data in the initial labeling data is obtained.
[0075] In this embodiment of the application, the corrections in steps (1) and (2) result in the corrected annotation information Dataset. labeled_newThe correct annotation information is used to replace the incorrect annotation data to obtain the target annotation data.
[0076] Step S14: Use the recognition model to be trained to predict the test dataset and obtain the prediction result. If the prediction result does not meet the preset conditions, iteratively train the recognition model based on the test dataset and the recognition model to be trained until the trained recognition model meets the preset conditions.
[0077] In this embodiment of the application, if the prediction result does not meet the preset conditions, iterative training is performed based on the test dataset and the recognition model to be trained until the trained recognition model meets the preset conditions, including the following steps C1-C3:
[0078] Step C1-: Obtain the model prediction error dataset and suspected error annotation data based on the prediction results.
[0079] Step C2: Obtain the corrected data obtained by performing correction operations on the prediction error dataset and the suspected error labeled data.
[0080] Step C3: Update the test dataset using the corrected data to obtain the updated test dataset. Iteratively train the recognition model to be trained using the updated test dataset until the recognition model meets the preset conditions.
[0081] In this embodiment of the application, the corrected data obtained by performing correction operations on the prediction error dataset and the suspected error labeled data includes the following steps D1-D4:
[0082] Step D1: Detect the annotation operations applied to the suspected erroneous labeled data, and obtain the corrected suspected erroneous labeled data based on the annotation operations.
[0083] Step D2: Based on the medical knowledge graph, obtain the synonyms and subordinate entities of the erroneous entities in the prediction error dataset. Expand the prediction error dataset using the synonyms and subordinate entities of each erroneous entity to obtain the target prediction error dataset. Then, correct the target error dataset to obtain the corrected dataset.
[0084] Step D3: The longest common subsequence algorithm is used to select multiple candidate medical order data from the unlabeled medical order data corresponding to the target hospital, whose similarity to the erroneous medical order data in the prediction error dataset is greater than a preset threshold.
[0085] If the candidate medical order data meets the preset requirements, the candidate medical order data will be added to the correction data set as pseudo-label data.
[0086] In this embodiment, erroneous entities are extracted from the predicted error dataset. These entities are then expanded using synonyms and subordinate terms based on a medical knowledge graph. The expanded entities are matched with entities in the test set to correct any errors in entity boundaries or labels. For example, if the erroneous entity "chest CT" is incorrectly predicted as "operation" and labeled as "examination," the expanded entity becomes "chest CT," "left chest CT," or "CT (chest) plain scan." If the training data (Train) contains these entities, such as the order "Today's CT (chest) plain scan," and if "CT (chest) plain scan" is predicted as "operation," the label is corrected to "examination."
[0087] In this embodiment, after obtaining the corrected data set, for each incorrectly predicted medical order text, the longest common subsequence algorithm is used to select N medical order data with similarity greater than a threshold from the new hospital's unlabeled medical order data. The current medical order NER model and dictionary back-labeling method are used to predict the N medical order data, which are then submitted to labelers for random checks. If the checks meet the requirements, the predicted pseudo-label data is added to the corrected data set.
[0088] The method provided in this application uses confidence and model voting to filter valuable labeled data; it uses model back-labeling through error data analysis and medical knowledge graphs to help correct errors in the training dataset; and it uses semi-supervised methods to augment data for scenarios not covered by the model, thereby improving the model's generalization ability.
[0089] Figure 2 This is a block diagram of a training device for a recognition model provided in an embodiment of this application. This device can be implemented as part or all of an electronic device through software, hardware, or a combination of both. Figure 2 As shown, the device includes:
[0090] The first acquisition module 21 is used to acquire the initial unlabeled dataset corresponding to the target hospital, wherein the target hospital is the hospital for which knowledge transfer is to be performed, and the initial unlabeled dataset includes at least one medical order data.
[0091] The second acquisition module 22 is used to acquire the recognition results obtained by performing named entity recognition on the medical order data in the initial dataset to be labeled, and to generate the target dataset to be labeled using the target medical order data that meets the preset conditions based on the recognition results.
[0092] The generation module 23 is used to detect the annotation operation applied to the target dataset to be annotated, obtain the target annotation data corresponding to each target medical order data in the target dataset to be annotated, and generate the test dataset corresponding to the target hospital based on the target medical order data and the target annotation data.
[0093] The training module 24 is used to predict the test dataset using the recognition model to be trained, obtain the prediction result, and perform iterative training based on the test dataset and the recognition model to be trained if the prediction result does not meet the preset conditions, until the trained recognition model meets the preset conditions.
[0094] In this embodiment of the application, the second acquisition module 22 is used to acquire two original recognition models corresponding to the original hospital. The original recognition models are trained using the original medical order data of the original hospital. The original recognition models are used to perform named entity recognition on the medical order data in the initial dataset to be labeled to obtain the recognition results. The medical order data whose recognition results meet the preset conditions are determined as the target medical order data, and the data to be labeled is generated based on the target medical order.
[0095] In this embodiment of the application, the original recognition model includes: a first original recognition model and a second original recognition model;
[0096] The second acquisition module 22 is used to obtain the first entity name corresponding to each medical order data and the confidence level corresponding to the first entity name by using the first original recognition model to process the medical order data in the initial dataset to be labeled; and to obtain the second entity name corresponding to each medical order data by using the second original recognition model to process the medical order data in the initial dataset to be labeled.
[0097] In this embodiment of the application, the second acquisition module 22 is used to determine the medical order data with a confidence level lower than a preset confidence level as the first target medical order data, and to determine the medical order data with a second entity name that is inconsistent with the first entity name as the second target medical order data; and to generate data to be labeled based on the first target medical order data and the second target medical order data.
[0098] In this embodiment, the generation module 23 is used to detect the annotation operation applied to each target medical order data in the target to be annotated dataset, and determine the initial annotation data based on the annotation operation; use a detection tool for identifying named entity boundaries to detect the first annotation data whose boundaries are inconsistent with the original annotation data of the original hospital; use a medical order classification model to predict the target medical order data, obtain the prediction result, and use the initial annotation information that is inconsistent with the prediction result as the second annotation data; obtain the erroneous annotation data in the initial annotation data based on the first annotation data and the second annotation data, and obtain the correct annotation data corresponding to the erroneous annotation data; replace the erroneous annotation data with the correct annotation data to obtain the target annotation data.
[0099] In this embodiment, the training module 24 is used to obtain a model prediction error dataset and suspicious error labeling data based on the prediction results; obtain corrected data by performing correction operations on the prediction error dataset and suspicious error labeling data; update the test dataset using the corrected data to obtain the updated test dataset; and iteratively train the recognition model to be trained using the updated test dataset until the recognition model meets the preset conditions.
[0100] In this embodiment, the training module 24 is used to detect annotation operations applied to suspicious erroneous labeled data, and obtain corrected suspicious erroneous labeled data based on the annotation operations; it obtains synonyms and subordinate entities of erroneous entities in the predicted erroneous dataset based on the medical knowledge graph, expands the predicted erroneous dataset using the synonyms and subordinate entities of each erroneous entity to obtain the target predicted erroneous dataset, and corrects the target erroneous dataset to obtain the corrected dataset; it uses the longest common subsequence algorithm to select multiple candidate medical order data from the unlabeled medical order data corresponding to the target hospital that have a similarity greater than a preset threshold with the erroneous medical order data in the predicted erroneous dataset; if the candidate medical order data meets the preset requirements, it adds the candidate medical order data as pseudo-label data to the corrected dataset.
[0101] This application also provides an electronic device, such as... Figure 3 As shown, the electronic device may include: a processor 1501, a communication interface 1502, a memory 1503, and a communication bus 1504, wherein the processor 1501, the communication interface 1502, and the memory 1503 communicate with each other through the communication bus 1504.
[0102] Memory 1503 is used to store computer programs;
[0103] When the processor 1501 executes the computer program stored in the memory 1503, it implements the steps of the above embodiments.
[0104] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.
[0105] The communication interface is used for communication between the aforementioned terminal and other devices.
[0106] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.
[0107] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.
[0108] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores instructions that, when executed on a computer, cause the computer to perform the training method of any of the recognition models described in the above embodiments.
[0109] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute the training method of any of the recognition models described in the above embodiments.
[0110] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state drive).
[0111] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.
[0112] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.
Claims
1. A method for training a recognition model, characterized by, The method comprises the following steps: obtaining an initial annotation data set corresponding to a target hospital, wherein the target hospital is a hospital to which knowledge migration is to be performed, and the initial annotation data set comprises at least one medical order data; obtaining an identification result obtained by performing named entity recognition on the medical order data in the initial annotation data set, and generating a target annotation data set by using target medical order data satisfying a preset condition in the identification result; detecting a labeling operation acting on the target annotation data set to obtain target labeling data corresponding to each target medical order data in the target annotation data set, and generating a test data set corresponding to the target hospital based on the target medical order data and the target labeling data; using a to-be-trained identification model to predict the test data set to obtain a prediction result, and performing iterative training based on the test data set and the to-be-trained identification model until the trained identification model satisfies the preset condition in the case that the prediction result does not satisfy the preset condition; the step of obtaining an identification result obtained by performing named entity recognition on the medical order data in the initial annotation data set, and generating a target annotation data set by using target medical order data satisfying a preset condition in the identification result comprises the following steps: obtaining two original identification models corresponding to an original hospital, wherein the original identification models are trained by using original medical order data of the original hospital; performing named entity recognition on the medical order data in the initial annotation data set by using the original identification models to obtain an identification result; determining the medical order data satisfying a preset condition in the identification result as the target medical order data, and generating annotation data based on the target medical order; the original identification models comprise a first original identification model and a second original identification model; the step of performing named entity recognition on the medical order data in the initial annotation data set by using the original identification models to obtain an identification result comprises the following steps: performing named entity recognition on the medical order data in the initial annotation data set by using the first original identification model to obtain a first entity name corresponding to each medical order data and a confidence degree corresponding to the first entity name; performing named entity recognition on the medical order data in the initial annotation data set by using the second original identification model to obtain a second entity name corresponding to each medical order data; the step of determining the medical order data satisfying a preset condition in the identification result as the target medical order data, and generating annotation data based on the target medical order comprises the following steps: determining medical order data with a confidence degree less than a preset confidence degree as first target medical order data, and determining medical order data with a second entity name inconsistent with the first entity name as second target medical order data; generating the annotation data based on the first target medical order data and the second target medical order data; and The detection acts on the labeling operation corresponding to the target to-be-labeled data set, and obtains target labeling data corresponding to each target medical order data in the target to-be-labeled data set, including: detecting the labeling operation acting on each target medical order data in the target to-be-labeled data set, and determining initial labeling data based on the labeling operation; using a detection tool for identifying the boundary of a named entity to detect first labeling data with inconsistent boundaries in the initial labeling data and the original labeling data of the original hospital; using a medical order classification model to predict the target medical order data to obtain a prediction result, and taking the initial labeling information inconsistent with the prediction result as second labeling data; obtaining error labeling data in the initial labeling data based on the first labeling data and the second labeling data, and obtaining correct labeling data corresponding to the error labeling data; replacing the error labeling data with the correct labeling data to obtain the target labeling data.
2. The method of claim 1, wherein, In the case that the prediction result does not satisfy the preset condition, the test data set and the to-be-trained identification model are iteratively trained until the trained identification model satisfies the preset condition, including: Based on the prediction result, a model prediction error data set and suspicious error labeling data are obtained; Obtain the correction data obtained by performing a correction operation on the prediction error data set and the suspicious error labeling data; Update the test data set using the correction data to obtain an updated test data set, and iteratively train the to-be-trained identification model using the updated test data set until the identification model satisfies the preset condition.
3. The method of claim 2, wherein, The correction data obtained by performing a correction operation on the prediction error data set and the suspicious error labeling data, including: Detect the labeling operation acting on the suspicious error labeling data, and obtain the corrected suspicious error labeling data based on the labeling operation; Based on the medical knowledge graph, obtain the synonyms and lower entities of the error entities in the prediction error data set, expand the prediction error data set using the synonyms and lower entities of each error entity to obtain a target error data set, and correct the target error data set to obtain a correction data set; Use the longest common subsequence algorithm to select multiple candidate medical order data with a similarity greater than a preset threshold from the unlabeled medical order data corresponding to the target hospital in the prediction error data set; In the case that the candidate medical order data satisfies the preset requirement, the candidate medical order data is added to the correction data set as pseudo-label data.
4. A training apparatus of a recognition model characterized by comprising: Including: The first acquisition module is used for acquiring an initial to-be-labeled data set corresponding to a target hospital, wherein the target hospital is a hospital to be subjected to knowledge migration, and the initial to-be-labeled data set includes at least one medical order data; The second acquisition module is used for acquiring an identification result obtained by performing named entity recognition on the medical order data in the initial to-be-labeled data set, and generating a target to-be-labeled data set using target medical order data satisfying a preset condition. The generating module is configured to detect a labeling operation acting on the target to-be-labeled data set, obtain target labeling data corresponding to each target medical order data in the target to-be-labeled data set, and generate a test data set corresponding to the target hospital based on the target medical order data and the target labeling data; The training module is configured to use the to-be-trained identification model to predict the test data set, obtain a prediction result, and perform iterative training based on the test data set and the to-be-trained identification model until the trained identification model meets the preset condition, in a case where the prediction result does not meet the preset condition. The obtaining of the identification result of the named entity recognition of the medical order data in the initial to-be-labeled data set and the generation of a target to-be-labeled data set using the target medical order data satisfying a preset condition include: obtaining two original identification models corresponding to an original hospital, wherein the original identification models are trained using original medical order data of the original hospital; performing named entity recognition on the medical order data in the initial to-be-labeled data set using the original identification models to obtain an identification result; determining the medical order data satisfying a preset condition as the target medical order data, and generating to-be-labeled data based on the target medical order; The original identification models include a first original identification model and a second original identification model. The obtaining of the identification result of the named entity recognition of the medical order data in the initial to-be-labeled data set using the original identification models includes: obtaining, using the first original identification model, a first entity name corresponding to each medical order data and a confidence degree corresponding to the first entity name; and obtaining, using the second original identification model, a second entity name corresponding to each medical order data. The determination of the medical order data satisfying a preset condition as the target medical order data and the generation of to-be-labeled data based on the target medical order include: determining medical order data with a confidence degree less than a preset confidence degree as first target medical order data, and determining medical order data with a second entity name inconsistent with the first entity name as second target medical order data; and generating the to-be-labeled data based on the first target medical order data and the second target medical order data. The detection acts on the labeling operation corresponding to the target to-be-labeled data set, to obtain target labeling data corresponding to each target medical order data in the target to-be-labeled data set, comprising: detecting the labeling operation acting on each target medical order data in the target to-be-labeled data set, and determining initial labeling data based on the labeling operation; using a detection tool for recognizing the boundary of a named entity to detect first labeling data in which the boundary of the initial labeling data is inconsistent with the original labeling data of the original hospital; using a medical order classification model to predict the target medical order data to obtain a prediction result, and taking the initial labeling information inconsistent with the prediction result as second labeling data; obtaining error labeling data in the initial labeling data based on the first labeling data and the second labeling data, and obtaining correct labeling data corresponding to the error labeling data; replacing the error labeling data with the correct labeling data to obtain the target labeling data.
5. A storage medium, characterized by The storage medium includes a stored program, wherein the program performs the method steps of any one of claims 1 to 3 when running.
6. An electronic device, comprising: It comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete mutual communication through the communication bus; wherein: The memory is used to store computer programs; The processor is used to execute the method steps of any one of claims 1 to 3 by running the programs stored in the memory.
Citation Information
Patent Citations
System for automatically extracting medical advices and application
CN111209924A
Entity recognition model generation method and device and computer readable storage medium
CN112420205A