Automatic verification of medical data

Abstract

Embodiments of the present disclosure relate to automatic verification of medical data. Some embodiments of the present disclosure provide a method for medical data verification. The method includes obtaining target medical data generated in a medical test, and obtaining a machine learning model for verifying the medical data. The machine learning model represents an association between medical data and a verification result, which indicates information related to a predetermined action to be performed on the medical data. The method also includes determining a target verification result for the target medical data by applying the target medical data to the machine learning model, the target verification result indicating information related to a target action to be performed on the target medical data, the target action being selected from a predetermined action. Through this scheme, automatic medical data verification can be achieved with high accuracy, high efficiency, and less manual work.

Description

Technical Field

[0001] Embodiments of the present disclosure relate generally to the field of computer science, and in particular to methods, apparatus, and computer program products for automatic validation of medical data. Background Art

[0002] Medical tests are performed almost every day in medical laboratories, so a large number of medical test reports are generated to present medical data. Before a medical test report is issued to a clinical department or patient, a validation process is initiated to ensure that the medical data presented in the report is valid to avoid misdiagnosis of the patient. However, even if some automatic functions have been introduced, a lot of work is still required in the current validation process.

[0003] Currently, rule-based engines are used to validate medical test reports. The rule-based engine is configured to verify whether the report can be directly delivered to the clinical department or the patient by determining whether the test results in the medical test report meet the predetermined rules. Medical test reports that do not meet one or more predetermined rules will be provided to laboratory experts for manual review with their medical experience and knowledge.

[0004] Even with the support of current rule-based functions, verification of medical data must be labor-intensive. In addition to human resource costs, the accuracy and efficiency of verification may vary depending on the experience and expertise of laboratory technicians. Therefore, it is desirable to obtain a solution for automatic verification of medical data to achieve high accuracy and efficiency with less manual work. Summary of the invention

[0005] In general, example embodiments of the present disclosure provide a solution for automatic verification of medical data.

[0006] In a first aspect, a method for medical data verification is provided. The method includes obtaining target medical data generated in a medical test, and obtaining a machine learning model for verifying the medical data. The machine learning model represents an association between the medical data and a verification result, and the verification result indicates information related to a predetermined action to be performed on the medical data. The method also includes determining a target verification result for the target medical data by applying the target medical data to the machine learning model, the target verification result indicating information related to a target action to be performed on the target medical data, the target action being selected from the predetermined action.

[0007] In a second aspect, a method for providing a machine learning model for validating medical data is provided. The method includes: obtaining training data, the training data including historical medical data and associated annotation information. The annotation information indicates a predetermined action performed on the historical medical data. The method also includes: generating a first machine learning model for validating the medical data based on the training data, so that the first machine learning model represents an association between the medical data and a verification result, the verification result indicating information related to the predetermined action to be performed on the medical data.

[0008] In a third aspect, an electronic device is provided, comprising at least one processor and at least one memory comprising computer-readable instructions, wherein the computer-readable instructions, when executed by the at least one processor of the electronic device, cause the electronic device to perform the steps of the method in the first aspect.

[0009] In a fourth aspect, an electronic device is provided, comprising: at least one processor and at least one memory comprising computer-readable instructions, wherein the computer-readable instructions, when executed by the at least one processor of the electronic device, enable the electronic device to perform the steps of the method in the second aspect.

[0010] In a fifth aspect, a computer program product is provided, wherein the computer program product comprises instructions, and when the instructions are executed by a processor of a device, the device is caused to perform the steps of any one of the methods in the first aspect.

[0011] In a sixth aspect, a computer program product is provided, wherein the computer program product comprises instructions, and when the instructions are executed by a processor of a device, the device executes the steps of any one of the methods in the second aspect.

[0012] It should be understood that the invention summary is not intended to identify the key or essential features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The following detailed description of embodiments of the invention may be better understood when read in conjunction with the following drawings, in which:

[0014] Figure 1 illustrates an example environment in which embodiments of the present disclosure may be implemented;

[0015] Figure 2 illustrates a block diagram of a system for medical data verification according to some embodiments of the present disclosure;

[0016] Figure 3 The diagram illustrates a method for training according to some embodiments of the present disclosure. Figure 2a block diagram of a computing device for one or more machine learning models in a system;

[0017] Figure 4 illustrates example clustering results for a historical medical data set according to some embodiments of the present disclosure;

[0018] Figure 5 The present invention is illustrated in some embodiments of the present invention. Figure 2 a block diagram of a computing device for one or more machine learning models in a system;

[0019] Figure 6 illustrates an example medical test report according to some embodiments of the present disclosure;

[0020] Figure 7 FIG. 1 illustrates a method for training according to some other example embodiments of the present disclosure. Figure 2 a block diagram of a computing device for one or more machine learning models in a system;

[0021] Figure 8 illustrates a block diagram of a system according to some embodiments of the present disclosure, wherein one or more machine learning models are compatible with a rule-based engine and a laboratory information system (LIS);

[0022] Fig. 9 illustrates a flow chart of an example process for validating medical data according to some embodiments of the present disclosure;

[0023] Fig.10 illustrates a flowchart of an example process for providing a machine learning model for validating medical data according to some embodiments of the present disclosure; and

[0024] Fig.11 A block diagram of an example computing system / device suitable for implementing example embodiments of the present disclosure is illustrated.

[0025] Throughout the drawings, the same or similar reference numerals denote the same or similar elements. DETAILED DESCRIPTION

[0026] The principle of the present disclosure will now be described with reference to some embodiments. It should be understood that these embodiments are described only for the purpose of explanation and to help those skilled in the art understand and implement the present disclosure, rather than to imply any limitation on the scope of the present disclosure. The disclosure described herein can be implemented in various ways different from the following description.

[0027] In the following description and claims, unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

[0028] In this disclosure, references to "one embodiment," "an embodiment," "an example embodiment," etc. indicate that the described embodiment may include a particular feature, structure, or characteristic, but not every embodiment necessarily includes the particular feature, structure, or characteristic. Moreover, such phrases do not necessarily refer to the same embodiment. In addition, when a particular feature, structure, or characteristic is described in conjunction with an example embodiment, it is considered to be within the knowledge of those skilled in the art to affect such feature, structure, or characteristic in conjunction with other embodiments, whether or not such a combination is explicitly described.

[0029] It should be understood that although the terms "first" and "second" etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, without departing from the scope of the exemplary embodiments, the first element may be referred to as the second element, and similarly, the second element may be referred to as the first element. As used herein, the term "and / or" includes any and all combinations of one or more of the listed terms.

[0030] The terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the exemplary embodiments. As used herein, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are also intended to include plural forms. It will also be understood that when used herein, the terms "include", "included", "have", "have", "include" and / or "included" specify the presence of stated features, elements and / or components, etc., but do not exclude the presence or increase of one or more other features, elements, components and / or combinations thereof.

[0031] Example Environment

[0032] As mentioned above, the validation process is important to ensure the validity of the medical data generated in various medical tests. Figure 1 An environment 100 in which an example embodiment of the present disclosure may be implemented is illustrated. The environment 100 relates to a typical workflow for medical diagnostic testing. The workflow typically includes performing a medical test on a test sample for medical diagnosis, generating medical data in the medical test, and verifying the generated medical data.

[0033] exist Figure 1In the environment 100 shown, the medical test system 110 is configured to perform medical tests on the test sample 102 and generate medical data 112 associated with the test sample 102. The medical test may include an in vitro diagnostic test, such as a biochemical detection test or an immunoassay test. The medical test system 110 may include one or more automatic laboratory instruments or analytical devices, which are designed to analyze the test sample by various chemical, biological, physical or other medical testing procedures. In some examples, the instrument or analytical device may be configured to cause the reaction of the sample with the reagent to obtain a measurement value. Examples of such instruments or analytical devices are clinical chemistry analyzers, coagulation analyzers, immunochemical analyzers, hematology analyzers, urine analyzers and nucleic acid analyzers, which are used to qualitatively and / or quantitatively detect analytes present in the sample to detect the results of a chemical or biological reaction and / or monitor the progress of a chemical or biological reaction.

[0034] The medical testing system 110 may be operable to perform a medical test to measure a parameter of a sample or at least one of its analytes. The medical test may involve one or more test items performed on the sample 102. The medical testing system 110 may return test results corresponding to the various test items as medical data 112. Possible test results returned by the medical testing system 110 may be obtained by determining the concentration of the analyte in the sample, a numerical (yes or no) result indicating the presence of the analyte in the sample (corresponding to a concentration above the detection level), data obtained from mass spectrometry of proteins or metabolites, and various types of physical, mechanical, optical, electrical or chemical parameters, etc.

[0035] Some specific examples of test item types may include levels of alanine aminotransferase (ALT), aspartate aminotransferase (AST), glutamate dehydrogenase (GLDH), sodium concentration (NA), age, hemoglobin, plasma protein, albumin (ALB), globulin (GLB), total bilirubin (TBIL), direct bilirubin (DBIL), total bile acid (TBA), blood urea nitrogen (BUN), etc. The examples listed here are not exhaustive. The test items to be performed in a specific medical test may be specified by the entity requesting the medical test, such as an outpatient department, a physical examination center, a doctor, a patient, etc.

[0036] The test sample 102 may also be referred to as a biological sample, which is a biological material suspected of containing one or more analytes of interest, and whose detection, characterization and / or quantification may be associated with a clinical condition. The biological sample is from a biological source, such as a physiological fluid, including blood, saliva, lens fluid, cerebrospinal fluid, sweat, urine, feces, semen, milk, ascites fluid, mucus, synovial fluid, peritoneal fluid, amniotic fluid, tissue, cells, etc. Such a biological source may be collected from a biological subject, such as a patient, a human, an animal, etc.

[0037] The biological sample can be pre-treated before use, such as preparing plasma or serum from blood. The method of treatment can involve centrifugation, filtration, distillation, dilution, concentration and / or separation of sample components (including the analyte of interest), inactivation of interfering components, and addition of reagents. The biological sample can be used directly after being obtained from the source, or it can be used after a pre-treatment to change the characteristics of the sample. In some embodiments, the original solid or semi-solid biological material can be made liquid by dissolving or suspending the original solid or semi-solid biological material using a suitable liquid medium.

[0038] The term "reagent" refers to a substance that is added to a biological sample when a specific medical test is performed on the biological sample to cause a specific reaction in the sample. The reagent may be specific for a specific test or determination. For example, in a situation where the partial thrombosis time of a blood sample should be determined, the analyzer may be configured to add an activator as a reagent to the blood sample to activate the intrinsic pathway of coagulation. In different situations, the specific substance may be a "modifier" or a "reagent". In some examples, the reagent may not be added to the biological sample to be tested.

[0039] The medical data 112 associated with the test sample 102 may include one or more test results of a test item implemented in a medical test at the medical testing system 110. The type of test result may be specified by an operator of the medical testing system 110 (e.g., a laboratory technician), or automatically identified from an electronic application form via an information system connected to the medical testing system 110. In some examples, the medical data 112 may be organized in a medical test report, with specific test items and corresponding test results listed on the medical test report. In some examples, in addition to the test results generated in the medical test, the medical data 112 may also include auxiliary information, such as information related to the test sample 102 and / or information related to a biological object (such as a patient) from which the test sample 102 was collected.

[0040] The medical data 112 is provided to a verification system 120 to evaluate the validity of the medical data 112 and determine whether the medical data 112 can be released. Verification is required because many problems may occur during the sample collection and testing process. For example, a patient sample may be mislabeled, resulting in a reported test result being associated with the wrong patient. As another example, a patient sample may be incorrectly drawn or improperly processed, resulting in sample contamination and erroneous test results. In addition, a laboratory analyzer may malfunction or deviate from a calibration state, also resulting in the analyzer reporting an erroneous result.

[0041] The verification system 120 provides a verification result 122 for the medical data 112. If the medical data 112 passes the verification, the verification result 122 may indicate that the medical data 112 is valid enough to be directly released (or released after a simple manual review) to the requester of the medical test. If the medical data 112 is determined to be suspicious or abnormal, the verification result 122 may indicate that the medical data 112 has not passed the verification. In this case, further action may need to be taken on the medical data 112. The verification result 122 and the medical data 112 can be provided to an information system such as a laboratory information system (LIS) for archiving, recording, presentation, further processing, etc. Depending on the verification result 122, subsequent manual work may or may not be involved.

[0042] In the current verification process, rule-based engines have been proposed to verify medical test reports. The rule-based engine is configured to verify whether the medical test report can be directly delivered to the clinical department or the patient by determining whether the test results in the report meet predetermined rules. These rules are usually set to standard value ranges for various test items. If one or more predetermined rules are not met, the medical test report will be provided to laboratory experts for manual review using their medical experience and knowledge. The entire procedure is prone to errors and incurs high human resource / time consumption. Therefore, it is desirable to obtain a solution for automatic verification of medical data to achieve high accuracy and efficiency with less manual work.

[0043] Working Principle and Example Verification System

[0044] According to an example embodiment of the present disclosure, a scheme for automatic verification of medical data is proposed. In the scheme, one or more machine learning models are learned from historical medical data and associated annotation information indicating predetermined actions performed on the historical medical data. The machine learning model represents the association between the medical data and the verification result. One or more machine learning models can be used to verify the target medical data and output a target verification result, which indicates information related to an action to be performed on the target medical data in the predetermined action. Through this scheme, information can be learned from historical data to achieve highly automatic verification of medical data, which can significantly reduce the manual work expended in reviewing medical data and improve the accuracy and quality of medical data verification.

[0045] Hereinafter, exemplary embodiments of the present disclosure are described with reference to the accompanying drawings. Figure 2 Reference is made to which illustrates a system for medical data verification according to some embodiments of the present disclosure. Figure 2 The system in can be implemented as Figure 1The verification system 120 in the embodiment of the present invention is used in the verification phase of the entire workflow for medical diagnostic testing. The verification system 120 can also be referred to as an automatic verification system 120.

[0046] According to an embodiment of the present disclosure, one or more machine learning techniques are used to automatically analyze and verify medical data with human-like accuracy. Machine learning techniques may also be referred to as artificial intelligence (AI) techniques. Typically, a machine learning model may be established that receives input information and makes predictions based on the input information. For example, a classification model may predict the category of input information from a predetermined set of categories.

[0047] Performing machine learning typically involves the following three phases: a training phase, which is used to train a machine learning model with a training dataset by pairing inputs with expected outputs; an evaluation / testing phase, which is used to estimate how well the model has been trained by estimating model performance characteristics (e.g., classification error of a classifier, etc.) using an evaluation dataset and / or a test dataset; and an application phase, which is used to apply real data to the trained machine learning model to obtain results. Figure 2 In the verification system 120 shown in , the computing device 210 is configured to implement the training phase and possible evaluation / testing phase of machine learning, while the computing device 230 is configured to implement the application phase of machine learning. It should be understood that although shown separately, the computing devices 210 and 230 can be implemented as a single physical device to perform their functionality described herein.

[0048] Specifically, the computing device 210 is configured to train one or more machine learning models, for example, machine learning models 212-1, 212-2, ..., 212-N, for verifying medical data. N is any integer equal to or greater than 1. Machine learning models 212-1, 212-2, ..., 212-N are collectively referred to as or individually referred to as machine learning models 212 hereinafter. (One or more) machine learning models 212 are trained to automatically process input medical data and output verification results for the input medical data. The trained (one or more) machine learning models 212 each represent an association between medical data and verification results. This association makes it possible to guide the machine learning model 212 to predict a suitable verification result for the input medical data.

[0049] The medical data to be verified can be generated in the medical test of the test sample and can be organized into the form of a medical test report, for example. The medical data includes one or more test results of the test item, which may include measurement values ​​related to the test item and / or a numerical (yes or no) result indicating the presence of a certain analyte in the test sample. The verification result indicates information related to the action to be performed on the medical data, which can be regarded as a suggestion for the system or user to automatically or manually decide how the medical data can be processed in the next step of the entire medical diagnostic test workflow. The machine learning model 212 selects an action from a plurality of predetermined actions suitable for subsequent processing of the medical data.

[0050] The verification process is to find potential errors in the medical data before releasing the medical data to the entity (such as a clinical department or patient) requesting the medical test. If the medical data is verified to be correct and without errors, the next step for the medical data is to release the medical data to the entity (or require a quick manual review and then release it to the entity). In this case, one possible action to be performed on the medical data is to release the medical data to the entity requesting the medical test related to the medical data directly or after a quick manual review. For ease of discussion, this action is sometimes referred to as the first action in this article.

[0051] In other cases, medical data is verified to have errors, which may be due to the test sample, the medical diagnostic test procedure performed, the reagents used in the medical test, the physical condition of the biological object that does not match the test sample, insufficient information for making a decision, etc. In this case, a corresponding action needs to be performed to correct the error. The action indicated in the verification result for the medical data is a recommendation to further verify the medical data (referred to as a "second action"). The second action is a general recommendation, which means that the current medical data should not be released and a manual review is required to decide how the medical data can be further verified.

[0052] In some examples, the machine learning model 212 can be trained to determine one or more specific actions, such actions are actions that can be performed for further verification, including the action of re-running a medical test related to the medical data (referred to as the "third action"); the action of checking the historical patient medical records (referred to as the "fourth action"); the action of checking the reagent reaction in the medical test (referred to as the "fifth action"), such as checking the reagent reaction curve; the action of checking the test sample collected for use in the medical test (referred to as the "sixth action"); the action of checking the medical data in conjunction with the clinical diagnosis (referred to as the "seventh action"); and the action of checking the patient's medication status (referred to as the "eighth action"); etc. It should be understood that the next step actions listed above are only some specific examples, and more, fewer, or different actions can also be specified according to the needs of the actual use case.

[0053] By indicating one of those next actions in the validation result, the system or laboratory technician can more easily and efficiently determine the next step to be performed in the medical diagnostic test workflow. For example, if the validation result indicates the third action of re-running the medical test, the validation system 120 can automatically request Figure 1 The medical test system 110 in the embodiment of the present invention reruns the medical test to update the medical data. Since the machine learning model 212 has learned from historical experience and knowledge to provide one of these actions, the requirements and pressure on laboratory technicians can be significantly reduced. Laboratory technicians with limited experience and knowledge may be able to decide the next step for further verification without spending a lot of time manually reviewing medical data.

[0054] In some embodiments, the validation results from the machine learning model 212 may include a clear indication of the action to be performed on the medical data and / or the confidence level of the machine learning model 212 when selecting an action for the current medical data. When determining the action to output, the machine learning model 212 measures the corresponding probabilities of the predetermined actions and selects the action with the highest probability. The probability of an action can indicate how confident the machine learning model 212 is in selecting the action as the next action. The confidence level can be considered as the risk level of the action. The confidence level can also be used as a reference for laboratory technicians or systems to confirm whether the recommended operation is appropriate for the medical data 112.

[0055] The input (i.e., medical data) and output (i.e., verification results) of the machine learning model 212 have been described. The training phase of the machine learning implemented in the computing device 210 is to enable (one or more) machine learning models 212 to learn the association between input and output. Based on the training data 202 from the database 201, (one or more) machine learning models 212 are trained in a supervised manner. The training data 202 includes historical medical data and annotation information associated therewith. The historical medical data may include many medical test reports generated in different medical tests for one or more patients. The annotation information indicates the corresponding actions performed on the medical test reports. In this way, the annotation information can be used as a true indication of the actions to be performed on the historical medical data. Typically, the annotated actions of the historical medical data are those actions that are considered to be correct actions for those medical data and / or those actions that are manually marked by laboratory experts.

[0056] The machine learning model(s) 212 are trained to provide a verification result indicating the same or similar action for the input medical data as indicated in the annotation information associated with the input medical data. In this way, the trained machine learning model(s) 212 can represent the association between the medical data and the verification result. The training of the machine learning model(s) 212 by the computing device 210 will be discussed in detail below.

[0057] The generated (one or more) machine learning models 212 may be provided by the computing device 210 to the computing device 230 via a communication network, or may be stored by the computing device 210 in a storage device accessible to the computing device 230. The computing device 230 may use one or more of the generated machine learning models 212 to verify the medical data. The computing device 230 is configured to obtain the medical data 112 to be verified, and apply the medical data 112 to one or more machine learning models 212 to obtain the verification result 122 for the medical data 112.

[0058] The medical data 112 may sometimes be referred to as target medical data, which may be retrieved from a storage device or received from the medical testing system 110. The verification result 122 may sometimes be referred to as a target verification result, which indicates information related to a target action to be performed on the target medical data, the target action being selected from a plurality of predetermined actions. The verification result 122 may include an indication of the target action and / or a confidence level of the (one or more) machine learning model 212 when the target action is selected. The computing device 230 may present the verification result 122 to a user (such as a laboratory technician), or store the verification result 122 for further use. The application of the computing device 230 to the (one or more) machine learning model 212 will be discussed in detail below.

[0059] Training of machine learning models

[0060] Figure 3 The block diagram of the architecture for implementing the training of one or more machine learning models 212 in the computing device 210 is illustrated. As shown in the figure, the computing device 210 includes a data preparation module 310 for preprocessing the training data 202 and a model training module 320 for implementing the model training process based on the training data prepared by the data preparation module 310. Figure 3 , only one trained machine learning model 212 is depicted for illustrative purposes. In some cases, more than one machine learning model 212 may be trained.

[0061] In real-world scenarios, a large portion of medical test reports are verified by laboratory experts as valid and can be published, which means that their associated annotation information can indicate the first action mentioned above. Such medical data and associated annotation information can be considered as positive samples for model training. Due to the large number of available positive samples, it is desirable to select those samples with higher reliability, which can ensure the accuracy of subsequent machine learning.

[0062] Specifically, the data preparation module 310 is configured to obtain a set of available historical medical data (also referred to as a "first set of historical medical data") that is labeled as being associated with annotation information indicating a first action from the training data 202. The annotation information may be manually annotated to the first set of historical medical data by a laboratory expert and / or automatically annotated to the first set of historical medical data by some known available verification system (such as a rule-based verification engine). The data preparation module 310 may further filter historical medical data having a higher reliability in the annotation information than other historical medical data in the group from the first set of historical medical data, and provide the selected historical medical data and the associated annotation information for the model training module to use for training (one or more) machine learning models 212.

[0063] The reliability of the associated annotation information can depend on various factors, such as the data source from which the historical medical data and / or annotation information is collected, the condition of the patient associated with the historical medical data and / or annotation information, and the like. For example, if historical medical data and annotation information are collected from a relatively large number of medical examination centers where patients to be examined are in a healthy state, such annotation information has a higher reliability than those annotation information from different clinics. In addition, the reliability of the annotation information of the historical medical data collected from ordinary clinics may be higher than the reliability of the annotation information obtained from some intensive care units. As another example, the patient's age, diagnosed disease, specialty and / or other aspects can be set as certain criteria to filter out annotation information with higher reliability. Compared with the historical medical data and annotation information associated with the elderly, historical medical data and annotation information associated with young people can be selected.

[0064] In addition to positive samples, negative samples may also be needed, that is, historical medical data with annotation information indicating actions other than the first action, for training (one or more) machine learning models 212. In some cases, due to the large amount of medical data that can be directly published, the percentage of medical data that is annotated as medical data that requires further verification (e.g., any action other than the first action) is relatively low. Although manual annotation is a possible option for obtaining sufficient training data for negative samples, its main disadvantages are high cost and time consumption. In some embodiments, the data preparation module 310 can be configured to facilitate the annotation of some historical medical data in the training data 202, especially to determine the annotation information to indicate a second general action for further verification of the medical data.

[0065] The data preparation module 310 can utilize unsupervised methods to facilitate the selection and labeling of negative samples for training. Specifically, the data preparation module 310 is configured to select abnormal historical medical data from an available historical medical data set (referred to as the "second set of historical medical data"). Some data in the second set of historical medical data may have been marked with associated labeling information, for example, labeling information indicating a first action. In some examples, the second set of historical medical data can be obtained from part or all of the training data 202 stored in the database 201. Abnormal historical medical data may be those historical medical data having abnormal values ​​for test results of some test items. In general, medical data with abnormal values ​​has a higher probability of requiring further verification and may therefore be associated with the second action.

[0066] The data preparation module 310 may be configured to cluster the second set of historical medical data to find abnormal historical medical data. Figure 4 An exemplary clustering result for a set of historical medical data is illustrated. Note that in this example, for the purpose of illustration only, each piece of historical data (e.g., each historical medical test report) is shown as including two-dimensional data. After clustering the historical medical data, three clusters 410, 420, and 430 are formed. Multiple pieces of abnormal medical data 401-1 to 410-9 are far from the centroids of the three clusters and can therefore be regarded as abnormal historical medical data. It should be understood that the above diagram is provided for the purpose of illustration only. Figure 4 The examples shown in FIG. 1 are not intended to imply any limitation on the scope of the present disclosure.

[0067] Some or all of the abnormal historical medical data may be marked with annotation information indicating a second action that requires further verification. In some embodiments, the data preparation module 310 is configured to present the abnormal historical medical data to the user for cross-marking. The abnormal historical medical data may be presented via a user interface of the computing device 230 or other terminal device. Upon receiving a user input indicating one of the predetermined actions, the data preparation module 310 is configured to mark the abnormal historical medical data as associated with the annotation information of the indicated action.

[0068] The user can use his / her experience and knowledge to explicitly specify multiple predetermined actions for abnormal historical medical data. In this case, the user is allowed to mark the associated annotation information to indicate all other actions except the first action. In some other examples, a simple user confirmation is required to confirm whether the abnormal historical medical data can be annotated with the second action. The data preparation module 310 can then determine the abnormal historical medical data and the associated annotation information as negative samples for use in model training. By filtering only the abnormal historical medical data for manual annotation, less manual work and lower time consumption and cost are required in the data preparation stage.

[0069] In some embodiments, the machine learning model 212 may be trained to verify medical data including test results for a relatively large number of test items. However, historical medical data in some medical test reports may include past test results for some but not all test items to be considered, which results in high sparsity because few medical test reports cover all test items involved in the machine learning model 212. In order to handle data sparsity, the data preparation module 310 may be configured to further process the training data to be used, for example, the training data obtained after the determination of the positive samples and negative samples described above.

[0070] The data preparation module 310 can be configured to process historical medical data by filling in the test results of other test items that do not have actual values. The filled test results for each test item can be determined based on the actual test results of the test item included in other historical medical data. For example, the filled test results for the test items can be calculated as the median or average of the reference range for each test item, wherein the reference range is determined based on the actual value of the actual test results for the test item. The reference range can also be determined as a nominal reference range for the test item. As an alternative, the data preparation module 310 can convert high-dimensional but sparse historical medical data into low-dimensional but dense historical medical data by using dimensionality reduction techniques such as singular value decomposition (SVD) and principal component analysis (PCA).

[0071] Although the historical medical data and the annotation information in the negative samples can be obtained in the manner described above, the amount of available negative samples may still be less than the amount of positive samples (where the annotation information indicates the first action), resulting in imbalanced training data. In order to control the imbalance, the data preparation module 310 can be configured to limit the ratio of negative samples to positive samples in the training of each machine learning model 212 to a reasonable level.

[0072] Specifically, the data preparation module 310 can be configured to select the first historical medical data and the second historical medical data from the entire set of available historical medical data (referred to as the "third set of historical medical data") based on a predetermined ratio of the amount of the first historical medical data to the amount of the second historical medical data. Here, "first historical medical data" refers to historical medical data associated with annotation information indicating a first action, and "second historical medical data" refers to historical medical data associated with annotation information indicating an action different from the first action in a predetermined action. The ratio of the selected first medical data to the selected second medical data may be equal to or lower than a predetermined ratio. The predetermined ratio may be set to any reasonable value, such as 10:1, 5:1, 20:1, etc., depending on the actual requirements in the model training. The data preparation module 310 may randomly sample the first and second historical medical data from all available training data.

[0073] Data preparation in the training phase has been discussed above. The training data 202 determined after processing by the data preparation module 310 can be provided to the model training module 320. The model training module 320 can be configured to train one or more machine learning models 212. In one embodiment, the training data 202 from the data preparation module 310 can be divided into different training data sets to train different machine learning models 212. In another embodiment, the same training data 202 can be used to train two or more different machine learning models 212. Each machine learning model 212 represents a different association between medical data and possible verification results. Different machine learning models can be constructed using different types of machine learning and / or different model configurations for the same type of machine learning. For the same type of machine learning, different configurations can be achieved by selecting different hyperparameters when designing the model. The use of multiple machine learning models can benefit from the use of different types of models, and exploring different possible associations between medical data and its verification results based on different model configurations.

[0074] As mentioned above, it is desirable that the machine learning model 212 is trained to identify the target action from a plurality of predetermined actions to be indicated by the verification result 122. In some embodiments, the machine learning model 212 may be designed as a classification model for classifying / assigning medical data to one of the categories corresponding to a plurality of predetermined actions. The machine learning model 212 that implements classification is referred to as a classifier.

[0075] Some examples of classification models are provided here, such as decision trees, Bayesian models, random forest models, support vector machines, K-nearest neighbor (KNN) models, neural networks, etc. Depending on the total number of possible categories (i.e., the number of actions to be performed on the medical data), the machine learning model 212 can be based on binary classification or multi-class classification. In some embodiments, the machine learning model 212 can be constructed by integrating multiple learning algorithms (e.g., neural networks or decision trees) to improve the predictive performance of a single classifier.

[0076] In order to perform supervised machine learning, known validation results are required for historical medical data. The machine learning model 212 needs to learn from a large amount of training data to predict or classify how new medical data should be classified. However, labeling a large amount of data in order to train the model is very time-consuming and involves a lot of manual work. Therefore, in some embodiments, (one or more) machine learning models 212 can be specially designed for a small training data set so that the model can collect as much information as possible to distinguish multiple actions.

[0077] When training (one or more) machine learning models 212, the model training module 320 can adopt various training methods that are known or to be developed in the future. The scope of the embodiments of the present disclosure is not limited in this regard. During the model training process, the model training module 320 can iteratively update the parameters of (one or more) machine learning models 212 until (one or more) models 212 can represent the association between medical data and verification results, for example, the input historical medical data can be correctly or almost correctly mapped to the verification results indicating the real action indicated in the annotation information.

[0078] In some embodiments, the model training module 320 or other modules (not shown) in the computing device 210 may be configured to perform cross-validation on the (one or more) machine learning models 212 trained by the machine training module 320 to ensure that the (one or more) models 212 can obtain satisfactory performance on the training data without overfitting. Various methods for model cross-validation may be applied, and the scope of the embodiments of the present disclosure is not limited in this regard.

[0079] Application of machine learning models

[0080] Figure 5 A block diagram of the architecture of an application implementing one or more trained machine learning models 212 in a computing device 230 is illustrated. Figure 5 , a trained machine learning model 212 is depicted for illustration purposes only. As shown, the computing device 230 includes a model application module 510 for obtaining (one or more) machine learning models 212 trained by the computing device 210, and then applying the target medical data 112 to the obtained (one or more) machine learning models 212. The target medical data 112 is provided as an input to each machine learning model 212. The (one or more) machine learning models 212 then process the target medical data 112 and associate it with an action in the predetermined action. Each of the (one or more) machine learning models 212 then generates a verification result that includes an indication of an action and / or a confidence level of the machine learning model 212 when the target action is selected.

[0081] The computing device 230 also includes a result combination module 520 for providing a target verification result 122 for the target medical data 112. If one machine learning model 212 is utilized, the verification result output by the model can be directly provided as the target verification result 122. If two or more machine learning models 212 are utilized, the result combination module 520 determines the target verification result 122 based on the corresponding verification results from those models 212.

[0082] In some embodiments, the result combination module 520 may calculate a weighted average of the verification results, for example, by averaging the values ​​indicating the confidence levels included in the corresponding verification results. If the verification result indicates that different actions are to be performed on the target medical data 112, weighted averages for different actions may be determined. The result combination module 520 may determine the target verification result 122 to indicate the same action predicted by all used machine learning models 212 and / or indicate a weighted average as a confidence level. In the case where two or more possible actions are predicted, the result combination module 520 may compare the weighted averages with each other to select a higher weighted average and / or corresponding action indicated in the target verification result 122. Alternatively, instead of calculating a weighted average, the result combination module 520 may directly select a verification result determined by the machine learning model 212 to have a higher confidence level than those indicated in other verification results. The result combination module 520 then determines the selected verification result as the target verification result 122.

[0083] The target verification result 122 and the target medical data 112 may be stored in association with each other. If the target verification result 122 indicates information related to a target action other than the first action, which means that the target medical data 112 requires some type of further verification, the target verification result 122 and the target medical data 112 may be presented to the user together to guide the next step.

[0084] Figure 6 An exemplary medical test report 600 is shown, which presents a medical data portion 610-1 of the test results of the corresponding test items, a medical data portion 610-2 of the auxiliary information, and a verification result portion 620 indicating the next action and the confidence level about this next action. As shown, (one or more) machine learning models 212 recommend re-running the medical test, and the confidence level (i.e., risk level) for such a recommendation is high (score of 100). Typically, the confidence level calculated by the machine learning model is in the range of 0 to 1. To better illustrate the risk, in Figure 6 In the example of , the confidence level is mapped to a range of 0 to 100. It should be understood that the confidence level may be indicated in any other range of values ​​or by a non-numeric indicator.

[0085] In some embodiments, if the computing device 210 has trained multiple machine learning models 212 available for use, the computing device 230 (such as the model application module 520 included therein) can be configured to select one or more machine learning models 212 for use based on the corresponding performance metrics of those models. For example, such performance metrics can be determined by the computing device 210 based on the training data during the validation / testing phase. Of course, the computing device 230 can calculate the performance metrics itself. In some embodiments, the performance metrics of the machine learning models 212 can be calculated at the computing device 210 and then provided to the computing device 230.

[0086] The performance of each trained machine learning model 212 can be measured based on different characteristics associated with the model. In some embodiments, a receiver operating characteristic (ROC) curve and / or area under the curve (AUC) can be calculated in the validation phase to indicate the performance of the model 212. In statistics, the ROC curve is a graph of the true positive rate (TPR) and the false positive rate (FPR) for different decision thresholds. The true positive rate is also called sensitivity or recall. The false positive rate is 1 specificity. The ROC curve can evaluate the discriminative ability of the classifier and identify the optimal decision threshold based on sensitivity and specificity. The ROC curve of a model without discriminative ability will be a 45-degree diagonal line. The area under the curve (AUC) is the probability that the classifier score of a randomly selected positive observation is higher than the probability of a randomly selected negative observation. The AUC can summarize the performance of the classifier in a single value.

[0087] Alternatively or additionally, a confusion matrix of a machine learning model can be determined as a performance metric. A confusion matrix is ​​a table that summarizes the prediction accuracy for a classification problem. Each row of the table represents the number of instances in each predicted class, while each column represents the number of instances in each true class. The confusion matrix can also indicate the prediction accuracy for each class.

[0088] In addition to providing the target verification result 122 for the target medical data 112 or as an alternative, the computing device 230 may also provide the user with other medical data having similar attributes to the target medical data 112, which may help and guide the user (especially a junior technician or doctor) to review and understand the target medical data and / or verification result. Figure 5 , the computing device 230 includes a reference determination module 530 configured to provide such reference medical data.

[0089] Specifically, the reference determination module 530 is configured to determine the similarity between the target medical data 112 and the candidate medical data. The candidate medical data may have been annotated or assigned with information related to one of the predetermined actions that has been performed or is expected to be performed. The historical medical data used to generate the machine learning model 212, such as Figure 5 Such candidate medical data are selected from those included in the training data 202 illustrated in FIG. Alternatively or in addition, candidate medical data may also be selected from historical medical data that has been previously applied to the machine learning model 212.

[0090] In order to narrow down the medical data that can be selected as candidate medical data and speed up the determination of reference medical data, in some embodiments, the candidate medical data can be selected to have the same target action to be performed on it and / or have the same type of test items as the target medical data. Such candidate medical data may be considered as a more significant and meaningful reference for the target medical data 112.

[0091] In some embodiments, the reference determination module 530 can determine the similarity by calculating one or more test item similarities between one or more test results of the test items indicated in the target medical data and one or more test results of the test items indicated in the candidate medical data. Test item similarity can be measured by Euclidean distance, which is a mathematical formula based on calculating the distance between two data points (i.e., two test results). The Euclidean distances between multiple pairs of test results can be combined (e.g., weighted and summed) to represent the test item similarity. In other examples, any other algorithm can be used to measure the distance between two data points.

[0092] If the similarity exceeds a predetermined similarity threshold, the reference determination module 530 may select the candidate medical data as the reference medical data 532 of the target medical data 112. In some embodiments, the candidate medical data may be a medical test report in a medical test, and the similarity between the target medical data 112 and other candidate medical data (other medical test reports) is calculated. In this case, the reference determination module 530 may rank the candidate medical data based on the similarity and select the candidate medical data ranked high.

[0093] Similarity measurements can be performed in real time or can be pre-calculated. Pre-calculated similarity measurements can be triggered by certain criteria, such as when computing resources are not used every night. For pre-calculated similarity measurements, the reference determination module 530 can calculate the similarities between all historical medical data and / or previously verified medical data, and then divide the medical data into different clusters. In the case where new medical data is received and verified, the reference determination module 530 can only determine the similarity between the new medical data and the cluster (such as the medical data in the centroid of the cluster). New medical data can be classified into one of these clusters based on similarity (for example, a cluster with a relatively high or highest similarity to the new medical data), and previous medical data can be used as reference medical data for the new medical data. In this way, reference determination can be achieved in a timely manner.

[0094] The Evolution of Machine Learning Models

[0095] As the trained(s) machine learning model(s) 212 are applied over a period of time and / or as requirements for medical data validation change, it is desirable to train new models to further improve the performance of medical data validation. Figure 7 Another example of a computing device 210 is illustrated that also includes a model evolution module 730 to determine one or more additional machine learning models 212 for validating medical data.

[0096] The evolution of the machine learning model at the model evolution module 730 can be initiated based on one or more predetermined triggers. The trigger for the evolution of the machine learning model can be based on the available training data for model training. In the initial stage, the training data may be limited, and therefore the limited training data can be used to train one or more machine learning models 212 for use. As (one or more) models 212 are applied, new medical data can be collected and its annotation information can be determined automatically by the previous model and / or manually based on user feedback. If the amount of new training data has reached a certain threshold, the evolution of the machine learning model can be initiated.

[0097] Alternatively, or in addition to triggers based on training data, in some embodiments, according to actual requirements, one or more test items will be performed in the medical test, and thus their results will be added to the generated medical data. In this case, a new machine learning model 212 will be trained to learn how to classify medical data using such new test items. In some cases, laboratory experts can identify one or more new actions as possible actions to be performed on medical data that is verified to be abnormal or suspicious, and / or one or more existing actions can be identified as actions that are not required in automatic verification. In this way, one or more new machine learning models 212 can be trained to output verification results that indicate information related to the target action from the updated action set. In this way, machine learning models can be evolved to adapt to technological updates and experience accumulation in medical diagnostic tests. Some other possible triggers for the evolution of machine learning models may include explicit instructions from users (such as laboratory technicians / experts), time-based triggers, and the like.

[0098] In some embodiments, if the model evolution module 730 is determined to trigger model evolution, the module may request the model training module 320 to determine another machine learning model 212 by updating the machine learning model 212 that has already been trained. The updating of the machine learning model 212 may be performed by improving the model parameters using new training data. The model evolution module 730 may be further configured to determine another machine learning model 212 by generating a new machine learning model based on previously obtained training data and / or new training data. The new machine learning model may be designed to have a different model configuration than the already trained (one or more) machine learning models 212. Different model configurations may be set to have different hyperparameters and / or be associated with different types of machine learning than the trained (one or more) machine learning models 212.

[0099] In the event that one or more new actions are added and / or one or more existing actions are to be cancelled, a new model with a new model configuration is required, which can be designed to be trained to represent the association between the medical data and the additional verification results, which indicate the updated actions to be performed on the medical data. For example, due to limited training data in the initial stage, one or more machine learning models 212 can initially be designed to only implement binary classification between a first action and a second action that is generally recommended for further verification to ensure model accuracy. As the available training data increases, one or more additional machine learning models 212 can be trained to perform multi-class classification.

[0100] In some embodiments, if the number of available machine learning models 212 increases with evolution, the (one or more) machine learning models 212 that can be provided for use can be determined based on their performance metrics. That is, instead of directly providing all updated / new machine learning models 212 to replace the (one or more) previous machine learning models 212 currently in use, only models 212 with high performance metrics are suitable for use in order to improve validation performance.

[0101] In some embodiments, if it is determined that the new / updated machine learning model 212 has a higher performance metric than the machine learning model 212 currently in use at the computing device 230, the new / updated machine learning model may be provided to replace the previous machine learning model 212. The performance metric of the machine learning model may be determined in a similar manner as discussed above. The decision whether to replace the model in use may be made by the computing device 210 or the computing device 230.

[0102] Compatibility of machine learning models

[0103] The machine learning model 212 can be applied independently to verify medical data. In some embodiments, the machine learning model proposed in the present disclosure can also be easily compatible with legacy components / functions for medical data verification. In some legacy verification processes, as mentioned above, a rule-based engine is applied to verify medical data. In some cases, a LIS may also be included to archive, record, present and / or further process medical data and verification results. The machine learning model proposed herein may also be suitable for legacy medical data verification environments.

[0104] Figure 8 A block diagram of a system is shown in which one or more machine learning models 212 are compatible with a rule-based engine 810 and a LIS 820. Figure 8 In the embodiment, only one machine learning model 212 is depicted for illustration purposes, and more than one machine learning model 212 may also be compatible with the rule-based engine 810 and the LIS 820. It should be understood that in some other cases, the rule-based engine 810 or the LIS 820 may be omitted. Figure 8 In the example of , rule-based engine 810 and / or LIS 820 may be implemented in a device separate from computing device 230 , or may be integrated within computing device 230 .

[0105] Conventionally, the rule-based engine 810 and the LIS 820 can be coupled together to implement medical data verification. The automatic verification based on one or more machine learning models 212 proposed herein is implemented as an independent system that is connected between the rule-based engine 810 and the LIS 820, for example, by reusing the current communication interface of the rule-based engine 810 and the LIS 820. Through the adapter module, the rule-based engine 810, the machine learning model 212, and the LIS 820 can be connected to process medical data and provide verification results.

[0106] Specifically, Figure 8 As shown in , the medical data 802 to be verified can first be input into the rule-based engine 810 for verification based on one or more predetermined rules. If the medical data 802 is verified by the rule-based engine 810 as being able to be released to the entity requesting the medical test, such verification results and the medical data can be provided to the LIS 820. If the rule-based engine 810 determines that some of the medical data 802 needs to be further verified (i.e., the medical data 802 cannot be released), the medical data 802 is provided as the target medical data 112 to the machine learning model 212 for further processing.

[0107] Through this process, the rule-based engine 810 is used to filter reliable medical data to be released, and thus the machine learning model 212 processes less medical data, which in turn can reduce overall resource consumption - because the execution of the model usually requires more resources. In view of the fact that fixed rules are usually manually set to have a higher conservatism, the filtering of the rule-based engine will not result in performance sacrifice, and thus the verification results indicating the action of releasing medical data are considered to have a higher degree of reliability. In some embodiments, the verification results of the rule-based engine 810 can also be presented in the medical test report together with the verification results from the machine learning model 212. For example, returning to Figure 6 , the medical test report 600 is shown further presenting a verification result portion 630. The verification result portion 630 indicates that the rule-based engine 810 determined the medical data to be "failed", thereby requiring further verification.

[0108] In some embodiments, the machine learning model 212 may be disabled by the computing device 230 based on actual needs, and in this case only the medical data validation is performed by the rule-based engine 810. In some embodiments, the rule-based engine 810 may be disabled, and thus all medical data 802 may be validated by the machine learning model 212. The enabling and disabling of the machine learning model 212 and / or the rule-based engine 810 may be determined based on feedback from a user, such as a lab technician / expert.

[0109] In some embodiments, communication with the LIS 820 may follow some predefined specifications. Typically, the LIS 820 may support receiving corresponding test results for test items in the medical data, each test result being considered as a data item in a medical test report containing the medical data. The target verification result determined by the machine learning model may be provided to the LIS 820 in association with the target medical data. In order to meet the specifications supported by the LIS 820, communication with the LIS 820 may be easily extended so that the target verification result may be considered as a test result of a virtual customized test item (i.e., a new data item) contained in the medical test report. In this way, there is no need to modify the specifications for communication with the LIS 820 to support special verification results from the machine learning model. The LIS 820 may parse the virtual test item to obtain the target verification result from the machine learning model 212.

[0110] The communication between the machine learning model 212 / rule-based engine 810 and the LIS 820 can be scheduled based on any message scheduling method, such as a message bus scheduling that can implement asynchronous decoupled processing. It is convenient to expand the processing performance of the system through message bus scheduling and expand the processing capacity by adding multiple message receivers at the LIS 820. In some embodiments where the rule-based engine 810, one or more machine learning models 212, and the LIS 820 are implemented as a distributed system, each machine learning model 212 can be packaged as a machine learning model service through microservices. In this way, service registration, service discovery, and service version management can be performed through the model factory.

[0111] Example Process

[0112] Fig. 9 A flowchart of an example process 900 for verifying medical data according to some embodiments of the present disclosure is illustrated. The method 900 may be performed by Figure 2 For discussion purposes, reference will be made to the computing device 230 in FIG. Figure 2 To describe method 900.

[0113] At box 910, the computing device 230 obtains target medical data generated in a medical test. At box 920, the computing device 230 obtains a machine learning model for validating the medical data. The machine learning model represents an association between the medical data and a validation result, and the validation result indicates information related to a predetermined action to be performed on the medical data. At box 930, the computing device 230 determines a target validation result for the target medical data by applying the target medical data to the machine learning model. The target validation result indicates information related to a target action to be performed on the target medical data, the target action being selected from the predetermined action.

[0114] In some embodiments, determining a target verification result includes: obtaining another machine learning model for verifying medical data, the other machine learning model representing an association between medical data and verification results that is different from the association represented by the above machine learning model; applying the target medical data to the machine learning model and the other machine learning model, respectively, to obtain corresponding verification results; and determining the target verification result based on the corresponding verification results.

[0115] In some embodiments, process 900 also includes determining a similarity between the target medical data and candidate medical data, the candidate medical data being selected from: historical medical data used to generate a machine learning model, and / or historical medical data that has been applied to the machine learning model; in response to the similarity exceeding a predetermined similarity threshold, selecting the candidate medical data as reference medical data for the target medical data; and providing the reference medical data in association with the target medical data for presentation to a viewer of the target medical data.

[0116] In some embodiments, determining the similarity includes selecting the candidate medical data based on at least one of: determining that a target action is to be performed on the candidate medical data, and determining that the candidate medical data has one or more test items in common with the target medical data.

[0117] In some embodiments, the predetermined action includes at least one of: a first action of releasing the medical data to an entity requesting a medical test related to the medical data, a second action of further verifying the medical data, a third action of rerunning the medical test related to the medical data, a fourth action of checking historical patient medical records, a fifth action of checking reagent reactions in the medical test, a sixth action of checking test samples collected for use in the medical test, a seventh action of checking the medical data in conjunction with a clinical diagnosis, and an eighth action of checking the patient's medication status.

[0118] In some embodiments, the machine learning model includes a classification model for classifying medical data into categories corresponding to predetermined actions.

[0119] In some embodiments, the information related to the target action includes at least one of: an indication of the target action, and a confidence level of the machine learning model in selecting the target action for the target medical data.

[0120] In some embodiments, the target medical data includes medical data generated during an in vitro diagnostic test.

[0121] In some embodiments, a machine learning model is selected from a plurality of available machine learning models based on respective performance metrics of the plurality of available machine learning models.

[0122] In some embodiments, obtaining the target medical data includes obtaining the target medical data determined by a rule-based engine to be further verified, the rule-based engine being configured to verify the target medical data based on at least one predetermined rule.

[0123] In some embodiments, process 900 also includes providing the target verification result to a laboratory information system (LIS) in association with target medical data, the target medical data including at least one data item presented in the medical test report, and the target verification result is presented as another data item in the medical test report.

[0124] In some embodiments, the machine learning model is provided by Fig.10 The process 1000 described provides or trains.

[0125] Fig.10 A flowchart of an example process 1000 for providing a machine learning model for validating medical data according to some embodiments of the present disclosure is illustrated. The method 1000 may be performed by Figure 2 For discussion purposes, reference will be made to the computing device 210 in FIG. Figure 2 To describe method 1000.

[0126] At block 1010, the computing device 210 obtains training data, the training data including historical medical data and associated annotation information. The annotation information indicates a predetermined action to be performed on the historical medical data. At block 1020, the computing device 210 generates a first machine learning model for verifying the medical data based on the training data, so that the first machine learning model represents an association between the medical data and a verification result indicating information related to the predetermined action to be performed on the medical data.

[0127] In some embodiments, the predetermined action includes a first action of publishing medical data to an entity requesting a medical test related to the medical data, and obtaining training data includes: obtaining a first set of available historical medical data that is labeled as associated with annotation information indicating the first action; selecting historical medical data from the first set of available historical medical data, the selected historical medical data having a higher reliability in the annotation information than other historical medical data in the first set; and determining the selected historical medical data and the associated annotation information as training data.

[0128] In some embodiments, obtaining training data includes: selecting abnormal historical medical data from a second set of available historical medical data; presenting the abnormal historical medical data to a user; in response to receiving user input from the user indicating one of the predetermined actions, marking the abnormal historical medical data as associated with annotation information indicating the indicated action; and determining the abnormal historical medical data and the associated annotation information as training data.

[0129] In some embodiments, the predetermined action includes a first action of publishing medical data to an entity requesting a medical test related to the medical data, and obtaining training data includes: based on a predetermined ratio of an amount of the first medical data to an amount of the second medical data, selecting first historical medical data and second historical medical data from a third set of available historical medical data, the first historical medical data being associated with annotation information indicating the first action, and the second historical medical data being associated with annotation information indicating another action in the predetermined action that is different from the first action.

[0130] In some embodiments, process 1000 also includes: in response to a predetermined trigger for model evolution, determining a second machine learning model for verifying medical data by: updating the first machine learning model, or generating a new machine learning model based on the training data, the new machine learning model having a model configuration different from the first machine learning model.

[0131] In some embodiments, determining a second machine learning model includes: in response to determining that another action is to be performed on the medical data, adding additional historical medical data and associated additional annotation information to the training data, the additional annotation information indicating the other action; and generating a new machine learning model as a second machine learning model, so that the second machine learning model represents an association between the medical data and additional verification results, the additional verification results indicating a predetermined action to be performed on the medical data and another action.

[0132] In some embodiments, a first machine learning model is provided for use in validating medical data, the method further comprising: determining a first performance metric of the first machine learning model and a second performance metric of the second machine learning model; and in response to the second performance metric exceeding the first performance metric, providing a second machine learning model to replace the first machine learning model in use.

[0133] In some embodiments, the historical medical data includes test results for at least one test item among multiple predetermined test items, and generating a first machine learning model includes: processing the historical medical data by filling in test results for other test items among the multiple predetermined test items except the at least one test item, the filled-in test results are determined based on the test results of the other examination items included in the other historical medical data; and generating the first machine learning model based on the processed historical medical data.

[0134] In some embodiments, the predetermined action includes at least one of the following: a first action of releasing the medical data to an entity requesting the medical test, a second action of further verifying the medical data, a third action of rerunning the medical test related to the medical data, a fourth action of checking historical patient medical records, a fifth action of checking reagent reactions in the medical test, a sixth action of checking test samples collected for use in the medical test, a seventh action of checking the medical data in conjunction with a clinical diagnosis, and an eighth action of checking the patient's medication status.

[0135] Example Device

[0136] Fig.11 A block diagram of an example computing system / device 1100 suitable for implementing example embodiments of the present disclosure is illustrated. The system / device 1100 may be implemented as or in Figure 2 The computing device 210 and / or Figure 2 The system / device 1100 may be a general purpose computer, a physical computing device, or a portable electronic device, or may be implemented in a distributed cloud computing environment where tasks are performed by remote processing devices linked via a communications network. The system / device 1100 may be used to implement Fig. 9 Process 900 and / or Fig.10 of process 1000.

[0137] As depicted, the system / device 1100 includes a processor 1101 that can perform various processes according to a program stored in a read-only memory (ROM) 1102 or a program loaded from a storage unit 1108 to a random access memory (RAM) 1103. In the RAM 1103, data required when the processor 1101 performs various processes, etc., is also stored as needed. The processor 1101, the ROM 1102, and the RAM 1103 are connected to each other via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0138] Processor 1101 may be of any type suitable for the local technology network, and may include, as non-limiting examples, one or more of the following: a general purpose computer, a special purpose computer, a microprocessor, a digital signal processor (DSP), a graphics processing unit (GPU), a coprocessor, and a processor based on a multi-core processor architecture. System / device 1100 may have multiple processors, such as application specific integrated circuit chips, which are time-slaved to a clock synchronized with a main processor.

[0139] A number of components in the system / device 1100 are connected to an I / O interface 1105, including an input unit 1106, such as a keyboard, a mouse, etc., an output unit 1107, including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker, etc., a storage unit 1108, such as a magnetic disk and an optical disk, etc., and a communication unit 1109, such as a network card, a modem, a wireless transceiver, etc. The communication unit 1109 allows the system / device 1100 to exchange information / data with other devices via a communication network such as the Internet, various telecommunication networks, etc.

[0140] The above methods and processes, such as process 900 and / or process 1000, may also be performed by processor 1101. In some embodiments, process 900 and / or process 1000 may be implemented as a computer program product or computer software program tangibly included in a computer-readable medium (e.g., storage unit 1108). In some embodiments, the computer program may be partially or completely loaded and / or implemented to system / device 1100 via ROM 1102 and / or communication unit 1109. The computer program includes computer executable instructions executed by the associated processor 1101. When the computer program is loaded into RAM 1103 and executed by processor 1101, one or more actions in process 900 and / or process 1000 described above may be implemented. Alternatively, in other embodiments, processor 1101 may be configured to perform process 900 and / or process 1000 via any other suitable means (e.g., by means of firmware).

[0141] Example Embodiment List

[0142] The embodiments of the present disclosure may be embodied in any form described herein. For example, the exemplary embodiments listed below describe some structures, features and functionalities of some aspects of the present disclosure disclosed herein.

[0143] In a first aspect, an exemplary embodiment of the present disclosure provides a method for medical data verification. The method includes: obtaining target medical data generated in a medical test; and obtaining a machine learning model for verifying the medical data, the machine learning model representing an association between the medical data and a verification result, and the verification result indicating information related to a predetermined action to be performed on the medical data; and determining a target verification result for the target medical data by applying the target medical data to the machine learning model, the target verification result indicating information related to a target action to be performed on the target medical data, the target action being selected from the predetermined action.

[0144] In some embodiments, determining a target verification result includes: obtaining another machine learning model for verifying medical data, the other machine learning model representing an association between medical data and a verification result that is different from the association represented by the machine learning model, applying the target medical data to the machine learning model and the other machine learning model, respectively, to obtain a corresponding verification result; and determining the target verification result based on the corresponding verification result.

[0145] In some embodiments, the method also includes determining a similarity between the target medical data and candidate medical data, the candidate medical data being selected from: historical medical data used to generate a machine learning model, and / or historical medical data that has been applied to the machine learning model; in response to the similarity exceeding a predetermined similarity threshold, selecting the candidate medical data as reference medical data for the target medical data; and providing the reference medical data in association with the target medical data for presentation to a viewer of the target medical data.

[0146] In some embodiments, determining the similarity includes selecting the candidate medical data based on at least one of: determining that a target action is to be performed on the candidate medical data, and determining that the candidate medical data has one or more test items in common with the target medical data.

[0147] In some embodiments, the predetermined action includes at least one of the following: a first action of releasing the medical data to an entity requesting an action test related to the medical data, a second action of further verifying the medical data, a third action of rerunning the medical test related to the medical data, a fourth action of checking historical patient medical records, a fifth action of checking reagent reactions in the medical test, a sixth action of checking test samples collected for use in the medical test, a seventh action of checking the medical data in conjunction with a clinical diagnosis, and an eighth action of checking the patient's medication status.

[0148] In some embodiments, the machine learning model includes a classification model for classifying medical data into categories corresponding to predetermined actions.

[0149] In some embodiments, the information related to the target action includes at least one of: an indication of the target action, and a confidence level in selecting the target action by the machine learning model for the target medical data.

[0150] In some embodiments, the target medical data includes medical data generated during an in vitro diagnostic test.

[0151] In some embodiments, a machine learning model is selected from a plurality of available machine learning models based on respective performance metrics of the plurality of available machine learning models.

[0152] In some embodiments, obtaining the target medical data includes obtaining the target medical data determined by a rule-based engine to be further verified, the rule-based engine being configured to verify the target medical data based on at least one predetermined rule.

[0153] In some embodiments, the method further comprises providing the target verification result to a laboratory information system (LIS) in association with target medical data, the target medical data comprising at least one data item presented in a medical test report, and the target test result being presented as another data item in the medical test report.

[0154] In some embodiments, the machine learning model is provided by the method of the second aspect.

[0155] In a second aspect, an exemplary embodiment of the present disclosure provides a method for providing a machine learning model for verifying medical data. The method includes obtaining training data, the training data including historical medical data and associated annotation information, the annotation information indicating a predetermined action performed on the historical medical data; and generating a first machine learning model for verifying the medical data based on the training data, so that the first machine learning model represents an association between the medical data and a verification result, the verification result indicating information related to the predetermined action to be performed on the medical data.

[0156] In some embodiments, the predetermined action includes a first action of publishing medical data to an entity requesting a medical test related to the medical data, and obtaining training data includes: obtaining a first set of available historical medical data that is labeled as associated with annotation information indicating the first action; selecting historical medical data from the first set of available historical medical data, the selected historical medical data having a higher reliability in the annotation information than other historical medical data in the first set; and determining the selected historical medical data and the associated annotation information as training data.

[0157] In some embodiments, obtaining training data includes: selecting abnormal historical medical data from a second set of available historical medical data; presenting the abnormal historical medical data to a user; in response to receiving user input from the user indicating one of the predetermined actions, marking the abnormal historical medical data as associated with annotation information indicating the indicated action; and determining the abnormal historical medical data and the associated annotation information as training data.

[0158] In some embodiments, the predetermined action includes a first action of publishing medical data to an entity requesting a medical test related to the medical data, and obtaining training data includes: based on a predetermined ratio of an amount of the first medical data to an amount of the second medical data, selecting first historical medical data and second historical medical data from a third set of available historical medical data, the first historical medical data being associated with annotation information indicating the first action, and the second historical medical data being associated with annotation information indicating another action in the predetermined action that is different from the first action.

[0159] In some embodiments, the method also includes: in response to a predetermined trigger for model evolution, determining a second machine learning model for verifying the medical data by: updating the first machine learning model, or generating a new machine learning model based on the training data, the new machine learning model having a model configuration different from the first machine learning model.

[0160] In some embodiments, determining a second machine learning model includes: in response to determining that another action is to be performed on the medical data, adding additional historical medical data and associated additional annotation information to the training data, the additional annotation information indicating the other action; and generating a new machine learning model as a second machine learning model, so that the second machine learning model represents an association between the medical data and additional verification results, the additional verification results indicating a predetermined action to be performed on the medical data and another action.

[0161] In some embodiments, a first machine learning model is provided for use in validating medical data, the method further comprising: determining a first performance metric of the first machine learning model and a second performance metric of the second machine learning model; and in response to the second performance metric exceeding the first performance metric, providing a second machine learning model to replace the first machine learning model in use.

[0162] In some embodiments, the historical medical data includes test results for at least one test item among multiple predetermined test items, and generating a first machine learning model includes: processing the historical medical data by filling in test results for other test items among the multiple predetermined test items except the at least one test item, the filled-in test results are determined based on the examination results of the other examination items included in the other historical medical data; and generating the first machine learning model based on the processed historical medical data.

[0163] In some embodiments, the predetermined action includes at least one of the following: a first action of releasing the medical data to an entity requesting the medical test, a second action of further verifying the medical data, a third action of rerunning the medical test related to the medical data, a fourth action of checking historical patient medical records, a fifth action of checking reagent reactions in the medical test, a sixth action of checking test samples collected for use in the medical test, a seventh action of checking the medical data in conjunction with a clinical diagnosis, and an eighth action of checking the patient's medication status.

[0164] In a third aspect, an exemplary embodiment of the present disclosure provides an electronic device. The electronic device includes at least one processor; and at least one memory including computer-readable instructions, which, when executed by at least one processor of the electronic device, causes the electronic device to perform the steps of the method in the first aspect above.

[0165] In a fourth aspect, an exemplary embodiment of the present disclosure provides an electronic device. The electronic device includes at least one processor; and at least one memory including computer-readable instructions, which, when executed by at least one processor of the electronic device, causes the electronic device to perform the steps of the method in the second aspect above.

[0166] In a fifth aspect, an example embodiment of the present disclosure provides a computer program product, the computer program product comprising instructions, which, when executed by a processor of a device, cause the device to perform the steps of any one of the methods of the first aspect described above.

[0167] In a sixth aspect, an example embodiment of the present disclosure provides a computer program product, the computer program product comprising instructions, which, when executed by the processor, cause the apparatus to perform the steps of any one of the methods in the second aspect above.

[0168] In a seventh aspect, an exemplary embodiment of the present disclosure provides a computer-readable medium, the computer-readable medium comprising program instructions, the program instructions being used to cause an apparatus to at least perform the method in the first aspect described above. In some embodiments, the computer-readable medium may be a non-transitory computer-readable medium.

[0169] In an eighth aspect, an exemplary embodiment of the present disclosure provides a computer-readable medium, the computer-readable medium comprising program instructions, the program instructions being used to cause an apparatus to at least perform the method in the second aspect described above. In some embodiments, the computer-readable medium may be a non-transitory computer-readable medium.

[0170] In general, various example embodiments of the present disclosure may be implemented in hardware or dedicated circuits, software, logic, or any combination thereof. Some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software, which may be executed by a controller, microprocessor, or other computing device. Although various aspects of the example embodiments of the present disclosure are illustrated and described as block diagrams, flow charts, or using some other graphical representations, it should be understood that, as non-limiting examples, the boxes, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general hardware or controllers, or other computing devices, or some combination of the foregoing.

[0171] The present disclosure also provides at least one computer program product tangibly stored on a non-transitory computer-readable storage medium. The computer program product includes computer executable instructions, such as those included in program modules, those instructions executed in devices on target actual or virtual processors, to perform the method / process as described above. Typically, program modules include routines, programs, libraries, objects, classes, components, data structures, etc. that perform specific tasks or implement specific abstract data types. In various embodiments, the functionality of program modules can be combined or divided between program modules. Computer executable instructions for program modules can be executed in local or distributed devices. In distributed devices, program modules can be located in local and remote storage media.

[0172] The computer readable medium may be a computer readable signal medium or a computer readable storage medium. The computer readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. More specific examples of computer readable storage media will include an electrical connection with one or more wires, a portable computer floppy disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0173] The computer program code for performing the method disclosed herein can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device so that the program code enables the function / operation specified in the flow chart and / or the block diagram to be implemented when the program code is executed by the processor or controller. The program code can be executed completely on a computer, partially on a computer, executed as an independent software package, partially on a computer and partially on a remote computer, or completely on a remote computer or server. The program code can be distributed on a specially programmed device, which is generally referred to as a "module" in this article. The software component part of the module can be written in any computer language and can be part of the overall code base, or can be developed in more discrete code parts, such as usually developed in an object-oriented computer language. In addition, the module can be distributed on multiple computer platforms, servers, terminals, mobile devices, etc. A given module can even be implemented so that the described function is performed by a separate processor and / or computing hardware platform.

[0174] Although operation is described in a specific order, this should not be construed as requiring to perform such operation in the specific order shown or in a continuous order, or to perform all illustrated operations to achieve desired results. In some cases, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the above discussion, these should not be construed as limitations on the scope of the present disclosure, but should be construed as descriptions of features that may be specific to a particular embodiment. Certain features described in the context of the separated embodiment may also be implemented in combination in a single embodiment. On the contrary, the various features described in the context of a single embodiment may also be implemented in a plurality of embodiments or in any suitable subcombination, respectively.

[0175] Although the disclosure has been described in language specific to structural features and / or methodological acts, it should be understood that the disclosure defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims

1. A method for medical data verification, comprising: obtaining target medical data generated in a medical test; obtaining a machine learning model for validating medical data, wherein the medical data is generated in a medical test by performing the medical test on a test sample for medical diagnosis, and wherein the validation is based on a validation process that includes discovering potential errors in the medical data before releasing the medical data to an entity requesting the medical test, thereby creating a validation result, the machine learning model represents an association between the medical data and the validation result, and the validation result indicates information related to a predetermined action to be performed on the medical data, and the predetermined action includes a first action of releasing the medical data to an entity requesting a medical test related to the medical data; and determining a target verification result for the target medical data by applying the target medical data to the machine learning model, the target verification result indicating information related to a target action to be performed on the target medical data, the target action being selected from the predetermined action, wherein in response to the medical data being verified as correct and free of errors, the medical data is released to the entity that requested the test, The machine learning model includes a classification model for classifying the medical data into categories corresponding to the predetermined actions.

2. The method according to claim 1, wherein determining the target verification result comprises: obtaining another machine learning model for verifying the medical data, the another machine learning model representing an association between the medical data and the verification result that is different from the association represented by the machine learning model; Applying the target medical data to the machine learning model and the another machine learning model respectively to obtain corresponding verification results; as well as The target verification result is determined based on the corresponding verification result.

3. The method according to any one of claims 1 and 2, further comprising: Determining a similarity between the target medical data and candidate medical data, the candidate medical data being selected from: historical medical data used to generate the machine learning model, and / or historical medical data that has been applied to the machine learning model; In response to the similarity exceeding a predetermined similarity threshold, selecting the candidate medical data as reference medical data for the target medical data; as well as Reference medical data is provided in association with the target medical data for presentation to a viewer of the target medical data.

4. The method of claim 3, wherein determining the similarity comprises: The candidate medical data is selected based on at least one of the following: determining to perform the target action on the candidate medical data, and It is determined that the candidate medical data has one or more test items that are the same as the target medical data.

5. The method according to claim 1, wherein the predetermined action further comprises at least one of the following: a second action of further verifying the medical data, a third action of re-running said medical test associated with said medical data, The fourth action of checking the historical patient medical records, a fifth act of checking a reagent reaction in said medical test, a sixth act of inspecting a test sample collected for use in said medical test, a seventh act of examining the medical data in conjunction with a clinical diagnosis, and The eighth action is to check the patient's medication status.

6. A method according to claim 1, wherein the information related to the target action includes at least one of the following: an indication of the target action, and a confidence level of the machine learning model in selecting the target action for the target medical data.

7. The method of claim 1, wherein the target medical data comprises medical data generated in an in vitro diagnostic test.

8. The method of claim 1, wherein the machine learning model is selected from a plurality of available machine learning models based on respective performance metrics of the plurality of available machine learning models.

9. The method according to claim 1, wherein obtaining the target medical data comprises: The target medical data determined by a rule-based engine to be further validated is obtained, the rule-based engine being configured to validate the target medical data based on at least one predetermined rule.

10. The method according to claim 1, further comprising: The target verification result is provided to a laboratory information system (LIS) in association with the target medical data, the target medical data comprising at least one data item presented in a medical test report, and the target verification result is presented as another data item in the medical test report.

11. A method of providing a machine learning model for validating medical data, wherein the medical data is generated in a medical test by performing the medical test on a test sample for medical diagnosis, and wherein the validation is based on a validation process that includes discovering potential errors in the medical data before releasing the medical data to an entity requesting the medical test, thereby creating a validation result, the method comprising: obtaining training data, the training data comprising historical medical data and associated annotation information, the annotation information indicating predetermined actions performed on the historical medical data; and generating a first machine learning model for validating medical data based on the training data, such that the first machine learning model represents an association between the medical data and a validation result indicating information related to the predetermined action to be performed on the medical data, wherein the predetermined action comprises a first action of releasing the medical data to an entity requesting a medical test related to the medical data, The first machine learning model includes a classification model for classifying the medical data into categories corresponding to the predetermined actions.

12. The method according to claim 11, wherein obtaining the training data comprises: obtaining a first set of available historical medical data marked as being associated with annotation information indicating the first action; Selecting historical medical data from the first set of available historical medical data, the selected historical medical data having a higher reliability in the annotated information than other historical medical data in the first set; as well as The selected historical medical data and associated annotation information are determined as the training data.

13. The method according to any one of claims 11 and 12, wherein obtaining the training data comprises: selecting abnormal historical medical data from a second set of available historical medical data; Presenting the abnormal historical medical data to a user; in response to receiving a user input from the user indicating one of the predetermined actions, marking the abnormal historical medical data as associated with annotation information indicating the indicated action; as well as The abnormal historical medical data and the associated annotation information are determined as the training data.

14. The method of claim 11, wherein the predetermined action comprises a first action of releasing the medical data to an entity requesting a medical test related to the medical data, and obtaining the training data comprises: Based on a predetermined ratio of the amount of first historical medical data to the amount of second historical medical data, the first medical data and the second medical data are selected from a third set of available historical medical data, the first historical medical data being associated with annotation information indicating the first action, and the second historical medical data being associated with annotation information indicating another action in the predetermined actions that is different from the first action.

15. The method according to claim 11, further comprising: In response to a predetermined trigger for model evolution, determining a second machine learning model for validating the medical data by: updating the first machine learning model, or A new machine learning model is generated based on the training data, wherein the new machine learning model has a model configuration different from that of the first machine learning model.

16. The method of claim 15, wherein determining the second machine learning model comprises: In response to determining that another action is to be performed on the medical data, adding further historical medical data and associated further annotation information to the training data, the further annotation information indicating the other action; and Generate the new machine learning model as the second machine learning model, so that the second machine learning model represents the association between the medical data and another verification result, and the other verification result indicates the predetermined action and the other action to be performed on the medical data.

17. The method according to any one of claims 15 and 16, wherein the first machine learning model is provided for use in validating medical data, the method further comprising: Determining a first performance metric of the first machine learning model and a second performance metric of the second machine learning model; as well as In response to the second performance metric exceeding the first performance metric, providing the second machine learning model to replace the first machine learning model in use.

18. The method of claim 11, wherein the historical medical data includes test results for at least one of a plurality of predetermined test items, and generating the first machine learning model includes: Processing the historical medical data by filling in test results for other test items among the plurality of predetermined test items except for the at least one test item, the filled in test results being determined based on the test results of the other test items included in the other historical medical data; and The first machine learning model is generated based on the processed historical medical data.

19. The method according to claim 11, wherein the predetermined action comprises at least one of the following: a second action of further verifying the medical data, a third action of re-running said medical test associated with said medical data, The fourth action of checking the historical patient medical records, a fifth act of checking a reagent reaction in said medical test, a sixth act of inspecting a test sample collected for use in said medical test, a seventh act of examining the medical data in conjunction with a clinical diagnosis, and The eighth action is to check the patient's medication status.

20. The method of claim 1, wherein the machine learning model is provided by a method according to any one of claims 11 to 19.

21. An electronic device, comprising: at least one processor; and At least one memory comprising computer readable instructions which, when executed by the at least one processor of the electronic device, cause the electronic device to perform the steps of the method according to any one of claims 1 to 10 and claim 20.

22. An electronic device, comprising: at least one processor; and At least one memory comprising computer readable instructions which, when executed by the at least one processor of the electronic device, cause the electronic device to perform the steps of the method according to any one of claims 11 to 19.

23. A computer program product comprising instructions which, when executed by a processor of an apparatus, cause the apparatus to perform the steps of the method according to any one of claims 1 to 10 and claim 20.

24. A computer program product comprising instructions which, when executed by a processor of an apparatus, cause the apparatus to perform the steps of the method according to any one of claims 11 to 19.

Images