AI-powered validation system and method for meeting appropriate standards.

Abstract

The objective is to obtain a validation system and method that incorporates AI functionality to ensure that artificial intelligence can accurately answer questions about various adverse drug events. [Solution] This is a validation system 1 that complies with appropriate standards and is equipped with an AI function that tests the AI ​​function after it goes into production, rather than before it goes into production. It comprises a validation management unit 33 that predetermines the frequency of verification, and an AI unit 9 that prepares a number of case patterns of standard evaluation test cases for re-verification that have been evaluated by experts and doctors, and can perform re-verification in the production environment of the implemented, trained AI function.

Description

Technical Field

[0001] The present invention relates to an appropriate standard-compliant validation system and method equipped with an AI function.

Background Art

[0002] In recent years, the use of artificial intelligence (AI) has been actively carried out. Also in the medical field, artificial intelligence has come to be used. Regarding the side effects of drugs, it has come to play a part in the work of doctors, pharmacists, pharmaceutical manufacturers' safety management departments, and reliability assurance departments.

[0003] For example, Patent Document 1 discloses a medication guidance support device 10 including a control unit 11 that outputs a medication guidance article regarding a drug, wherein the control unit 11 causes an artificial intelligence 20 to learn, as learning data, the relationship between dispensing data indicating the drug dispensed to a patient and the medication guidance article used for the medication guidance based on the dispensing data; prescription data input means for inputting, to the artificial intelligence 20, prescription data including drug identification information capable of identifying the drug indicated in the prescription issued by a doctor; article extraction means for causing the artificial intelligence 20 to extract, from a plurality of medication guidance articles regarding the drug, a medication guidance article that can be used for the medication guidance based on the prescription data, based on the prescription data and the learning data; and output control means for outputting the extracted medication guidance article to an output device 36.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, replacing information about various adverse drug events with artificial intelligence presents a problem: if the AI ​​malfunctions and provides incorrect answers, it could lead to serious consequences for patients.

[0006] This invention has been made in view of the above problems, and aims to provide a validation system and method that incorporates an AI function to ensure that artificial intelligence can reliably handle various adverse events related to drugs, in accordance with appropriate standards. [Means for solving the problem]

[0007] The present invention has been made in view of the above problems, and the invention of claim 1 is an appropriate standard compliance validation system equipped with an AI function that tests the AI ​​function after it has been put into production, rather than before it has been put into production, and comprises a verification management unit that determines the frequency of verification in advance, and an AI unit that prepares a number of case patterns of standard evaluation test cases for re-verification that have been evaluated by experts and doctors, and is capable of performing re-verification in the production environment of the implemented, trained AI function.

[0008] The invention according to claim 2 is a validation system that incorporates the AI ​​function described in claim 1 and includes a verification unit that records the check results and evidence after the AI ​​unit has performed the checks in a storage unit and displays and presents them during an audit.

[0009] The invention according to claim 3 is an appropriate standard-compliant validation system equipped with the AI ​​function described in claim 2, wherein the number of standard evaluation test cases for evaluated cases is prepared using the "√N+1" method.

[0010] The invention according to claim 4 is a validation system that incorporates the AI ​​function described in claim 3, wherein the pass / fail result of re-verification at a certain frequency is determined by one of the following: the accuracy rate after AI retraining must be 100%, an accuracy rate of 95% confidence interval is considered a pass, or an accuracy rate of 99% confidence interval is considered a pass.

[0011] The invention according to claim 5 is a validation system that complies with appropriate standards, equipped with the AI ​​function described in claim 3, which records the results of confirmation and decisions made by the reliability assurance department of each regulated company, and incorporates a function that allows for real-time searching, display, and presentation of these results as a response during audits.

[0012] The invention according to claim 6 is an AI-based validation method that conforms to appropriate standards and includes an AI function for testing the AI ​​function after it has gone into production, and comprises the steps of: a reliability assurance department predetermining the frequency of verification; and an AI department preparing a number of case patterns of standard evaluation test cases for re-verification that have been evaluated by experts and doctors, and enabling re-verification to be performed in the production environment of the implemented, trained AI function. [Effects of the Invention]

[0013] As described above, the present invention has the effect of enabling artificial intelligence to reliably answer questions about various adverse events related to drugs. [Brief explanation of the drawing]

[0014] The drawings illustrate specific embodiments of the present invention as described in the Notes relating to this disclosure, including not only essential components of the invention but also optional and preferred embodiments. [Figure 1] This is a schematic diagram of an AI-powered validation system that complies with appropriate standards. [Figure 2] This flowchart shows the operation of an AI-powered validation system that complies with appropriate standards. [Figure 3] This flowchart shows the operation of an AI-powered validation system that complies with appropriate standards. [Figure 4] This flowchart shows the operation of an AI-powered validation system that complies with appropriate standards. [Figure 5] This is an explanatory diagram for artificial intelligence. [Figure 6]It is a block diagram of an appropriate standard compliant validation system equipped with an AI function. [Figure 7] It is an explanatory diagram explaining a standard template case. [Figure 8] It is an explanatory diagram explaining severity. [Figure 9] It is an explanatory diagram explaining verification criteria. [Figure 10] It is an explanatory diagram explaining verification results.

Embodiments for Carrying Out the Invention

[0015] Hereinafter, each embodiment will be described in detail while referring to the attached drawings. In this example, descriptions of technologies that are already known will be omitted. Also, the devices and methods for embodying the technical idea of the invention are exemplified, and the technical idea of the present invention is not limited to the following. The technical idea of the present invention can be variously modified within the scope of the matters described in the claims. In particular, it should be noted that the drawings are schematic and different from the actual ones.

[0016] Hereinafter, various embodiments of the present invention will be described while referring to the drawings.

[0017] Figure 1 is a schematic diagram of the compliance standard - compliant validation system 1 equipped with an AI function. The compliance standard - compliant validation system 1 equipped with an AI function is equipped with advanced technology. The compliance standard - compliant validation system 1 equipped with an AI function consists of a computer, including a CPU (Central Processing Unit) 3, a control program operating on the CPU 3, and a ROM (Read - only Memory) 17 storing various data, etc., and a RAM 15 for temporarily storing various data. Further, it is equipped with an input unit 11 such as a keyboard and a display unit 13 such as a liquid crystal display panel. On the other hand, it is equipped with an AI unit 9 which is an artificial intelligence. The AI unit 9 is equipped with a learned learning model 5, a machine learning unit 7 for machine - learning the learning model 5, and a verification unit 41 for verifying the learning model 5. Also, the verification unit 41 verifies whether there is an incorrect process in the AI function of the AI unit 9 after the actual operation compared to before the actual operation.

[0018] Refer to FIGS. 2 to 4. It is a flowchart showing the operation of the compliance standard - compliant validation system 1 equipped with an AI function.

[0019] First, in step S1, the AI unit 9 conducts tests on two project phases, namely, "before actual operation" and "after actual operation". That is, the processes in steps S3 and S5 below are the processes "before actual operation", and the processes from step S7 to step S13 are the processes "after actual operation".

[0020] In step S3, the AI unit 9 determines factors such as "serious / non - serious", "known / unknown (new side effect? Reported before or described in the package insert?)", and the causal relationship between the administered drug and the side effect, and reports (reports by e - mail, etc.) to the regulatory authorities (PMDA, Ministry of Health, Labour and Welfare).

[0021] In step S5, the AI unit 9, in advance "before actual operation", conducts pre - confirmation work to ensure its reliability guarantee, records the results as a document in the ROM 17, and stores the evidence in the ROM 17 together.

[0022] In step S7, the AI ​​unit 9 predetermines an appropriate "frequency" at which it can respond (weekly, monthly, quarterly, semi-annually, annually, or daily). This re-verification schedule (frequency and duration) is recorded in advance as a verification plan in the appropriate standard compliance validation system 1 equipped with AI functionality. One month before the implementation date of this pre-verification plan, an alert message is displayed on the system screen (display unit 13), and an attention email is also sent to the management department and the person in charge.

[0023] In step S9, the AI ​​unit 9 prepares an appropriate number of case patterns as "standard evaluation test cases for re-verification" that have been evaluated by experts and doctors, using pre-prepared "standard template case examples". This re-verification is performed outside of normal business hours in a "reliability assurance environment" where the "production environment" of the implemented, trained AI function is completely frozen. The check results and evidence are recorded in this system and can be searched in real time during audits, and displayed and presented on the display unit 13.

[0024] In step S11, the AI ​​unit 9 prepares the number of human-evaluated "standard template cases" samples to re-verify using the "√N+1" method commonly used in industry. In other words, if re-verification is to be done monthly, the number of cases processed (obtained) in a month is set as the "destination: N," and from that, "√N+1" "evaluated: standard template cases" are prepared in advance and uploaded to the system's standard sample case template folder (ROM17). A function is then incorporated to allow real-time reference and use during periodic case evaluation checks.

[0025] In step S13, the AI ​​unit 9 determines what level of "passing percentage" the results of re-verification at a certain frequency should be. Specifically, whether to "pass only if the accuracy rate after AI retraining is 100%", "pass with an accuracy rate within a 95% confidence interval", or "pass with an accuracy rate within a 99% confidence interval", the results determined after confirmation by the reliability assurance department of each regulated company are recorded in the ROM 17, and a function is incorporated to allow real-time searching and display / presentation on the display unit 13 as a response during audits.

[0026] Refer to Figure 5. This is the machine learning of the AI ​​unit 9 "before going live". The AI ​​unit 9 "before going live" includes the machine learning unit 7. The machine learning unit 7 comprises a memory unit 21 that stores drug side effects, etc., an input unit 23 that receives information from the memory unit 2, a learning model 5 generated by machine learning, an output unit 25 that outputs a response from the learning model 5, and a memory unit 27 that stores the output response.

[0027] In this configuration, the learning model 5 uses the severity (severe, non-severe) and severity level (mild, moderate, severe, critical, fatal) as training data, and the drug side effects (symptoms) as input data to perform machine learning. By performing this machine learning on a large number of cases, it is possible to obtain a reliable learning model 5.

[0028] Refer to Figure 6. This shows the configuration of the AI ​​unit 9 "after going live". The AI ​​unit 9 "after going live" includes a memory unit 31 that stores a number of standard template cases, a verification management unit (reliability assurance unit) 33 that manages the frequency of verification (weekly, monthly, quarterly, semi-annually, once a year, or daily), a machine learning unit 7 that performs machine learning on the learning model 5, a memory unit 35 that stores the severity (e.g., critical, moderately critical, safe, etc.), a verification unit 41 that verifies the accuracy of the AI ​​unit 9 "after going live", a memory unit 37 that stores the verification criteria that serve as the basis for verifying the accuracy of the AI ​​unit 9 "after going live", and a memory unit 39 that stores the verification results (e.g., pass, fail, etc.). The machine learning unit 7 has the learning model 5.

[0029] Refer to Figure 7. This shows the data contents of the memory unit 31, which stores a number of standard template cases (standard evaluation test cases for re-verification that have been evaluated by experts and physicians) for each case. A standard template case is, for example, a case in which a patient's symptoms are abnormal in blood tests after taking drugs A and B. Multiple such standard template cases (standard evaluation test cases for re-verification that have been evaluated by experts and physicians) are prepared, and the AI ​​unit 9 makes a judgment for all cases and evaluates the verification of the AI ​​unit 9 based on the results. That is, for example, the fact that a patient's symptoms are abnormal in blood tests after taking drugs A and B is input to the learning model 5.

[0030] Refer to Figure 8. This shows the data contents of the memory unit 31, which stores the severity (severe, not severe) and severity level (mild, moderate, severe, critical, fatal) corresponding to the above inputs. For example, the AI ​​unit 9 outputs a result where the severity level is ranked as critical.

[0031] Refer to Figure 9, which shows the verification criteria. For example, drug A causes allergies. Drug B causes fluid to accumulate in the lungs. Information that caution is needed when taking drugs A and B simultaneously is stored in the memory unit 37. Based on this information, the AI ​​unit 9 verifies the output results.

[0032] Refer to Figure 10, which shows the verification results. For example, the memory unit 39 stores the verification result as "pass".

[0033] The above-described explanation of Figures 1 to 10 is merely for the purpose of deepening understanding of the present invention. The present invention is specifically implemented in the embodiments described below and may be implemented by various modifications without substantially exceeding the principles of the present invention. All such modifications are included within the scope of the present invention and the disclosure of this specification. [Explanation of symbols]

[0034] 1. AI-powered validation system for compliance with appropriate standards. 3 CPU 5. Learning Models 7. Machine Learning Department 9 AI Department 11 Input section 13 Display section 15 RAM 17 ROM

Claims

1. A validation system equipped with an AI function that tests the AI ​​function after going into production, rather than before going into production, and is designed to comply with appropriate standards. An AI-powered validation system that conforms to appropriate standards, comprising a validation management unit that predetermines the frequency of verification, and an AI unit that prepares a number of standard evaluation test cases for re-verification evaluated by experts and doctors, and enables re-verification to be performed in the production environment of the implemented, trained AI function.

2. A validation system equipped with the AI ​​function described in claim 1, comprising a verification unit that records the check results and evidence after the AI ​​unit has performed an action in a storage unit and displays and presents them during an audit.

3. The appropriate standard-compliant validation system equipped with the AI ​​function described in claim 2, wherein the number of standard evaluation test cases for evaluated cases is prepared using the "√N+1" method.

4. A validation system equipped with the AI ​​function described in claim 3, wherein the passing result of re-verification at a certain frequency is determined by one of the following criteria: the accuracy rate after AI retraining must be 100%, an accuracy rate within a 95% confidence interval is considered a passing result, or an accuracy rate within a 99% confidence interval is considered a passing result.

5. A validation system for compliance with appropriate standards equipped with the AI ​​function described in claim 3, which records the results of confirmation and decisions made by the reliability assurance department of each regulated company, and incorporates a function that allows for real-time searching, display, and presentation of these results as a response during audits.

6. A validation method incorporating an AI function that tests the AI ​​function after it goes into production, rather than before it goes into production, and is designed to comply with appropriate standards. An AI-powered validation method that conforms to appropriate standards, comprising the steps of: a reliability assurance department predetermining the frequency of verification; and an AI department preparing a number of case patterns of standard evaluation test cases for re-verification that have been evaluated by experts and doctors, and enabling re-verification to be performed in the production environment of the implemented, trained AI function.

Images