AI-equipped validation system and method for compliance with appropriate standards

Abstract

The objective is to obtain a validation system and method that complies with appropriate standards and is equipped with AI functions that ensure that artificial intelligence can accurately answer questions about various adverse events related to drugs. [Solution] The system is a validation system 1 that complies with appropriate standards and is equipped with an AI function that tests the AI ​​function after the actual operation, either before or after the actual operation. The system is equipped with a validation management unit 33 that determines the frequency of validation in advance, and an AI unit 9 that prepares standard evaluation test cases for revalidation that have been evaluated by experts and doctors for the number of case patterns, and is capable of revalidating the implemented AI function in the actual environment after learning.

Description

[Technical Field]

[0001] The present invention relates to an appropriate standard-compliant validation system and method equipped with AI functions. [Background technology]

[0002] In recent years, the use of artificial intelligence (AI) has become more widespread. It is also beginning to be used in the medical field. The side effects of drugs are now being handled by doctors, pharmacists, and pharmaceutical manufacturers' safety management and reliability assurance departments.

[0003] For example, Patent Document 1 discloses a technology in which a medication instruction support device 10 includes a control unit 11 that outputs medication instruction sentences related to medications, and the control unit 11 includes: a learning means that causes an artificial intelligence 20 to learn, as learning data, the relationship between dispensing data indicating medications to be dispensed to a medication recipient and medication instruction sentences used for medication instruction based on the dispensing data; a prescription data input means that inputs prescription data including drug identification information that can identify medications indicated on a prescription issued by a doctor to the artificial intelligence 20; a sentence extraction means that causes the artificial intelligence 20 to extract, based on the prescription data and the learning data, medication instruction sentences that can be used for medication instruction based on the prescription data from among multiple medication instruction sentences related to the medication; and an output control means that outputs the extracted medication instruction sentences to an output device 36. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2021-064183 Summary of the Invention [Problem to be solved by the invention]

[0005] However, there is a problem with using artificial intelligence to handle various adverse drug events, as it could lead to serious consequences for patients if the artificial intelligence malfunctions and presents an incorrect answer.

[0006] The present invention was made in consideration of the above-mentioned problems, and aims to provide an appropriate standards-compliant validation system and method equipped with AI functions that ensure that artificial intelligence can accurately handle various adverse events related to drugs. [Means for solving the problem]

[0007] The present invention has been made in consideration of the above problems, and the invention of claim 1 is a validation system for appropriate standards equipped with an AI function that tests the AI ​​function after the actual operation of the system, the validation system comprising: a verification management unit that determines the frequency of verification in advance; The AI ​​unit is provided with a learning model that is machine-learned using severity and / or seriousness as training data and side effects of medication as symptoms as input data, a machine learning unit that machine-learns the learning model, and a verification unit that verifies the learning model, and the AI ​​unit compares standard template cases prepared in advance with those of experts. Standard evaluation test cases for revalidation evaluated by doctors as A production environment for AI functions with a number of case patterns prepared and implemented The verification will be re-verified in a completely frozen reliability assurance environment, and the results of the re-verification will be judged as pass or fail at a frequency determined by the Verification Management Department. It is a validation system that complies with appropriate standards and is equipped with AI functions.

[0008] The invention according to claim 2 is as follows: The verification department will re-verify Check results and evidence after execution Machine Learning Department This is a validation system that complies with appropriate standards and is equipped with the AI ​​function described in claim 1, and a verification unit that records the data in a memory unit and displays and presents it during an audit.

[0009] The invention according to claim 3 is The AI ​​department This is an appropriate standard-compliant validation system equipped with the AI ​​function described in claim 2, which prepares the number of standard evaluation test cases for evaluated cases using the "√N+1" method.

[0010] The invention according to claim 4 is The pass / fail status of the re-verified results is determined based on the accuracy rate of the AI ​​after re-learning. This is an appropriate standard-compliant validation system equipped with the AI ​​function described in claim 3.

[0011] The invention according to claim 5 is The AI ​​department The quality assurance department of each regulated company Verification Management Department This is a validation system for compliance with appropriate standards that incorporates the AI ​​function described in claim 3, which records the results of confirmation and decisions, and incorporates a function that allows for real-time search, display, and presentation in response to audits.

[0012] The invention of claim 6 is a validation method for an AI function that tests an AI function after the start of production, out of the two before and after the start of production, and includes a step of determining the frequency of validation in advance, and when validating a machine-learned learning model using severity and / or seriousness as training data and side effects of medication as symptoms as input data, preparing a number of case patterns as standard evaluation test cases for re-validation that have been evaluated by experts and doctors using standard template cases that have been prepared in advance, and performing re-validation in a reliability assurance environment in which the production environment of the implemented AI function that has been trained is completely frozen, and , decided and a step of determining whether the revalidated results at a specified frequency are acceptable. [Effects of the Invention]

[0013] As described above, the present invention has the effect of enabling artificial intelligence to accurately answer various questions about adverse events related to drugs. [Brief explanation of the drawings]

[0014] The drawings illustrate specific embodiments of the present invention of the notebook according to the present disclosure, including essential features of the invention as well as alternative and preferred embodiments. [Figure 1] This is a schematic diagram of the appropriate standards-compliant validation system equipped with AI functions. [Figure 2] This is a flowchart showing the operation of the appropriate standards-compliant validation system equipped with AI functions. [Figure 3] This is a flowchart showing the operation of the appropriate standards-compliant validation system equipped with AI functions. [Figure 4]This is a flowchart showing the operation of the appropriate standards-compliant validation system equipped with AI functions. [Figure 5] FIG. 1 is an explanatory diagram illustrating artificial intelligence. [Figure 6] This is a block diagram of an appropriate standards-compliant validation system equipped with AI functions. [Figure 7] FIG. 10 is an explanatory diagram illustrating a standard template case. [Figure 8] FIG. 1 is an explanatory diagram illustrating severity. [Figure 9] FIG. 10 is an explanatory diagram illustrating a verification standard. [Figure 10] FIG. 10 is an explanatory diagram illustrating a verification result. DETAILED DESCRIPTION OF THE INVENTION

[0015] Each embodiment will be described in detail below with reference to the accompanying drawings. In these examples, a description of already known technologies will be omitted. Furthermore, the following examples illustrate devices and methods for embodying the technical concept of the invention, and the technical concept of the present invention is not limited to the following. Various modifications can be made to the technical concept of the present invention within the scope of the claims. It should be noted that the drawings are schematic and may differ from the actual product.

[0016] Various embodiments of the present invention will now be described with reference to the drawings.

[0017] FIG. 1 is a schematic diagram of an AI-equipped validation system 1 for appropriateness standards according to this embodiment. The AI-equipped validation system 1 is equipped with advanced technology. The AI-equipped validation system 1 is a computer, and includes a central processing unit (CPU) 3, a read-only memory (ROM) 17 storing a control program running on the CPU 3 and various data, and a random access memory (RAM) 15 for temporarily storing various data. The system also includes an input unit 11, such as a keyboard, and a display unit 13, such as an LCD panel. The system also includes an artificial intelligence (AI) unit 9. The AI ​​unit 9 includes a machine-learned learning model 5, a machine learning unit 7 for machine-learning the learning model 5, and a verification unit 41 for verifying the learning model 5. The verification unit 41 verifies whether or not there are any erroneous processing errors in the AI ​​function of the AI ​​unit 9 after the actual operation, either before or after the actual operation.

[0018] Please refer to Figures 2 to 4. These are flowcharts showing the operation of the appropriateness standard compatible validation system 1 equipped with AI functions.

[0019] First, in step S1, two project phases, "before production start" and "after production start," are tested for the AI ​​unit 9. That is, the processes in steps S3 and S5 below are "before production start" processes, and the processes in steps S7 to S13 are "after production start" processes.

[0020] In step S3, the AI ​​unit 9 determines whether the side effect is serious or non-serious, known or unknown (is it a new side effect? ​​Has it been reported previously or is it mentioned in the package insert?), and the causal relationship between the administered drug and the side effect, and reports this (by email, etc.) to the regulatory authorities (PMDA, Ministry of Health, Labour and Welfare).

[0021] In step S5, the AI ​​unit 9 records the preliminary confirmation work and the results thereof as a document in the ROM 17 in order to ensure reliability before "actual operation," and stores the evidence in the ROM 17 as well.

[0022] In step S7, the AI ​​unit 9 determines in advance an appropriate "frequency" that can be accommodated (weekly, monthly, quarterly, semi-annually, once a year, or daily). The re-verification schedule (frequency and period) is recorded in advance as a verification plan in the appropriate standards-compliant validation system 1 equipped with AI functions, and an alert message is displayed on the system screen (display unit 13) one month before the implementation date of the pre-verification plan, and furthermore, an attention email is sent by email to the management department and responsible person.

[0023] In step S9, the AI ​​unit 9 prepares an appropriate number of case patterns as "standard evaluation test cases for re-verification" that have been evaluated by experts and doctors from the "standard template cases" prepared in advance, and performs re-verification outside of normal business hours in a "reliability assurance environment" in which the "production environment" of the implemented AI function with learning function is completely frozen. The check results and evidence are recorded in this system, and the system is configured to be searchable in real time during audits and to be displayed and presented on the display unit 13.

[0024] In step S11, the AI ​​unit 9 prepares guidelines for how many samples of "standard template cases" evaluated by a human system should be prepared for re-verification using the "√N+1" method commonly used in industry. In other words, if re-verification is performed monthly, the number of cases processed (obtained) in a month is set as "population number: N," and from that number, "√N+1" "evaluated: standard template cases" are prepared in advance and uploaded to the standard sample case template folder (ROM17) of this system, and a function is incorporated that allows them to be referenced and used in real time during periodic case evaluation checks.

[0025] In step S13, the AI ​​unit 9 determines what level to set for the "pass %" of the results of re-verification at a fixed frequency. That is, whether "the accuracy rate after AI re-learning must be 100% to be considered a pass," "a 95% confidence interval for the accuracy rate" or "a 99% confidence interval for the accuracy rate" is determined after confirmation with the quality assurance department of each regulated company, and the determined results are recorded in the ROM 17, and a function is incorporated that allows them to be searched in real time for auditing and displayed / presented on the display unit 13.

[0026] See Figure 5. This shows machine learning by the AI ​​unit 9 "before actual operation." The AI ​​unit 9 "before actual operation" includes a machine learning unit 7. The machine learning unit 7 includes a memory unit 21 that stores information such as side effects of medications, an input unit 23 that inputs information from the memory unit 2, a learning model 5 generated by machine learning, an output unit 25 that outputs answers from the learning model 5, and a memory unit 27 that stores the output answers.

[0027] With this configuration, the learning model 5 performs machine learning using severity (serious, non-serious) and severity (mild, moderate, severe, critical, death) as training data and drug side effects (symptoms) as input data. By performing this machine learning on a large number of cases, it is possible to obtain a learning model 5 that can guarantee reliability.

[0028] Refer to Figure 6. This shows the configuration of the AI ​​unit 9 after "production operation." The AI ​​unit 9 after "production operation" includes a memory unit 31 that stores the number of standard template cases; a verification management unit (reliability assurance department) 33 that manages the verification frequency (weekly, monthly, quarterly, semi-annually, annually, or daily); a machine learning unit 7 that performs machine learning on the learning model 5; a memory unit 35 that stores severity (e.g., severe, moderately severe, safe, etc.); a verification unit 41 that verifies the accuracy of the AI ​​unit 9 after "production operation"; a memory unit 37 that stores verification criteria that serve as the basis for verifying the accuracy of the AI ​​unit 9 after "production operation"; and a memory unit 39 that stores verification results (e.g., pass, fail, etc.). The machine learning unit 7 has a learning model 5.

[0029] See Figure 7. The figure shows the data contents of the storage unit 31 that stores the number of standard template cases (standard evaluation test cases for re-verification that have been evaluated by experts and doctors) for the number of cases. A standard template case is, for example, a case where a patient's symptoms are abnormal blood test results when taking drugs A and B. A plurality of standard template cases (standard evaluation test cases for re-verification that have been evaluated by experts and doctors) are prepared, and the AI ​​unit 9 makes judgments for all cases, and the verification of the AI ​​unit 9 is evaluated based on the results. That is, for example, it is input to the learning model 5 that a patient's symptoms are abnormal blood test results when taking drugs A and B.

[0030] Please refer to Figure 8. This shows the data contents of the memory unit 31 that stores the severity (serious, non-serious) and the severity (mild, moderate, serious, critical, death) corresponding to the above input. For example, the AI ​​unit 9 outputs a result showing that the severity is ranked as severe.

[0031] Please refer to Figure 9. The verification criteria are shown. For example, there is an allergic reaction to drug A. Drug B causes fluid to accumulate in the lungs. Information that caution is required when taking drugs A and B simultaneously is stored in the memory unit 37. Based on this information, the AI ​​unit 9 verifies the output results.

[0032] Please refer to Figure 10. The verification result is shown. For example, the verification result "pass" is stored in the storage unit 39.

[0033] 1 to 10 are merely provided to deepen understanding of the present invention. The present invention is specifically embodied in the following embodiments, and may be embodied in various modifications without substantially departing from the principles of the present invention. All such modifications are included within the scope of the present invention and the disclosure of this specification. [Explanation of symbols]

[0034] 1. AI-equipped validation system that complies with appropriate standards 3 CPU 5 Learning Model 7. Machine Learning Department 9 AI Department 11 Input section 13 Display section 15 RAM 17 ROM

Claims

1. A validation system for appropriate standards equipped with an AI function that tests the AI ​​function after the actual operation of the system between before and after the actual operation, a verification management unit that predetermines the frequency of verification; an AI unit including a learning model trained by machine learning using severity and / or severity as training data and side effects of medication as symptoms as input data, a machine learning unit that trains the learning model by machine learning, and a verification unit that verifies the learning model; Equipped with The AI ​​unit A validation system that complies with appropriate standards and is equipped with AI functions. A number of case patterns are prepared as standard evaluation test cases for re-validation, which have been evaluated by experts and doctors using standard template cases prepared in advance, and re-validation is performed in a reliability assurance environment where the production environment of the implemented AI function with learning function is completely frozen, and the system determines whether the re-validated results pass at the frequency determined by the validation management department.

2. A validation system that complies with appropriate standards and is equipped with the AI ​​function described in claim 1, wherein the verification unit records the check results and evidence after performing the re-verification in the memory unit of the machine learning unit, and displays and presents them during an audit.

3. A validation system that complies with appropriate standards and is equipped with the AI ​​function described in claim 2, wherein the AI ​​section prepares the number of standard evaluation test cases for evaluated cases using the "√N+1" method.

4. A validation system that complies with appropriate standards and is equipped with the AI ​​function described in claim 3, characterized in that whether the re-verified results pass or fail is determined based on the accuracy rate after the AI ​​section re-learns.

5. A validation system that complies with appropriate standards and is equipped with the AI ​​function described in claim 3, in which the AI ​​department records the results of confirmation and decisions made by the verification management department, which is the reliability assurance department of each regulated company, and incorporates a function that allows it to search, display, and present the results in real time as a response to audits.

6. A validation method for appropriate standards equipped with an AI function that tests the AI ​​function after production operation between before production operation and after production operation, predetermining a frequency of verification; When verifying a machine-learned learning model using severity and / or severity as training data and side effects of medication as symptoms as input data, a number of case patterns are prepared as standard evaluation test cases for re-verification that have been evaluated by experts and doctors using standard template cases prepared in advance, and re-verification is performed in a reliability assurance environment in which the production environment of the implemented learned AI function is completely frozen, and a process is performed to determine whether the re-verification results at the determined frequency are acceptable; A validation method that complies with appropriate standards and is equipped with AI functions, including:

Images