Quality control document evaluation system, program, and quality control document evaluation method
The quality control document evaluation system efficiently creates compliant documents by analyzing their consistency with regulations using a trained model, addressing the time-consuming nature of meeting complex quality control requirements.
Patent Information
- Application Number
- JP2025077862
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2025-05-08
- Publication Date
- 2026-11-19
- Estimated Expiration
- 2045-05-08
AI Technical Summary
The creation of quality control documents that conform to regulations such as QMS Ministerial Ordinance and ISO is time-consuming due to the complexity and variety of requirements.
A quality control document evaluation system that includes a storage processing unit, reception unit, evaluation unit, generation unit, and display processing unit, utilizing a trained model to analyze the semantic relationship between documents and rule information, identifying inconsistencies, and generating suggestions for improvement.
Efficiently creates quality control documents that conform to quality control rules by identifying and addressing inconsistencies, thereby reducing the time and effort required.
Smart Images

Figure 2026181997000001_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a quality control document evaluation system, a program, and a quality control document evaluation method.
Background Art
[0002] Conventionally, systems related to document management and business management are known. For example, Patent Document 1 describes a system for exchanging clinical trial management files between pharmaceutical companies and medical institutions.
Prior Art Documents
Patent Documents
[0003]
Patent Document 1
Summary of the Invention
Problems to be Solved by the Invention
[0004] By the way, the quality control rules for medical devices, pharmaceuticals and the like, such as the QMS Ministerial Ordinance and ISO, are stipulated in detail for various requirements. For this reason, the creation and operation of quality control documents related to quality control that conform to rules such as the QMS Ministerial Ordinance and ISO require the creation of documents that satisfy these various requirements, which is time-consuming.
[0005] An object of the present invention is to provide a quality control document evaluation system, a program, and a quality control document evaluation method that can efficiently create quality control documents conforming to quality control rules.
Means for Solving the Problems
[0006] The quality control document evaluation system is a quality control document evaluation system for evaluating quality control documents, comprising: a storage processing unit that stores rule information relating to quality control rules in a database; a reception unit that receives input of a document to be evaluated, which is a quality control document to be evaluated; an evaluation unit that analyzes the semantic relationship between the document to be evaluated and the information in the database and evaluates the consistency of the document to be evaluated with respect to the rule information using a trained model; a generation unit that, if inconsistencies that do not match the rule information in the document to be evaluated are identified based on the consistency evaluation results, generates information about the inconsistencies and suggestions for improvement of the document to be evaluated; and a display processing unit that displays the information about the inconsistencies and the suggestions for improvement. [Effects of the Invention]
[0007] According to the present invention, quality control documents that conform to quality control rules can be efficiently created. [Brief explanation of the drawing]
[0008] [Figure 1] This is a schematic diagram showing a quality control document evaluation system and documents to be evaluated according to one embodiment of the present invention. [Figure 2] This is a schematic diagram illustrating the modularized functions of a quality control system. [Figure 3] This is a block diagram showing the hardware configuration of a quality control document evaluation system. [Figure 4] This block diagram shows the functional configuration of a quality control document evaluation system. [Figure 5] This is a schematic diagram showing an example of the display screen for registering company information. [Figure 6] This is a schematic diagram showing an example of the display screen for registering item information. [Figure 7] This is a schematic diagram illustrating the document hierarchy in quality control. [Figure 8] This is a schematic diagram illustrating an example of a high-level document in quality control. [Figure 9] This is a schematic diagram illustrating examples of subordinate documents in quality control. [Figure 10] This is a schematic diagram illustrating an example of how the evaluation results of consistency between rule information and the documents being evaluated by the quality control document evaluation system are displayed. [Figure 11] This is a schematic diagram showing an example of a display screen for requesting a review of a document to be evaluated from a reviewer. [Figure 12] This is a schematic diagram illustrating an example of the display of the review screen showing the evaluation results of the consistency between rule information and the documents being evaluated by the quality control document evaluation system. [Figure 13] This is a schematic diagram showing an example of a display screen that shows a list of quality control documents registered in the quality control system. [Figure 14] This is a schematic diagram illustrating an example of how quality control documents registered in a quality control system can be displayed using keyword search. [Figure 15] This flowchart shows the flow of the quality control document evaluation process related to one embodiment of the present invention. [Figure 16] This flowchart shows the flow of consistency evaluation processing in the quality control document evaluation process according to one embodiment of the present invention. [Figure 17] This flowchart shows the flow of checkpoint processing in the quality control document evaluation process according to one embodiment of the present invention. [Modes for carrying out the invention]
[0009] Embodiments of the present invention will be described below with reference to the drawings. Figure 1 is a schematic diagram showing a quality control document evaluation system 1 and a document to be evaluated 2 according to one embodiment of the present invention.
[0010] Quality control document evaluation system 1 is a system included in a quality control system (not shown) that supports the user's quality control.
[0011] The quality management system mainly includes a document management function for managing documents, an event management function for managing events, an education and training function for educating and training employees, and the like. Figure 2 is a schematic diagram showing the modularized functions included in the quality management system 100. In this specification, the term "document" means an electronic document in which character information and image information are digitized. One document means, for example, one digitized law, regulation, standard, manual, procedure document, or the like.
[0012] As shown in Figure 2, in the quality management system, functions such as a basic function 61, a document and record management function 62, an AI (Artificial Intelligence) support function 63, an education and training function 64, and an event management function 65 are modularized. The basic function 61 includes, for example, functions for setting user management, authority management, audit logs, notifications, multilingual support, and the like. The document and record management function 62 includes, for example, functions such as storage and editing of documents, version management, approval workflows, difference display for showing different parts between documents, access restriction to stored documents, and electronic signatures. The AI support function 63 includes, for example, functions such as presentation of required documents and actions, automatic generation of document drafts, automatic document checking, automatic creation of test questions, and analysis of accumulated data. The education and training function 64 includes, for example, functions such as creation and distribution of test questions, management of attendance records, and judgment of the competence of respondents. The event management function 65 includes template and process management functions, alert functions, and the like. Since each function of the quality management system is modularized, the system has a highly scalable function design.
[0013] The quality management document evaluation system 1 is included in the quality management system, and is a system mainly equipped with functions included in the document and record management function 62, the AI support function 63, and the like. Specifically, the quality management document evaluation system 1 performs quality management document evaluation processing that evaluates whether a created quality management document is a document conforming to rule information related to quality management rules, and presents information related to inconsistent parts (hereinafter referred to as inconsistent part related information) and the like.
[0014] A document consistent with the rule information is a document written in accordance with the requirements indicated by the rule information. In other words, it is a document that conforms to the rule information. Furthermore, a quality control document is a document related to quality control. In the following explanation, the quality control document to be evaluated by the Quality Control Document Evaluation System 1 will be referred to as the document to be evaluated 2.
[0015] The quality control document evaluation system 1 uses a database 3 containing rule information and a trained model 4 to evaluate the consistency between the rule information and the document to be evaluated 2, and presents the evaluation results to the user. The consistency between the rule and the document to be evaluated 2 refers to the degree to which the description in the document to be evaluated 2 satisfies the requirements indicated by the rule information. In this embodiment, quality control for medical devices and pharmaceuticals will be explained as an example. The database 3 and the trained model 4 will be described later.
[0016] An example of the hardware configuration of Quality Control Document Evaluation System 1 will be explained with reference to Figure 3. Figure 3 is a block diagram showing the hardware configuration of Quality Control Document Evaluation System 1.
[0017] As shown in Figure 3, the quality control document evaluation system 1 includes a computer 18, a storage unit 13, a communication unit 14, an input unit 15, and a display unit 16. A bus 17 and the like connect these units.
[0018] The computer 18 includes a processor 10 and ROM (read-only memory) 11 and RAM (random-access memory) 12 as main memory. The processor 10 is a CPU (central processing unit), MPU (micro processing unit), SoC (system on a chip), DSP (digital signal processor), GPU (graphics processing unit), ASIC (application specific integrated circuit), PLD (programmable logic device), or FPGA (field-programmable gate array), etc. Alternatively, the processor 10 is a combination of several of these. Furthermore, the processor 10 may also be a combination of these with hardware accelerators, etc. The processor 10 controls each part to realize various functions of the quality control document evaluation system 1 based on programs such as firmware, system software, and application software stored in the ROM 11, RAM 12, or auxiliary memory which is part of the memory unit 13. Note that some or all of the said programs may be incorporated into the circuit of the processor 10.
[0019] The memory unit 13 is a storage area for various programs and data necessary for the hardware group to function as a quality control document evaluation system 1, and can be composed of ROM, RAM, flash memory, a semiconductor drive (SSD), or a hard disk drive (HDD). Specifically, the memory unit 13 stores programs that cause the computer 18 to execute each function of this embodiment. The memory unit 13 may also include a database 3 for storing rule information, etc. The memory unit 13 may also store a trained model 4. In this embodiment, the database 3 and the trained model 4 are located outside the quality management system, such as on the cloud.
[0020] The communication unit 14 performs processing to enable the quality control document evaluation system 1 to communicate with other devices via the network. For example, the quality control document evaluation system 1 can connect to the database 3 via the communication unit 14 and refer to or retrieve information stored in the database 3. For example, the quality control document evaluation system 1 can evaluate the consistency of the document to be evaluated 2 by communicating with the trained model 4 via the communication unit 14.
[0021] The input unit 15 is a user interface electrically connected to the computer 18. The input unit 15 consists of buttons, a mouse, a keyboard, a display, etc. The display is composed of, for example, a liquid crystal display (LCD) or an electro-luminescent (OLED) display, and a touch panel is provided on the image display surface of the display to detect the user's touch position. The user can input information by touching the image display surface of the display.
[0022] The display unit 16 is a user interface electrically connected to the computer 18. The display unit 16 is composed of a display. Images and various information transmitted from the storage unit 13, communication unit 14, input unit 15, computer 18, etc. are displayed on the display unit 16.
[0023] Next, the various functions realized by the hardware configuration of the Quality Control Document Evaluation System 1 will be explained with reference to Figure 4. As shown in Figure 4, the Quality Control Document Evaluation System 1 includes a reception unit 101, a storage processing unit 102, an evaluation unit 110, a generation unit 103, a display processing unit 104, a review support unit 105, and a machine learning unit 106 as functional units that operate on the processor 10.
[0024] The reception unit 101 performs processing to receive information entered by the user. For example, the reception unit 101 receives input of the evaluation target document 2 entered by the user via the communication unit 14 or the input unit 15. Also, for example, the reception unit 101 performs processing to receive rule information and evaluation-related information via the communication unit 14 or the input unit 15.
[0025] Rule information may be information described in a document concerning quality control rules, for example, or it may be information presented in a form other than a document. Examples of non-documental forms include images, audio, and video. In this embodiment, the rule information is information described in a document.
[0026] Rules and information regarding quality control for medical devices and pharmaceuticals include, for example, information contained in the QMS (Quality Management System) Ordinance, GMP (Good Manufacturing Practice) Ordinance, ISO (International Organization for Standardization) 13485, ISO 14971, and other relevant guidelines and quality manuals.
[0027] The rule information preferably includes information contained in the internal regulations concerning quality control within the organization to which the user belongs (hereinafter referred to as "internal regulations"), such as company rules. This allows the user to confirm whether the document to be evaluated (Document 2) conforms not only to laws and regulations concerning quality control of medical devices and pharmaceuticals, and standards of external organizations, but also to the regulations of their own organization.
[0028] Evaluation-related information is information that affects the evaluation of consistency between rule information and the document under evaluation 2, and includes, for example, organizational information such as company information and item information. Organizational information is information about the organization to which the user of the quality control document evaluation system 1 belongs. Item information is information about the items such as medical devices and pharmaceuticals that the user handles and that are the subject of the document under evaluation 2.
[0029] The company information and item information as organizational information, and the input of this information, will be explained with reference to Figures 5 and 6. Figure 5 is a schematic diagram showing an example of the display of the company information registration screen 51, which is part of the system registration screen 5 displayed on the display unit 16. Figure 6 is a schematic diagram showing an example of the display of the item information registration screen 52, which is part of the system registration screen 5 displayed on the display unit 16.
[0030] The system registration screen 5 is a screen for registering information related to the evaluation target document 2, including evaluation-related information. The system registration screen 5 consists of a company information registration screen 51 and an item information registration screen 52. As shown in Figures 5 and 6, a display bar 50 is displayed at the top of the system registration screen 5 to indicate whether the page displayed on the display unit 16 is the company information registration screen 51 or the item information registration screen 52.
[0031] The company information registration screen 51 is a screen for registering information about the organization to which the user belongs, and consists of multiple input fields 53 and page-turning buttons 54. For example, as shown in Figure 5, the input fields 53 include fields for entering the company name, representative name, address, telephone number, email address, website, year of establishment, number of employees, and business description. The reception unit 101 accepts the information by reading the information entered in each input field 53. The company information may also include information showing the organizational structure, such as an organizational chart, business license information, manufacturing plant information, and certification information. Business license information is, for example, information showing the type of business license that the organization to which the user belongs has obtained. Examples of business license types include manufacturing license, manufacturing and sales license, and repair license. Manufacturing plant information is information about the manufacturing plant of the user's products. Certification information is, for example, information showing the type of certification, such as ISO 13485, that the organization to which the user belongs has obtained.
[0032] The item information registration screen 52 is an image for registering item information for the product subject to evaluation document 2, and consists of multiple input fields 55, a page return button 56, and a registration button 57. For example, as shown in Figure 6, the input fields 55 include fields for entering the product's item name, item code, general name classification, intended use or effect, manufacturing plant, storage conditions, sterilization method, applicable standards, etc.
[0033] The memory processing unit 102 executes the process of storing the evaluation target document 2, rule information, and evaluation-related information acquired via the reception unit 101. The memory processing unit 102 stores the evaluation target document 2 in the storage unit 13 and stores the rule information and evaluation-related information in the database 3. Preferably, the rule information and evaluation-related information as documents are divided into multiple semantic chunks by the data processing unit 111 (described later) and stored in the database 3 in a vectorized state.
[0034] The evaluation unit 110 performs a consistency evaluation process that evaluates the consistency between the rule information and the document to be evaluated 2, based on the document to be evaluated 2 received by the reception unit 101 and the information in the database 3.
[0035] The consistency evaluation process analyzes the semantic relationship between the document to be evaluated (Document 2) and the information in the database (Database 3). Based on the results of this analysis, the consistency of Document 2 with the rule information is evaluated. Semantic relationship refers to the degree to which different pieces of information are semantically related, such as between different documents, between sentences, or between words.
[0036] In this embodiment, the evaluation unit 110 uses Retrieval Augmented Generation (RAG) to evaluate the consistency between the rule information and the document to be evaluated 2. Specifically, the evaluation unit 110 inputs the document to be evaluated 2 and the information in the database 3 related to the document to be evaluated 2 into the trained model 4, and has the trained model 4 evaluate the degree of match between the document to be evaluated and the rule information. Then, the evaluation unit 110 evaluates the degree of match between the document to be evaluated and the rule information based on the evaluation results output from the trained model 4.
[0037] The evaluation unit 110 includes a data processing unit 111, a hierarchical discrimination unit 112, a search processing unit 113, and a language processing unit 114.
[0038] The data processing unit 111 performs a process to convert documents such as the document to be evaluated 2, rule information, and evaluation-related documents into information that can be analyzed by a computer.
[0039] The data processing unit 111 divides the text data, such as the evaluation target document 2, received by the reception unit 101 into multiple semantic chunks. For example, the data processing unit 111 divides the sentence "5.3 Final inspection: Inspect the appearance, dimensions, and function of the product. The inspection rate will be determined separately." which is described in the product inspection procedure manual as the evaluation target document 2, into the following three chunks A to C. Chunk A: "5.3 Final Inspection" Chunk B: "Inspect the product's appearance, dimensions, and function." Chunk C: "The testing rate will be determined separately."
[0040] The data processing unit 111 vectorizes each semantic chunk. For example, the data processing unit 111 vectorizes chunk B as follows: Chunk B: [0.12, -0.34, 0.56...]
[0041] The rule information and evaluation-related information, which have been divided into multiple semantic chunks and vectorized, are transferred to the memory processing unit 102 and stored in the database 3.
[0042] The hierarchy determination unit 112 performs a process to determine which hierarchy of the document hierarchy in quality control the vectorized document belongs to.
[0043] In this embodiment of the quality control document evaluation system 1, documents related to the quality control of medical devices and pharmaceuticals are divided into multiple hierarchies according to their type and managed accordingly. The document hierarchy in the quality control of medical devices and pharmaceuticals (hereinafter referred to as the document hierarchy) will be explained with reference to Figures 7 to 9. Figure 7 is a schematic diagram showing the document hierarchy in the quality control of medical devices and pharmaceuticals. Figure 8 is a schematic diagram showing an example of a higher-level document in the quality control of medical devices and pharmaceuticals. Figure 9 is a schematic diagram showing an example of a lower-level document in the quality control of medical devices and pharmaceuticals.
[0044] As shown in Figure 7, in this embodiment, the documents are arranged in the following order from higher-level documents (hereinafter referred to as "higher-level documents") to lower-level documents (hereinafter referred to as "lower-level documents"): documents describing basic concepts of quality such as "Quality Manuals," documents concerning basic operational rules for implementing those basic concepts such as "Regulations," documents describing the actual work performed by each department such as "Procedure Manuals," documents used for work such as "Plans, Ledgers, and Forms," and records of actual work performed such as "Quality Records." The description in Evaluation Target Document 2 must be higher in level than Evaluation Target Document 2 and must conform to the requirements of related documents.
[0045] The hierarchy determination unit 112 determines the hierarchy to which a document belongs, for example, based on the title of the vectorized document. As shown in Figure 8, the hierarchy determination unit 112 classifies "Quality Manual" and "Regulations" as higher-level documents, and various "Procedure Manuals" such as Standard Operating Procedures (SOPs) and work instructions, "Plans, Ledgers, Forms," and "Quality Records" as lower-level documents. Among the lower-level documents, for example, "Quality Records" are classified as lower-level documents than "Procedure Manuals" because the record items are filled in according to the corresponding work instructions and other various "Procedure Manuals." In this embodiment, each rule information is stored in the database 3 with a predetermined document hierarchy.
[0046] In this embodiment, the hierarchy determination unit 112 determines the document hierarchy of the evaluation target document 2, which has been vectorized by the data processing unit 111. As a result, the evaluation unit 110 can identify which level of the document hierarchy the evaluation target document 2 belongs to, and can identify the higher-level document which is rule information that the evaluation target document 2 must comply with.
[0047] The search processing unit 113 performs a process to extract rule information and evaluation-related information related to the document to be evaluated 2 from the database 3. The search processing unit 113 may also calculate the similarity between vectorized documents and extract information related to the document to be evaluated 2 from the database 3 based on the calculated similarity. Similarity may be obtained, for example, by calculating the dot product between vectorized documents, sentences, or chunks. For example, the search processing unit 113 may calculate the similarity between the title of the vectorized document to be evaluated 2 and the titles of the vectorized rule information and evaluation-related information, and extract information related to the document to be evaluated 2 from the database 3 based on the calculated similarity. In this embodiment, the search processing unit 113 identifies documents higher in rank than the document to be evaluated 2 among the rule information stored in the database 3, and extracts documents related to the document to be evaluated 2 from among the identified higher-ranking documents.
[0048] For example, the search processing unit 113 may extract a predetermined number of documents from among multiple documents in the database 3 that have the highest similarity. The predetermined number may be, for example, 10 or more, or less than 10. The search processing unit 113 may extract an entire document related to the document to be evaluated 2, or it may extract only the part of a document related to the document to be evaluated 2. For example, if the rule information related to the document to be evaluated 2 is a QMS ordinance, the search processing unit 113 may extract the entire QMS ordinance, extract only the articles of the QMS ordinance related to the document to be evaluated 2, or extract only the articles of the QMS ordinance that contain requirements for the document to be evaluated 2. For example, the search processing unit 113 may extract only the articles that contain requirements for the document to be evaluated 2, as follows. "Article 44 of the QMS Ordinance": "The standards, methods, and results of product inspections must be documented."
[0049] The search processing unit 113 inputs the document to be evaluated 2, the extracted rule information and evaluation-related information, and a prompt to evaluate the consistency between the extracted rule information and the document to be evaluated 2 as input information to the trained model 4.
[0050] The language processing unit 114 evaluates the consistency between the rule information and the document to be evaluated 2 based on the response from the trained model 4, which is output when the input information is input to the trained model 4.
[0051] The trained model 4 in this embodiment is a large-scale language model that interprets document content, including the context and nuances of the document to be evaluated 2, rule information, and evaluation-related documents, through natural language understanding. By interpreting the document content, the trained model 4 identifies the requirements that the document to be evaluated 2 must apply from among the requirements indicated by the rule information. The trained model 4 then determines whether the document to be evaluated 2 matches the identified requirements and outputs the result of this determination as the consistency evaluation result for the document to be evaluated 2.
[0052] Since the pre-trained model 4 has both rule information and evaluation-related information input, the pre-trained model 4 can identify the requirements that the document under evaluation 2 must apply from among the requirements indicated by the rule information, and their contents, based on the evaluation-related information. In other words, the language processing unit 114 identifies the requirements that the document under evaluation 2 must apply from among the requirements indicated by the rule information, based on the evaluation-related document.
[0053] Here, the rules applicable to Evaluation Document 2 may differ depending on company information such as the number of employees and business activities. For example, the rules applicable in the QMS Ministerial Ordinance, etc., will differ depending on whether the business activities include manufacturing and sales, manufacturing only, or sales only, and the items and content that should be included in Evaluation Document 2 will also differ. Furthermore, the items and content that should be included in Evaluation Document 2 will also differ depending on the size of the company, such as the number of employees. In addition, the person responsible for quality control, the approval flow for various Evaluation Document 2s, and the reviewers will differ depending on the organizational structure, such as the company's organizational chart.
[0054] Furthermore, the rules applicable to Document 2 under evaluation differ depending on the item information, such as the item name, manufacturing site, storage conditions, sterilization method, and applicable standards. For example, the rules applicable to medical devices and pharmaceuticals are different. Medical devices are subject to the QMS Ordinance, while pharmaceuticals are subject to the GMP Ordinance. The rules applicable in the QMS Ordinance also differ between endoscopes and program medical devices, which are software-type medical devices. Specifically, since program medical devices do not have a manufacturing process, the requirements regarding physical storage and transportation conditions, and sterilization and cleaning processes required for endoscopes, etc., do not apply. In addition, the rules applicable in the QMS Ordinance and the items to be entered differ depending on whether or not there is a sterilization process.
[0055] The language processing unit 114 identifies inconsistencies between the rule information in the document to be evaluated and the consistency evaluation results output from the trained model 4.
[0056] The generation unit 103 executes a process to generate information about inconsistencies (hereinafter referred to as inconsistency-related information) and proposed improvements for the evaluation target document 2 for the inconsistencies, based on the consistency evaluation results. The generation unit 103 may generate the inconsistency-related information and proposed improvements using the trained model 4. Specifically, the generation unit 103 may have the trained model 4 generate and output the inconsistency-related information and proposed improvements. For example, the generation unit 103 may generate the following information as inconsistency-related information. "Section 3.4 of the Quality Manual states that risk assessment is mandatory when making design changes, but the Design Change Procedure Manual SOP-07 does not include a corresponding requirement."
[0057] The generation unit 103 may, for example, generate improvement proposals based on rule information and evaluation-related information to change the description of inconsistent sections to a description that conforms to the requirements of the rule information. For example, the generation unit 103 may generate an improvement proposal to add a risk assessment process to Section 5 of the design change procedure manual SOP-07 based on the quality manual. The generation unit 103 may then generate a message regarding the improvement proposal stating, "We recommend revising Section 5 of the design change procedure manual SOP-07 to add a risk assessment process and to meet the requirements of Section 3.4 of the quality manual."
[0058] In the following explanation, the process by which the evaluation unit 110 and the generation unit 103 evaluate the consistency of the document 2 to be evaluated with respect to the rule information, and generate information related to the inconsistent parts when inconsistencies are detected, will be referred to as the consistency evaluation process.
[0059] The display processing unit 104 executes a process to display the inconsistency-related information and improvement suggestions generated by the generation unit 103 on the display unit 16.
[0060] The evaluation results of consistency between the generated and displayed rule information of the evaluation target document 2, information related to inconsistencies, and improvement suggestions will be explained with reference to Figure 10. Figure 10 is a schematic diagram showing an example of the display of the evaluation results of consistency between the rule information of the evaluation target document 2, the product inspection procedure manual, information related to inconsistencies, and improvement suggestions.
[0061] Information related to inconsistencies may include, for example, the document inconsistent, the basis for the claim that the document's content is inconsistent, the risk classification of the inconsistency issue, the number of inconsistencies in each risk classification, and the regulatory consistency rate.
[0062] In the example shown in Figure 10, the document name display unit 70, the caution information display unit 71, the document display unit 72, the inconsistency information display unit 73, the risk classification display unit 74, and the list display unit 75 are displayed on the display unit 16.
[0063] The document name display unit 70 displays the title of the document to be evaluated 2. In the example shown in Figure 10, "Product Inspection Procedure Manual PQC-001" is displayed as the title of the document to be evaluated 2 in the document name display unit 70.
[0064] The warning information display unit 71 displays a message to alert the user about the consistency evaluation results. For example, if a serious inconsistency is found in the document 2 being evaluated, the warning information display unit 71 may display a message such as, "A serious problem has been found. Corrections are required to meet legal and regulatory requirements," as shown in Figure 10.
[0065] The document display unit 72 displays the content, creator, and creation date of the evaluation document 2, including the inconsistent document. As shown in Figure 10, the document display unit 72 highlights the inconsistent parts in the evaluation document 2. In the example shown in Figure 10, the inconsistent part is highlighted by displaying a different background color around the sentence, "3. Responsibility The quality control department is responsible for the implementation of this procedure. However, records of implementation are not required."
[0066] As shown in Figure 10, the inconsistency information display unit 73 displays the risks of the inconsistent areas (described later), the "problems" indicating the reasons for the inconsistencies, the "regulatory references" indicating the regulations that form the basis for the inconsistencies, as well as "recommendations" and "suggestions for improvement."
[0067] In the example shown in Figure 10, the message displayed as the reason for the inconsistency is, "'Records of implementation are not required' violates Article 9 (Record Management) of the QMS Ordinance." The relevant regulations include the rule information that underlies the inconsistency in the description of the inconsistent section, the articles and requirements within that rule information, etc. In addition, the message displayed as a suggestion for improvement is, "Please correct this to, 'The quality control department is responsible for the implementation and record management of this procedure.'"
[0068] The risk classification display unit 74 displays the risk classification results for all inconsistencies identified in the document under evaluation 2, as well as the regulatory compliance rate for the entire document under evaluation 2. The risk classification results include the risk of each inconsistency and the total number of inconsistencies for each risk classification. In the example shown in Figure 10, the risk of the inconsistencies is divided into three levels: high (critical), medium (warning), and low (minor). Each risk of an inconsistency is displayed in color according to the severity of the risk in the risk classification display unit 74 and the list display unit 75. In the example in Figure 10, inconsistencies with critical risk are displayed in red (the densest dots in Figure 10), inconsistencies that are not critical but require attention are displayed in yellow as a warning (the second densest dots in Figure 10), and inconsistencies with minor risk are displayed in blue (the sparsest dots in Figure 10).
[0069] When the generation unit 103 identifies multiple inconsistencies in the document under evaluation, it classifies the risks of all of these inconsistencies and performs an overall consistency evaluation of the entire document under evaluation 2 based on the classification results. Specifically, the generation unit 103 performs an overall consistency evaluation of the entire document under evaluation 2 by calculating the regulatory compliance rate.
[0070] The regulatory compliance rate is an indicator of the overall assessment of the consistency of the evaluated document 2 based on the risk classification results. A regulatory compliance rate of 100% means that the evaluated document 2 perfectly matches the rule information, while a lower regulatory compliance rate means that the evaluated document 2 does not match the rule information to the same degree. In the example shown in Figure 10, the regulatory compliance rate is 65%.
[0071] The regulatory compliance rate is calculated by weighting it according to the level of risk. For example, the regulatory compliance rate is calculated using a point deduction system, where -10 points are deducted for each non-conformity if the risk is significant, -5 points for a warning level risk, and -3 points for a minor risk. If the regulatory compliance rate falls below 0 points, it is displayed as 0 points.
[0072] The list display unit 75 displays the inconsistencies identified in the document under evaluation 2 and their associated risks as a list. Clicking on an inconsistency on the list display unit 75 displays detailed information such as information related to the clicked inconsistency and suggested improvements in the document display unit 72 and the inconsistency information display unit 73.
[0073] Figure 11 shows the review request screen 80, which allows the creator of document 2 to request a review of document 2 from a reviewer. A review refers to evaluating the validity of the content of document 2, which has undergone consistency evaluation processing.
[0074] The review request screen 80 consists of multiple input fields 81, a cancel button 82 to withdraw the review request, and an approval button 83 to proceed with the review request. For example, as shown in Figure 11, the input fields 81 include fields for entering the folder where the document to be reviewed (evaluation target document 2) is stored, the title of the document to be reviewed (evaluation target document 2), the date and time of the review request, the name of the requester, a password, and comments. The reception unit 101 accepts the review request by reading the information entered in each input field 81.
[0075] When the reception unit 101 receives a review request, the review support unit 105 automatically generates checkpoints for the document to be evaluated 2 based on the hierarchy to which the document to be evaluated 2 belongs, and displays the review reception unit 9 along with the checkpoints on the display unit 16.
[0076] Figure 12 is a schematic diagram showing an example of the display screen used by a reviewer to review document 2 under evaluation.
[0077] Checkpoints are pieces of information about inconsistencies that the reviewer of document 2 under evaluation should verify. Examples of checkpoints include sentences that were identified as inconsistent, the risks associated with the inconsistencies, a list of inconsistencies, and descriptions or records that lack consistency or validity.
[0078] The checkpoints are generated based on the hierarchy to which the document under evaluation 2 belongs. For example, if the document under evaluation 2 is a standard operating procedure, the reviewer checks for consistency between the standard operating procedure and its higher-level documents, such as the QMS ordinance or quality manual, and whether or not essential items are included. For this reason, the review support unit 105 may generate checkpoints such as sentences that were evaluated as inconsistent, the risks of inconsistent sections, and lists of inconsistent sections. Also, for example, if the document under evaluation 2 is a record document such as a quality record, the reviewer checks whether the record document conforms to its higher-level documents, such as the standard operating procedure, whether or not necessary record items are filled in, and whether or not there is a signature or date. For this reason, the review support unit 105 generates checkpoints such as sections that do not conform to the corresponding procedure, sections where signatures or dates are missing, and sections where there are deficiencies in the content of the description or record.
[0079] In the example shown in Figure 12, the document name display unit 70, creation information display unit 76, review status display unit 77, document display unit 72a, inconsistency information display unit 73a, risk classification display unit 74, list display unit 75a, review acceptance unit 9, etc., are displayed on the display unit 16.
[0080] As shown in Figure 12, the creation information display unit 76 displays the creator's name and creation date of the document to be evaluated 2. The review status display unit 77 displays the reviewer, the review deadline, and the status indicating whether the review is pending or in progress.
[0081] In the example shown in Figure 12, the document display unit 72a displays the content of the evaluation document 2, including the inconsistent document, as well as the department to which the creator of the evaluation document 2 belongs and the creation date. The document display unit 72a highlights the inconsistent parts in the evaluation document 2. In the example shown in Figure 12, the inconsistent parts are highlighted by displaying a different background color and layout around the sentence "5. Records Inspection results shall be recorded in the inspection record." Note that the overall comment input unit 94 of the review reception unit 9 is displayed on the document display unit 72a.
[0082] The inconsistency information display unit 73a displays the risk and content of the inconsistent location. When the user clicks "Details" displayed in the selection item 91, the inconsistency information display unit 73a displays information related to the inconsistent location, such as "Problem" which indicates the reason for the inconsistency, although this information is not shown in Figure 12.
[0083] The review reception unit 9 receives the validity judgment results from the reviewers regarding the consistency evaluation results from the evaluation unit 110. In the example shown in Figure 12, the review reception unit 9 displays the selection items 91, the review judgment result selection unit 92, the comment input unit 93, the overall comment input unit 94, and the review submission button 95.
[0084] When a user clicks "Review" in the selection item 91, the review reception unit 9 displays the review judgment result selection unit 92, as shown in Figure 12.
[0085] The review judgment result selection unit 92 allows the reviewer to select the validity evaluation result of the consistency evaluation result identified by the evaluation unit 110 as "agree," "disagree," or "under consideration." "Agree" is selected when the reviewer agrees with the evaluation unit 110's judgment that there is an inconsistent area that needs correction. "Disagree" is selected when the reviewer determines that the information displayed on the inconsistency information display unit 73a is incorrect and not an inconsistent area. "Under consideration" is selected when the reviewer determines that further confirmation is needed to determine the validity of the evaluation result displayed on the inconsistency information display unit 73a. The comment input unit 93 accepts comments on the inconsistent areas indicated by the evaluation unit 110. The reviewer can input the validity judgment result and comments on the consistency evaluation result displayed on the inconsistency information display unit 73a via the review reception unit 9.
[0086] The overall comment input section 94 accepts comments from reviewers regarding the entire document 2 under evaluation. The comments from the overall comment input section 94 are displayed on the document display section 72a. Once the reviewer has finished determining the validity of the consistency assessment results, pressing the review submission button 95 will send a notification to the review requester or the next reviewer.
[0087] The list display section 75a shows the inconsistencies and their risks identified in the document under evaluation 2, along with the reviewer's assessment of their validity. Below the list display section 75a, the progress of the review by the reviewer is displayed. The review progress indicates the number of inconsistencies in the document under evaluation 2 that have been assessed for validity by the reviewer out of all the inconsistencies pointed out.
[0088] Figure 13 shows the document list screen 30, which is a list of evaluation target documents 2 stored in the quality control document evaluation system 1. As shown in Figure 13, the document list screen 30 displays a task display unit 34 that displays a list of tasks such as event management and document management, a document information display unit 33 that displays the title of the stored document, the creation date, the status of approval by the reviewer, the version number of each document, and an icon for viewing each document, a keyword input unit 31 for entering keywords to search for documents, and a button 32 for registering a new document. By pressing the new document registration button 32 and selecting the document to be stored, the document is stored in the storage unit 13.
[0089] Figure 14 shows the search results for the evaluation target document 2 stored in the quality control document evaluation system 1. As shown in Figure 13, the document list screen 30 displays a search results display unit 35 that displays a list of searched documents, a keyword display unit 38 that displays the keywords used for the search, a first sort key 36 that sorts by the type of search target, and a second sort key 37 that sorts by approved / unapproved status. In the example shown in Figure 14, the search keyword is "safety," and as a result of searching for documents, safety-related documents are displayed in the search results display unit 35. In this embodiment, a trained model such as a large-scale language model searches for documents based on the keyword.
[0090] The machine learning unit 106 acquires and stores the consistency evaluation results between the rule information and the document 2 to be evaluated using the trained model 4, as well as the validity judgment results for the consistency evaluation results. The machine learning unit 106 then uses the pairs of consistency evaluation results as input data and validity judgment results as labels to further train the trained model 4. This improves the accuracy of the consistency evaluation of the document 2 to be evaluated between the rule information and the trained model 4.
[0091] Next, we will explain an example of the flow of the quality control document evaluation process performed by the quality control document evaluation system 1, referring to Figures 15 to 17.
[0092] Figure 15 is a flowchart showing the flow of the quality control document evaluation process. Figure 16 is a flowchart showing an example of the consistency evaluation process in the quality control document evaluation process. Figure 17 is a flowchart showing an example of the checkpoint process in the quality control document evaluation process. Database 3 stores evaluation-related information such as laws and regulations concerning the quality of medical devices and pharmaceuticals, such as the QMS Ministerial Ordinance, as well as rules such as internal company regulations, company information, and item information, all in vectorized form with a defined document hierarchy.
[0093] As shown in Figure 15, in step S1, the receiving unit 101 of the processor 10 receives the evaluation target document 2 input via the input unit 15.
[0094] In step S2, the evaluation unit 110 and the generation unit 103 perform a consistency evaluation process that evaluates the consistency between the document to be evaluated 2 received in step S1 and the rule information stored in the database 3 that is related to the document to be evaluated 2.
[0095] The details of the consistency evaluation process in step S2 will be explained with reference to Figure 16.
[0096] As shown in Figure 16, in step S21, the data processing unit 111 divides the text data in the document to be evaluated 2, which was received in step S1, into multiple semantic chunks.
[0097] In step S22, the data processing unit 111 vectorizes each semantic chunk that was divided in step S21.
[0098] In step S23, the hierarchy determination unit 112 determines which hierarchy of the document hierarchy in quality control the document vectorized in step S22 belongs to.
[0099] In step S24, the search processing unit 113 refers to the database 3 and extracts rule information and evaluation-related information related to the document 2 to be evaluated from the database 3. At this time, the search processing unit 113 may calculate the similarity between vectorized documents, sentences, and semantic chunks, and extract the rule information and evaluation-related information to be extracted from the database 3 based on the calculated similarity.
[0100] In step S25, the search processing unit 113 inputs the document to be evaluated 2, the rule information and evaluation-related information extracted in step S24, and a prompt to evaluate the consistency between the extracted rule information and the document to be evaluated 2 as input information to the trained model 4.
[0101] In step S26, the language processing unit 114 evaluates the consistency between the document to be evaluated 2 and the rule information extracted in step S24, based on the information output from the trained model 4 based on the input information in step S25. Based on the results of this consistency evaluation, the language processing unit 114 determines whether or not there are any inconsistencies between the rule information and the document to be evaluated 2.
[0102] In step S27, if the processor 10 determines in step S26 that there are no inconsistencies in the rule information of the document to be evaluated 2 (step S27; NO), it terminates the consistency evaluation process. If the processor 10 determines in step S26 that there are inconsistencies in the rule information of the document to be evaluated 2 (step S27; YES), it proceeds to step S28.
[0103] In step S28, the generation unit 103 performs a process to generate inconsistent information such as the inconsistent document, the basis for the claim that the document's contents are inconsistent, the risk classification of the problem in the inconsistent area, and the regulatory consistency rate. After that, the consistency evaluation process for the document to be evaluated 2 is completed.
[0104] As shown in Figure 15, in step S3, if the processor 10 determines, based on the consistency evaluation process in step S2, that there are no inconsistencies between the document to be evaluated 2 received in step S1 and the rule information (step S3; NO), it terminates the quality control document evaluation process. On the other hand, if the processor 10 determines that there are inconsistencies between the document to be evaluated 2 and the rule information (step S3; YES), it proceeds to step S4.
[0105] In step S4, the review support unit 105 performs checkpoint processing.
[0106] The details of the checkpoint process in step S4 will be explained with reference to Figure 17.
[0107] In Figure 17, in step S41, the review support unit 105 generates checkpoints that the reviewer should verify. For example, the review support unit 105 generates checkpoints such as the inconsistent document and the risk classification of the inconsistent location from the inconsistency-related information generated in step S28.
[0108] In step S42, the review support unit 105 displays the checkpoints generated in step S41, along with the review reception unit 9 which accepts input from the reviewer regarding the validity judgment results for inconsistent sections.
[0109] In step S43, the review support unit 105 obtains the validity judgment result entered by the reviewer via the review reception unit 9.
[0110] In step S44, the review support unit 105 stores the validity determination result obtained in step S43.
[0111] In step S5, the generation unit 103 generates proposed improvements to the evaluation document 2 for the inconsistent parts and displays them on the display unit 16.
[0112] In step S6, the machine learning unit 106 uses the pairs of inconsistencies detected in step S26 and the reviewer's judgment results for those inconsistencies, stored in step S44, as supervised data to further train the trained model 4. This additional training improves the accuracy of the trained model 4's evaluation of consistency between the rule information and the document 2 under evaluation. After that, the processor 10 terminates the quality control document evaluation process.
[0113] According to the embodiments described above, the following effects are achieved.
[0114] (1) The quality control document evaluation system 1 is a quality control document evaluation system 1 for evaluating quality control documents, and comprises: a storage processing unit 102 that stores rule information relating to quality control rules in a database 3; a reception unit 101 that receives input of an evaluation target document 2 which is a quality control document to be evaluated; an evaluation unit 110 that analyzes the semantic relationship between the evaluation target document 2 and the information in the database 3 and evaluates the consistency of the evaluation target document 2 with respect to the rule information using a trained model 4; a generation unit 103 that generates information about the inconsistent parts and improvement proposals for the evaluation target document 2 when inconsistent parts that do not match the rule information are identified in the evaluation target document 2 based on the consistency evaluation results; and a display processing unit 104 that displays the information about the inconsistent parts and improvement proposals.
[0115] For example, the Quality Management System (QMS) regulations and other rules for quality control of medical devices and other equipment specify various details. Therefore, in order to comply with the QMS regulations, it is necessary to create documents that meet the various requirements specified in the QMS regulations, which is time-consuming and costly.
[0116] According to this embodiment, the consistency between the created evaluation target document 2 and the rule information is evaluated by processing using the trained model 4, and improvement proposals are generated. Therefore, even for quality control rules that specify various matters in detail, the effort and cost of creating quality control documents that conform to them can be reduced. Furthermore, since the evaluation of the evaluation target document 2 using the trained model 4 uses information in the user-managed database 3, hallucination is prevented, and a more accurate evaluation of the consistency of the quality control document with respect to the rule information can be obtained. Thus, quality control documents that conform to quality control rules can be created efficiently.
[0117] (2) In the quality control document evaluation system 1 described in (1), the rule information is text data written in a document relating to quality control rules, and the evaluation unit 110 includes a hierarchy determination unit 112 that automatically determines which hierarchy of documents in quality control the document to be evaluated 2 belongs to, and evaluates the consistency of the document to be evaluated 2 with respect to the rule information by evaluating whether the document to be evaluated 2 matches the requirements of the rule information which is at a higher hierarchy than the document to be evaluated 2.
[0118] By determining the document hierarchy to which the evaluation target document 2 belongs in quality control, and then evaluating the consistency between it and rule information belonging to a higher hierarchy than the evaluation target document 2, it is possible to obtain a more reliable evaluation of the requirements that the evaluation target document 2 must comply with. Therefore, it is possible to create quality control documents that conform to the quality control rule information more efficiently and reliably.
[0119] (3) The quality control document evaluation system 1 described in (2) further includes a review support unit 105 that automatically generates checkpoints, which are information that the reviewer of the document to be evaluated should check, from among the information regarding inconsistencies, based on the hierarchy to which the document to be evaluated 2 belongs, and displays a review reception unit 9 that accepts input from the reviewer of the validity judgment result for the consistency evaluation result along with the generated checkpoints.
[0120] Since the reviewer's checkpoints among the information regarding inconsistencies are displayed along with the review reception unit 9, the reviewer can more efficiently determine the validity of the consistency evaluation results using the trained model 4.
[0121] The quality control document evaluation system 1 described in (4)(3) further includes a machine learning unit 106 that stores the validity judgment results received by the review reception unit 9 and uses the pairs of consistency evaluation results as input data and validity judgment results as labels as training data to further train the trained model 4.
[0122] This improves the accuracy of evaluating the consistency between the rule information and the document under evaluation (Document 2) using the trained model (Model 4).
[0123] (5) In the quality control document evaluation system 1 described in any of (1) to (4), the evaluation unit 110 divides the document into multiple semantic chunks, vectorizes each of the multiple semantic chunks, quantifies the semantic relationships by calculating the similarity between the vectorized multiple semantic chunks, and performs a process of interpreting the content of the document by natural language understanding using a large-scale language model as a trained model 4.
[0124] By calculating the similarity between vectorized semantic chunks and then performing natural language understanding using a large-scale language model, the content of the evaluation document 2 and the rule information can be grasped more accurately. Therefore, a more accurate evaluation of the consistency between the rule information and the evaluation document 2 can be obtained.
[0125] In the quality control document evaluation system 1 described in any of (6)(1) to (5), when the generation unit 103 identifies inconsistencies in the document to be evaluated 2, it classifies the risks of the inconsistencies identified in the document to be evaluated 2 and comprehensively evaluates the consistency of the entire document to be evaluated 2 with the rule information based on the classification results.
[0126] In addition to the risk of inconsistencies between the rule information in document 2 under evaluation and the overall rule information of document 2, taking these risks into account, users such as document creators and reviewers can more efficiently determine whether document 2 under evaluation is appropriate.
[0127] (7) In the quality control document evaluation system 1 described in any of (1) to (6), the rule information includes the QMS Ordinance for Medical Devices, ISO 13485, ISO 14971 and related guidelines.
[0128] This system enables the efficient creation of quality control documents that conform to the QMS Ordinance, ISO 13485, ISO 14971, and related guidelines.
[0129] (8) In the quality control document evaluation system 1 described in any of (1) to (7), the rule information includes the internal regulations of the organization to which the user belongs.
[0130] Since the rules information includes internal organizational regulations, it is possible to create evaluation document 2 that conforms not only to laws and ministerial ordinances concerning quality control of medical devices and pharmaceuticals, and standards of external organizations, but also to the regulations of one's own organization.
[0131] In the quality control document evaluation system 1 described in any of (9)(1) to (8), the memory processing unit 102 stores organizational information relating to the organization to which the user belongs in the database 3, and the evaluation unit 110 evaluates the consistency of the document to be evaluated 2 with respect to the rule information based on the organizational information.
[0132] In the quality control document evaluation system 1 described in any of (10)(1) to (9), the memory processing unit 102 stores item information relating to the items subject to quality control in the database 3, and the evaluation unit 110 evaluates the consistency of the document to be evaluated 2 with the rule information based on the item information.
[0133] Here, for example, the rules applicable under the QMS ministerial ordinance differ depending on company information such as the size and nature of the company's business, as well as product information, and the items and content that should be included in quality control documents will also differ. .
[0134] By adding organizational and item information to database 3, the requirements in the rule information for the document to be evaluated 2 are assessed, and improvement suggestions are displayed if inconsistencies are detected, enabling the creation of quality control documents more efficiently and with greater accuracy.
[0135] (11) The program is a program that causes a computer 18 to perform a process for evaluating a quality control document, the process including: a storage process that stores rule information relating to quality control rules in a database 3; a reception process that accepts input of a document to be evaluated, which is a quality control document to be evaluated, which is a document to be evaluated; an evaluation process that analyzes the semantic relationship between the document to be evaluated and the information in the database 3 and evaluates the consistency of the document to be evaluated with respect to the rule information using a trained model 4; a generation process that, if inconsistencies that do not match the rule information in the document to be evaluated are identified based on the consistency evaluation results, generates information about the inconsistencies and suggestions for improvement of the document to be evaluated; and a display process that displays the information about the inconsistencies and suggestions for improvement.
[0136] (12) A quality control document evaluation method is a quality control document evaluation method for evaluating a quality control document, comprising: a storage processing step of storing rule information relating to quality control rules in a database; a reception step of receiving input of an evaluation target document 2 which is a quality control document to be evaluated; an evaluation step of analyzing the semantic relationship between the evaluation target document 2 and the information in the database 3 and evaluating the consistency of the evaluation target document 2 with the rule information using a trained model 4; a generation step of generating information about the inconsistent parts and improvement proposals for the evaluation target document 2 if inconsistent parts that do not match the rule information are identified in the evaluation target document 2 based on the consistency evaluation results; and a display processing step of displaying the information about the inconsistent parts and improvement proposals.
[0137] Although embodiments of the present invention have been described above, the present invention is not limited to the above embodiments and can be modified as appropriate.
[0138] In the above embodiment, the search processing unit 113 extracted rule information and evaluation-related information related to the document to be evaluated 2 from the database 3. However, the trained model 4 may divide the document to be evaluated 2 into semantic chunks, vectorize them, directly refer to the database 3, and extract rule information and evaluation-related information related to the document to be evaluated 2 from the database 3.
[0139] In the above embodiment, the search processing unit 113 extracted evaluation-related information along with rule information related to the document to be evaluated 2 from the database 3, but it is also possible to extract only the rule information.
[0140] In the above embodiment, the quality control document evaluation system 1 generated improvement suggestions after performing checkpoint processing as shown in Figure 15. However, improvement suggestions may be generated after consistency evaluation processing and before checkpoint processing. [Explanation of Symbols]
[0141] 1. Quality Control Document Evaluation System 2. Documents to be evaluated 3 Databases 4. Pre-trained models 101 Reception Department 102 Memory Processing Unit 103 Generation part 104 Display Processing Unit 110 Evaluation Department
Claims
1. A quality control document evaluation system for evaluating quality control documents, A memory processing unit that stores rule information regarding quality control rules in a database, The reception area accepts input of the evaluation target documents, which are quality control documents subject to evaluation, An evaluation unit analyzes the semantic relationship between the document to be evaluated and the information in the database, and evaluates the consistency of the document to be evaluated with respect to the rule information using a trained model. If, based on the consistency evaluation results, inconsistencies are identified in the document under evaluation that do not conform to the rule information, a generation unit generates information about the inconsistencies and proposed improvements to the document under evaluation. A quality control document evaluation system comprising a display processing unit that displays information regarding the aforementioned inconsistencies and the suggested improvements.
2. The aforementioned rule information is information contained in a document relating to quality control rules. The evaluation unit, The system includes a hierarchy determination unit that automatically determines which hierarchy of documents in quality control the document being evaluated belongs to. A quality control document evaluation system according to claim 1, which evaluates the consistency of the document to be evaluated with respect to the rule information by evaluating whether the document to be evaluated conforms to the requirements of the rule information which is at a higher hierarchical level than the document to be evaluated.
3. The quality control document evaluation system according to claim 2, further comprising: a review support unit that automatically generates checkpoints, which are information that the reviewer of the document to be evaluated should check, from among the information regarding the inconsistent parts, based on the hierarchy to which the document to be evaluated belongs; and a review reception unit that displays a review reception unit that accepts input from the reviewer of the validity judgment result for the consistency evaluation result along with the generated checkpoints.
4. The quality control document evaluation system according to claim 3, further comprising a machine learning unit that stores the consistency evaluation results and the validity determination results received by the review reception unit, and uses the pairs of the consistency evaluation results as input data and the validity determination results as labels to further train the trained model as training data.
5. The quality control document evaluation system according to claim 1, wherein the evaluation unit divides the document into a plurality of semantic chunks, vectorizes each of the plurality of semantic chunks, quantifies the semantic relationship by calculating the similarity between the vectorized plurality of semantic chunks, and performs a process of interpreting the content of the document by natural language understanding using the large-scale language model as the trained model.
6. The quality control document evaluation system according to any one of claims 1, wherein the generation unit, upon identifying inconsistencies in the document to be evaluated, classifies the risks of the inconsistencies identified in the document to be evaluated, and comprehensively evaluates the consistency between the entire document to be evaluated and the rule information based on the classification results.
7. The aforementioned rule information includes the Ministerial Ordinance on QMS for Medical Devices, ISO 13485, ISO 14971 and related guidelines, as described in any one of claims 1 to 6.
8. The quality control document evaluation system according to claim 7, wherein the rule information includes internal organizational regulations of the organization to which the user belongs.
9. The memory processing unit stores organizational information relating to the organization to which the user belongs in the database. The quality control document evaluation system according to claim 1, wherein the evaluation unit evaluates the consistency of the document to be evaluated with respect to the rule information based on the organizational information.
10. The memory processing unit stores item information relating to the items subject to quality control in the database. The quality control document evaluation system according to claim 1, wherein the evaluation unit evaluates the consistency of the document to be evaluated with respect to the rule information based on the item information.
11. A program that causes a computer to perform a process for evaluating quality control documents, The aforementioned process is, A memory processing step that stores rule information regarding quality control rules in a database, The reception process involves receiving input for the evaluation target document, which is a quality control document to be evaluated. An evaluation step which involves analyzing the semantic relationship between the document to be evaluated and the information in the database, and evaluating the consistency of the document to be evaluated with respect to the rule information using a trained model, If, based on the consistency evaluation results, inconsistencies are identified in the document under evaluation that do not conform to the rule information, a generation process is performed to generate information about the inconsistencies and proposed improvements to the document under evaluation. A program including a display processing step that displays information regarding the inconsistent parts and the suggested improvements.
12. A method for evaluating quality control documents, A memory processing step that stores rule information regarding quality control rules in a database, The reception process involves receiving input for the evaluation target document, which is a quality control document to be evaluated. An evaluation step which involves analyzing the semantic relationship between the document to be evaluated and the information in the database, and evaluating the consistency of the document to be evaluated with respect to the rule information using a trained model, If, based on the consistency evaluation results, inconsistencies are identified in the document under evaluation that do not conform to the rule information, a generation process is performed to generate information about the inconsistencies and proposed improvements to the document under evaluation. A quality control document evaluation method comprising a display processing step that displays information regarding the aforementioned inconsistencies and the suggested improvements.
Citation Information
Patent Citations
Clinical test related file management system
JP2022095026A
