Quality control document evaluation system, program, and quality control document evaluation method

Abstract

To provide a quality control document evaluation system, program, and quality control document evaluation method that can efficiently create quality control documents that comply with quality control rules. [Solution] The quality control document evaluation system 1 comprises a storage processing unit 102 that stores rule information regarding quality control rules in a database 3, a reception unit 101 that receives input of an evaluation target document 2, which is a quality control document to be evaluated, an evaluation unit 110 that analyzes the semantic relevance between the evaluation target document 2 and the information in the database 3 and evaluates the consistency of the evaluation target document 2 with the rule information using a trained model 4, a generation unit 103 that, when an inconsistency in the evaluation target document 2 that is not consistent with the rule information is identified based on the consistency evaluation results, generates information regarding the inconsistency and an improvement proposal for the evaluation target document 2, and a display processing unit 104 that displays the information regarding the inconsistency and the improvement proposal.

Description

[Technical Field]

[0001] The present invention relates to a quality control document evaluation system, a program, and a quality control document evaluation method. [Background technology]

[0002] Conventionally, systems relating to document management and business management have been known. For example, Patent Document 1 describes a system for transferring clinical trial management files between pharmaceutical companies and medical institutions. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-95026 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the QMS Ministerial Ordinance, ISO, and other quality control rules for medical devices, pharmaceuticals, etc. specify a variety of detailed requirements. Therefore, creating and managing quality control documents related to quality control that conform to the QMS Ministerial Ordinance, ISO, and other rules requires creating documents that meet these various requirements, which is time-consuming.

[0005] An object of the present invention is to provide a quality control document evaluation system, program, and quality control document evaluation method that can efficiently create quality control documents that comply with quality control rules. [Means for solving the problem]

[0006] The quality control document evaluation system is a quality control document evaluation system for evaluating quality control documents, and includes a storage processing unit that stores rule information related to quality control rules in a database, a reception unit that receives input of an evaluation target document, which is a quality control document to be evaluated, an evaluation unit that analyzes the semantic relevance between the evaluation target document and information in the database and evaluates the consistency of the evaluation target document with the rule information using a trained model, a generation unit that, when an inconsistency in the evaluation target document that is not consistent with the rule information is identified based on the consistency evaluation result, generates information about the inconsistency and an improvement proposal for the evaluation target document, and a display processing unit that displays the information about the inconsistency and the improvement proposal. [Effects of the Invention]

[0007] According to the present invention, quality control documents that comply with quality control rules can be efficiently created. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a schematic diagram showing a quality control document evaluation system and an evaluation target document according to an embodiment of the present invention; [Figure 2] FIG. 2 is a schematic diagram showing modularized functions provided in a quality control system. [Figure 3] FIG. 1 is a block diagram showing a hardware configuration of a quality control document evaluation system. [Figure 4] FIG. 1 is a block diagram showing a functional configuration of a quality control document evaluation system. [Figure 5] FIG. 10 is a schematic diagram showing an example of a display screen for registering company information. [Figure 6] FIG. 10 is a schematic diagram showing an example of a display screen for registering item information. [Figure 7] FIG. 1 is a schematic diagram showing document hierarchy in quality control. [Figure 8] FIG. 1 is a schematic diagram showing an example of a higher-level document in quality control. [Figure 9] FIG. 10 is a schematic diagram showing an example of a sub-document in quality control. [Figure 10] 10 is a schematic diagram showing an example of a display of an evaluation result of consistency between rule information and a document to be evaluated by the quality control document evaluation system. FIG. [Figure 11] FIG. 10 is a schematic diagram illustrating an example of a display screen for requesting a reviewer to review a document to be evaluated. [Figure 12] 10 is a schematic diagram showing an example of a display screen for reviewing the results of an evaluation of consistency between rule information and a document to be evaluated by a quality control document evaluation system. FIG. [Figure 13] 10 is a schematic diagram illustrating an example of a display screen showing a list of quality control documents registered in a quality control system. FIG. [Figure 14] FIG. 10 is a schematic diagram showing an example of displaying quality control documents registered in a quality control system by keyword search; [Figure 15] 10 is a flowchart showing the flow of a quality control document evaluation process according to an embodiment of the present invention. [Figure 16] 10 is a flowchart showing the flow of a consistency evaluation process in the quality control document evaluation process according to an embodiment of the present invention. [Figure 17] 10 is a flowchart showing the flow of checkpoint processing in the quality control document evaluation processing according to one embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0009] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Fig. 1 is a schematic diagram showing a quality control document evaluation system 1 and an evaluation target document 2 according to one embodiment of the present invention.

[0010] The quality control document evaluation system 1 is a system included in a quality control system (not shown) that supports quality control for users.

[0011] The quality control system mainly includes a document management function for managing documents, an event management function for managing events, and an education and training function for educating and training employees. FIG. 2 is a schematic diagram showing the modularized functions of the quality control system 100. In this specification, the term "document" refers to an electronic document in which text information and image information have been digitized. One document refers to, for example, one digitized law, standard, manual, procedure manual, etc.

[0012] As shown in Figure 2, the quality control system is modularized into various functions, including basic functions 61, document and record management functions 62, AI (artificial intelligence) support functions 63, education and training functions 64, and event management functions 65. The basic functions 61 include, for example, user management, authority management, audit logs, notifications, and multilingual settings. The document and record management functions 62 include, for example, document storage, editing, version management, approval workflows, document difference display (displaying differences between documents), access restrictions for stored documents, and electronic signatures. The AI ​​support functions 63 include, for example, presentation of necessary documents and actions, automatic document draft generation, automatic document checking, automatic test question creation, and accumulated data analysis. The education and training functions 64 include, for example, test question creation and distribution, attendance record management, and respondent competency assessment. The event management functions 65 include template and process management functions, alert functions, and other functions. Because each function in the quality control system is modularized, it has a highly scalable functional design.

[0013] The quality control document evaluation system 1 is a system included in a quality control system, and is mainly equipped with functions included in the document and record management function 62 and the AI ​​support function 63. Specifically, the quality control document evaluation system 1 evaluates whether the created quality control document is a document that complies with rule information related to quality control rules, and performs quality control document evaluation processing to present information related to inconsistencies (hereinafter referred to as inconsistency-related information), etc.

[0014] A document that conforms to rule information is a document that is written in accordance with the requirements specified by the rule information. In other words, it is a document that conforms to the rule information. A quality control document is a document related to quality control. In the following explanation, the quality control document to be evaluated by the quality control document evaluation system 1 is referred to as the evaluation target document 2.

[0015] The quality control document evaluation system 1 evaluates the consistency between the rule information and the document 2 to be evaluated using a database 3 storing rule information and the like and a trained model 4, and presents the evaluation results and the like to the user. The consistency between the rule and the document 2 to be evaluated refers to the degree to which the description in the document 2 to be evaluated satisfies the requirements indicated by the rule information. In this embodiment, the quality control of medical devices and pharmaceuticals will be described as an example. The database 3 and the trained model 4 will be described later.

[0016] An example of the hardware configuration of the quality control document evaluation system 1 will be described with reference to Fig. 3. Fig. 3 is a block diagram showing the hardware configuration of the quality control document evaluation system 1.

[0017] 3, the quality control document evaluation system 1 includes a computer 18, a storage unit 13, a communication unit 14, an input unit 15, and a display unit 16. A bus 17 and the like connect these units together.

[0018] The computer 18 includes a processor 10 and a read-only memory (ROM) 11 and a random-access memory (RAM) 12 as main storage devices. The processor 10 may be a central processing unit (CPU), a microprocessing unit (MPU), a system on a chip (SoC), a digital signal processor (DSP), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a field-programmable gate array (FPGA). Alternatively, the processor 10 may be a combination of these. The processor 10 may also be a combination of these with a hardware accelerator. The processor 10 controls each component to implement various functions of the quality control document evaluation system 1 based on programs such as firmware, system software, and application software stored in the ROM 11, the RAM 12, or an auxiliary storage device that is part of the storage unit 13. Note that some or all of these programs may be incorporated into the circuitry of the processor 10.

[0019] The storage unit 13 is a storage area for various programs and various data for causing the hardware group to function as the quality control document evaluation system 1, and can be configured with a ROM, RAM, flash memory, a solid-state drive (SSD), a hard disk drive (HDD), or the like. Specifically, the storage unit 13 stores programs for causing the computer 18 to execute each function of this embodiment. The storage unit 13 may also include a database 3 for storing rule information, etc. The storage unit 13 may also store a trained model 4. In this embodiment, the database 3 and the trained model 4 are located outside the quality control system, such as on the cloud.

[0020] The communication unit 14 executes processing for the quality control document evaluation system 1 to communicate with other devices via a network. For example, the quality control document evaluation system 1 can connect to the database 3 via the communication unit 14 and refer to or acquire information stored in the database 3. For example, the quality control document evaluation system 1 can evaluate the consistency of the document 2 to be evaluated by communicating with the trained model 4 via the communication unit 14.

[0021] The input unit 15 is a user interface electrically connected to the computer 18. The input unit 15 is composed of buttons, a mouse, a keyboard, a display, etc. The display is composed of, for example, a liquid crystal display (LCD) or an organic electroluminescent (EL) display, and a touch panel that detects the position touched by the user is provided on the image display surface of the display. The user can input information by touching the image display surface of the display.

[0022] The display unit 16 is a user interface electrically connected to the computer 18. The display unit 16 is configured by a display. Images and various information transmitted from the memory unit 13, the communication unit 14, the input unit 15, the computer 18, etc. are displayed on the display unit 16.

[0023] Next, various functions realized by the hardware configuration of the quality control document evaluation system 1 will be described with reference to Fig. 4. As shown in Fig. 4, the quality control document evaluation system 1 includes a reception unit 101, a storage processing unit 102, an evaluation unit 110, a generation unit 103, a display processing unit 104, a review support unit 105, and a machine learning unit 106 as functional units operating on the processor 10.

[0024] The receiving unit 101 executes a process of receiving information input by a user. For example, the receiving unit 101 receives an input of a document 2 to be evaluated input by a user via the communication unit 14 or the input unit 15. Furthermore, for example, the receiving unit 101 executes a process of receiving rule information and evaluation-related information via the communication unit 14 or the input unit 15.

[0025] The rule information may be, for example, information written in a document related to quality control rules, or may be information expressed in a form other than a document. Examples of forms other than a document include images, audio, and video. The rule information in this embodiment is information written in a document.

[0026] Examples of rule information regarding the quality control of medical devices and pharmaceuticals include information contained in the QMS (Quality Management System) Ministerial Ordinance, GMP (Good Manufacturing Practice) Ministerial Ordinance, ISO (International Organization for Standardization) 13485, ISO 14971, other related guidelines, quality manuals, etc.

[0027] The rule information preferably includes information described in the organization's internal regulations (hereinafter referred to as "organizational regulations") related to quality control in the organization to which the user belongs, such as company internal regulations. This allows the user to check whether the document 2 to be evaluated complies with not only laws, ministerial ordinances, and external organization standards related to the quality control of pharmaceutical devices and drugs, but also the regulations of the user's own organization.

[0028] The evaluation-related information is information that affects the evaluation of the consistency between the rule information and the document to be evaluated 2, and examples thereof include organizational information such as company information, and item information. The organizational information is information about the organization to which the user of the quality control document evaluation system 1 belongs. The item information is information about items such as medical devices and pharmaceuticals that the user handles and that are the subject of the document to be evaluated 2.

[0029] The company information and item information as organizational information and how to input this information will be described with reference to Fig. 5 and Fig. 6. Fig. 5 is a schematic diagram showing an example of a company information registration screen 51 among the system registration screens 5 displayed on the display unit 16. Fig. 6 is a schematic diagram showing an example of an item information registration screen 52 among the system registration screens 5 displayed on the display unit 16.

[0030] The system registration screen 5 is a screen for registering information about the document 2 to be evaluated, including evaluation-related information, etc. The system registration screen 5 includes a company information registration screen 51 and an item information registration screen 52. As shown in FIGS. 5 and 6, an indicator bar 50 is displayed at the top of the system registration screen 5 to indicate whether the page displayed on the display unit 16 is the company information registration screen 51 or the item information registration screen 52.

[0031] The company information registration screen 51 is a screen for registering information about the organization to which the user belongs, and is composed of multiple input fields 53 and page forwarding buttons 54. For example, as shown in FIG. 5 , the input fields 53 include fields for inputting the company's name, representative name, address, telephone number, email address, website, year of establishment, number of employees, and business details. The reception unit 101 reads the information entered in each input field 53 and accepts the information. The company information may include information indicating the organizational structure, such as an organizational chart, business license information, manufacturing plant information, and certification information. The business license information is, for example, information indicating the type of business license obtained by the organization to which the user belongs. Examples of types of business licenses include a manufacturing license, a manufacturing and sales license, and a repair license. The manufacturing plant information is information about the manufacturing plant of the user's products. The certification information is, for example, information indicating the type of certification, such as ISO 13485, obtained by the organization to which the user belongs.

[0032] The item information registration screen 52 is an image for registering item information about the product that is the subject of the document to be evaluated 2, and is composed of multiple input fields 55, a page return button 56, and a registration button 57. For example, as shown in Figure 6, the input fields 55 include fields for inputting the product's item name, item code, general name classification, intended use or effect, manufacturing site, storage conditions, sterilization method, applicable standards, etc.

[0033] The storage processing unit 102 executes a process of storing the document 2 to be evaluated, the rule information, and the evaluation-related information acquired via the receiving unit 101. The storage processing unit 102 stores the document 2 to be evaluated in the storage unit 13, and stores the rule information and the evaluation-related information in the database 3. Note that the rule information and the evaluation-related information as a document are preferably divided into multiple semantic chunks by the data processing unit 111 (described later) and stored in the database 3 in a vectorized state.

[0034] The evaluation unit 110 executes a consistency evaluation process for evaluating the consistency between the rule information and the document 2 to be evaluated, based on the document 2 to be evaluated received by the reception unit 101 and the information in the database 3 .

[0035] In the consistency evaluation process, the semantic relevance between the document 2 to be evaluated and the information in the database 3 is analyzed, and the consistency of the document 2 to be evaluated with the rule information is evaluated based on the analysis results. Semantic relevance refers to the degree of semantic association between different information such as different documents, sentences, or words.

[0036] The evaluation unit 110 of this embodiment uses Retrieval Augmented Generation (RAG) to evaluate the consistency between the rule information and the document to be evaluated 2. Specifically, the evaluation unit 110 inputs the document to be evaluated 2 and information in the database 3 related to the document to be evaluated 2 into the trained model 4, and causes the trained model 4 to evaluate the degree of match of the document to be evaluated 2 with the rule information. Then, based on the evaluation results output from the trained model 4, the evaluation unit 110 evaluates the degree of match of the document to be evaluated 2 with the rule information.

[0037] The evaluation unit 110 includes a data processing unit 111 , a hierarchy determination unit 112 , a search processing unit 113 , and a language processing unit 114 .

[0038] The data processing unit 111 executes a process of converting documents such as the evaluation target document 2, rule information, evaluation-related documents, etc. into information that can be analyzed by a computer.

[0039] The data processing unit 111 divides text data such as the evaluation target document 2 received by the receiving unit 101 into multiple semantic chunks. For example, the data processing unit 111 divides the sentence "5.3 Final inspection: Inspect the appearance, dimensions, and functions of the product. The inspection rate will be determined separately," which is written in the product inspection procedure manual as the evaluation target document 2, into the following three chunks A to C. Chunk A: "5.3 Final Inspection" Chunk B: "Inspect the product's appearance, dimensions, and functionality." Chunk C: "Inspection rate will be determined separately."

[0040] The data processing unit 111 vectorizes each semantic chunk. For example, the data processing unit 111 vectorizes chunk B as follows: Chunk B: [0.12, -0.34, 0.56]

[0041] The rule information and evaluation-related information that have been divided into a plurality of semantic chunks and vectorized are transferred to the storage processing unit 102 and stored in the database 3.

[0042] The layer discrimination unit 112 executes a process of discriminating to which layer of document layers in quality control the vectorized document belongs.

[0043] Here, in the quality control document evaluation system 1 of this embodiment, documents related to the quality control of medical devices and medicines are divided into multiple hierarchies according to their type and managed. The document hierarchy in the quality control of medical devices and medicines (hereinafter referred to as document hierarchy) will be described with reference to Figs. 7 to 9. Fig. 7 is a schematic diagram showing the document hierarchy in the quality control of medical devices and medicines. Fig. 8 is a schematic diagram showing an example of a higher-level document in the quality control of medical devices and medicines. Fig. 9 is a schematic diagram showing an example of a lower-level document in the quality control of medical devices and medicines.

[0044] As shown in Figure 7, in this embodiment, the documents listed in the hierarchy, from the highest document (hereinafter referred to as the highest document) to the lowest document (hereinafter referred to as the lowest document), include documents such as "quality manuals" that describe basic concepts of quality, documents related to basic business rules for carrying out these basic concepts, such as "regulations," documents that describe the actual business operations carried out by each department, such as "procedure manuals," documents used in business operations, such as "plans, ledgers, and forms," ​​and records of actual business operations, such as "quality records." The contents of evaluation target document 2 must be higher in rank than evaluation target document 2 and must conform to the requirements of related documents.

[0045] The hierarchical level discrimination unit 112 discriminates the level to which a vectorized document belongs, for example, based on the title of the document. As shown in FIG. 8, the hierarchical level discrimination unit 112 classifies "quality manuals" and "regulations" as higher-level documents, and classifies various "procedures" such as standard operating procedures (SOPs) and work instructions, "plans, ledgers, and formats," and "quality records" as lower-level documents. Among the lower-level documents, for example, "quality records" are classified as documents lower than "procedures" because record items are entered in accordance with various "procedures" such as corresponding work instructions. In this embodiment, each rule information is stored in the database 3 with the document level determined in advance.

[0046] The hierarchy determination unit 112 of this embodiment determines the document hierarchy of the document 2 to be evaluated that has been vectorized by the data processing unit 111. As a result, the evaluation unit 110 can identify to which hierarchy the document 2 to be evaluated belongs, and can identify a higher-level document that contains rule information that the document 2 to be evaluated must comply with.

[0047] The search processing unit 113 executes a process of extracting rule information and evaluation-related information related to the evaluation target document 2 from the database 3. The search processing unit 113 may calculate the similarity between vectorized documents and extract information related to the evaluation target document 2 from the database 3 based on the calculated similarity. The similarity may be found, for example, by calculating the inner product of vectorized documents, sentences, or chunks. For example, the search processing unit 113 may calculate the similarity between the title of the vectorized evaluation target document 2 and the titles of the vectorized rule information and evaluation-related information, and extract information related to the evaluation target document 2 from the database 3 based on the calculated similarity. In this embodiment, the search processing unit 113 identifies documents that are ranked higher than the evaluation target document 2 from the rule information stored in the database 3, and extracts documents related to the evaluation target document 2 from the identified higher-ranking documents.

[0048] For example, the search processing unit 113 may extract a predetermined number of documents from among the documents having the highest similarity among the multiple documents serving as rule information in the database 3. The predetermined number may be, for example, 10 or more, or may be less than 10. The search processing unit 113 may extract an entire document related to the evaluation target document 2, or may extract only a portion of a document related to the evaluation target document 2. For example, if the rule information related to the evaluation target document 2 is a QMS Ministerial Ordinance, the search processing unit 113 may extract the entire QMS Ministerial Ordinance, or may extract only the provisions of the QMS Ministerial Ordinance related to the evaluation target document 2, or may extract only the provisions of the QMS Ministerial Ordinance that state requirements for the evaluation target document 2. For example, the search processing unit 113 may extract only the provisions that state requirements for the evaluation target document 2 as follows: "QMS Ministerial Ordinance Article 44" "Document the standards, methods and results of product inspections"

[0049] The search processing unit 113 inputs the document 2 to be evaluated, the extracted rule information and evaluation-related information, and a prompt to evaluate the consistency between the extracted rule information and the document 2 to be evaluated as input information to the trained model 4.

[0050] The language processing unit 114 evaluates the consistency between the rule information and the document 2 to be evaluated based on the response from the trained model 4 that is output when input information is input to the trained model 4.

[0051] The trained model 4 of this embodiment is a large-scale language model that uses natural language understanding to interpret document content, including the context and nuances of the evaluation target document 2, rule information, and evaluation-related documents. By interpreting the document content, the trained model 4 identifies requirements indicated by the rule information that must be applied to the evaluation target document 2. The trained model 4 then determines whether the evaluation target document 2 conforms to the identified requirements, and outputs the determination result as an evaluation result of the consistency of the evaluation target document 2.

[0052] Since the evaluation-related information is input to the trained model 4 together with the rule information, the trained model 4 can identify, based on the evaluation-related information, the requirements indicated by the rule information that need to be applied to the document to be evaluated 2, and their contents. That is, the language processing unit 114 identifies, based on the evaluation-related document, the requirements indicated by the rule information that need to be applied to the document to be evaluated 2.

[0053] Here, the rules applied to Document 2 to be assessed may differ depending on company information such as the number of employees and business content. For example, if a company's business involves manufacturing and sales, if it only involves manufacturing, or if it only involves sales, the rules applied under the QMS Ministerial Ordinance, etc. will differ, and the items and content that should be included in Document 2 to be assessed will also differ. The items and content that should be included in Document 2 to be assessed will also differ depending on the company's size, such as the number of employees. Furthermore, the person responsible for quality control, the approval flow for various Documents 2 to be assessed, and reviewers, etc., will differ depending on the company's organizational structure, such as the organizational chart, etc.

[0054] Furthermore, the rules applied to assessment target document 2 vary depending on the item information, such as the item name, manufacturing site, storage conditions, sterilization method, and applicable standards. For example, different rules apply to medical devices and pharmaceuticals. Medical devices are subject to the QMS Ministerial Ordinance, while pharmaceuticals are subject to the GMP Ministerial Ordinance. The rules applied in the QMS Ministerial Ordinance also differ between endoscopes and programmed medical devices, which are software-based medical devices. Specifically, programmed medical devices do not have a manufacturing process, so requirements regarding physical storage and transportation conditions, sterilization and cleaning processes, etc., that are required for endoscopes, do not apply. Furthermore, the rules applied and the items that must be entered in the QMS Ministerial Ordinance differ depending on whether or not a sterilization process is involved.

[0055] The language processing unit 114 identifies inconsistencies between the rule information in the document 2 to be evaluated based on the consistency evaluation result output from the trained model 4.

[0056] The generation unit 103 executes a process of generating information about the inconsistency part (hereinafter referred to as inconsistency part related information) and an improvement plan for the inconsistency part of the document 2 to be evaluated, based on the consistency evaluation result. The generation unit 103 may generate the inconsistency part related information and the improvement plan using the trained model 4. Specifically, the generation unit 103 may cause the trained model 4 to generate and output the inconsistency part related information and the improvement plan. For example, the generation unit 103 may generate the following information as the inconsistency part related information. "Section 3.4 of the Quality Manual requires risk assessment when making design changes, but the Design Change Procedure Manual SOP-07 does not include any corresponding requirements."

[0057] The generating unit 103 may generate an improvement proposal for changing the description of the inconsistent part to a description that conforms to the requirements of the rule information, for example, based on the rule information and the evaluation-related information. For example, the generating unit 103 may generate an improvement proposal for adding a risk assessment process to Section 5 of the design change procedure manual SOP-07 based on the quality manual. Then, the generating unit 103 may generate a message about the improvement proposal stating, "We recommend that you add a risk assessment process to Section 5 of the design change procedure manual SOP-07 and revise it so that it satisfies the requirements of Section 3.4 of the quality manual."

[0058] In the following explanation, the process in which the evaluation unit 110 and the generation unit 103 evaluate the consistency of the document 2 to be evaluated with the rule information, and in which inconsistencies are detected and information related to the inconsistencies is generated, is referred to as the consistency evaluation process.

[0059] The display processing unit 104 executes a process of displaying the mismatch-related information and improvement plan generated by the generating unit 103 on the display unit 16 .

[0060] The generated and displayed evaluation results of consistency with the rule information of the evaluation target document 2, information related to inconsistencies, and improvement proposals will be described with reference to Fig. 10. Fig. 10 is a schematic diagram showing a display example of the evaluation results of consistency with the rule information of the product inspection procedure manual, which is the evaluation target document 2, information related to inconsistencies, improvement proposals, etc.

[0061] Examples of information related to inconsistencies include the document that is inconsistent, the basis information for claiming that the document description is inconsistent, the risk classification of the problem with the inconsistent part, the number of inconsistencies in each risk classification, and the regulatory compliance rate.

[0062] In the example shown in FIG. 10, a document name display section 70, a warning information display section 71, a document display section 72, an inconsistency information display section 73, a risk classification display section 74, and a list display section 75 are displayed on the display section 16.

[0063] The document name display section 70 displays the title of the document 2 to be evaluated. In the example shown in Fig. 10, "Product Inspection Procedure Manual PQC-001" is displayed in the document name display section 70 as the title of the document 2 to be evaluated.

[0064] The warning information display unit 71 displays a message to alert the user to the consistency evaluation result. For example, when a highly serious inconsistency is found in the document 2 to be evaluated, the warning information display unit 71 may display a message such as "A serious problem has been found. Corrections are required to satisfy legal requirements," as shown in Fig. 10.

[0065] The document display unit 72 displays the content, creator, and creation date of the document 2 to be evaluated, including the inconsistent document. As shown in Fig. 10, the document display unit 72 highlights the inconsistent parts in the document 2 to be evaluated. In the example shown in Fig. 10, the inconsistent parts are highlighted by displaying the background around the sentence "3. Responsibility The quality control department is responsible for implementing this procedure. However, it is not necessary to record the implementation." in a different color.

[0066] As shown in Figure 10, the inconsistency information display section 73 displays the risk of inconsistencies, which will be described later, "Problems" indicating the reasons for the inconsistencies, "Regulatory References" indicating the regulations that are the basis for the inconsistencies, "Recommendations," and "Improvement Suggestions."

[0067] In the example shown in Figure 10, the message displayed as the reason for the inconsistency is, "'Records of implementation are not required' violates Article 9 (Record Management) of the QMS Ministerial Ordinance." The underlying regulations include rule information that is the basis for the inconsistency in the description of the inconsistent part, and items and requirements such as clauses in the rule information. In addition, the message displayed as an improvement suggestion is, "Please revise this to 'The quality control department is responsible for implementing this procedure and managing records.'"

[0068] The risk classification display unit 74 displays the risk classification results for all inconsistencies identified in the document 2 to be evaluated and the regulatory consistency rate for the entire document 2 to be evaluated. The risk classification results include the risk of each inconsistency and the total number of inconsistencies for each risk classification. In the example shown in FIG. 10, the risk of an inconsistency is divided into three levels: high (serious), medium (warning), and low (minor). The risks of each inconsistency are displayed in different colors in the risk classification display unit 74 and list display unit 75 according to the severity of the risk. In the example shown in FIG. 10, inconsistencies with a serious risk are displayed in red (the densest dots in FIG. 10), inconsistencies that are not serious but require attention are displayed as a warning in yellow (the second densest dots in FIG. 10), and inconsistencies with a minor risk are displayed in blue (the sparsest dots in FIG. 10).

[0069] When the generation unit 103 identifies multiple inconsistencies in the document to be evaluated, it classifies the risks of all of the multiple inconsistencies and, based on the classification results, performs a comprehensive evaluation of the consistency of the entire document to be evaluated 2. Specifically, the generation unit 103 performs a comprehensive evaluation of the consistency of the entire document to be evaluated 2 by calculating a regulatory compliance rate.

[0070] The regulatory compliance rate is an index of the overall evaluation of the consistency of the document 2 to be evaluated based on the results of risk classification. A regulatory compliance rate of 100% means that the document 2 to be evaluated is in perfect conformance with the rule information, and the lower the regulatory compliance rate, the less the document 2 to be evaluated conforms to the rule information. In the example shown in Figure 10, the regulatory compliance rate is 65%.

[0071] The regulatory compliance rate is calculated by weighting according to risk. For example, the regulatory compliance rate is calculated using a deductible system, with -10 points for each non-compliance if the risk is serious, -5 points for a warning level risk, and -3 points for a minor risk. If the regulatory compliance rate is below 0 points, it will be displayed as 0 points.

[0072] The list display section 75 displays a list of inconsistencies and the associated risks identified in the document to be evaluated 2. By clicking on an inconsistency on the list display section 75, detailed information such as inconsistency-related information and improvement plans for the clicked inconsistency is displayed in the document display section 72 and the inconsistency information display section 73.

[0073] 11 shows a review request screen 80 for the creator of the document 2 to request a reviewer to review the document 2. A review refers to evaluating the validity of the content of the document 2 to be evaluated after the consistency evaluation process has been performed.

[0074] The review request screen 80 is composed of a plurality of input fields 81, a cancel button 82 for canceling the review request, and an approve button 83 for proceeding with the review request. For example, as shown in Fig. 11, the input fields 81 include fields for inputting the folder in which the evaluation target document 2 to be reviewed is stored, the title of the evaluation target document 2, the review request date and time, the requester's name, a password, and a comment. The reception unit 101 receives the review request by reading the information entered in each input field 81.

[0075] When the reception unit 101 receives a review request, the review support unit 105 automatically generates checkpoints for the document 2 to be evaluated based on the layer to which the document 2 to be evaluated belongs, and displays the review reception unit 9 on the display unit 16 together with the checkpoints.

[0076] FIG. 12 is a schematic diagram showing an example of a display screen on which a reviewer reviews the document 2 to be evaluated.

[0077] A checkpoint is information about an inconsistency that should be checked by a reviewer of the document to be evaluated 2. Examples of checkpoints include sentences that are evaluated as inconsistent, the risk of an inconsistency, a list of inconsistencies, and descriptions or records that are inconsistent, inappropriate, or incomplete.

[0078] Checkpoints are generated based on the hierarchy to which the document 2 to be evaluated belongs. For example, if the document 2 to be evaluated is a standard operating procedure, the reviewer will check the consistency between the standard operating procedure and its higher-level documents, such as the QMS Ministerial Ordinance or Quality Manual, and whether or not required items are present. For this reason, the review support unit 105 may generate, as checkpoints, statements evaluated as inconsistent, the risk of inconsistencies, and a list of inconsistencies. For example, if the document 2 to be evaluated is a record document such as a quality record, the reviewer will check whether the record document conforms to various higher-level documents, such as the standard operating procedure, whether required record items are filled in, whether signatures and dates are present, etc. For this reason, the review support unit 105 generates, as checkpoints, areas that do not conform to the corresponding procedure document, areas where signatures or dates are not entered, areas where there are deficiencies in the description or record content, etc.

[0079] In the example shown in Figure 12, a document name display section 70, a creation information display section 76, a review status display section 77, a document display section 72a, an inconsistency information display section 73a, a risk classification display section 74, a list display section 75a, a review reception section 9, etc. are displayed on the display section 16.

[0080] 12, the creation information display unit 76 displays the creator name and creation date of the evaluation target document 2. The review status display unit 77 displays the reviewer, the review deadline, and the status indicating whether the review is pending or ongoing.

[0081] In the example shown in Figure 12, the document display unit 72a displays the contents of the document 2 to be evaluated, including the inconsistent document, as well as the department to which the creator of the document 2 to be evaluated belongs and the creation date. The document display unit 72a highlights the inconsistent parts in the document 2 to be evaluated. In the example shown in Figure 12, the inconsistent parts are highlighted by displaying the background around the sentence "5. Recording: Record the inspection results in the inspection record." in a different color or layout. The document display unit 72a also displays the overall comment input section 94 of the review receiving unit 9.

[0082] The inconsistency information display section 73a displays the risk and details of the inconsistency. When the user clicks on "Details" displayed in the selection item 91, the inconsistency information display section 73a displays information related to the inconsistency, such as "Problems" that indicate the reason for the inconsistency, although this is not shown in FIG.

[0083] The review receiving unit 9 receives the result of the reviewer's judgment of the validity of the consistency evaluation result by the evaluation unit 110. In the example shown in FIG. 12 , the review receiving unit 9 displays selection items 91, a review judgment result selection unit 92, a comment input unit 93, an overall comment input unit 94, and a review submission button 95.

[0084] When the user clicks on "Review" displayed in the selection item 91, the review receiving unit 9 displays a review judgment result selection unit 92 as shown in FIG.

[0085] The review judgment result selection unit 92 can select the reviewer's evaluation result of the validity of the consistency evaluation result identified by the evaluation unit 110 from "agree," "reject," or "under consideration." "Agree" is selected when the reviewer agrees with the evaluation result by the evaluation unit 110 that the inconsistency is a point that needs to be corrected. "Reject" is selected when the reviewer has determined that the information displayed in the inconsistency information display unit 73a is incorrect and not a point of inconsistency. "Under consideration" is selected when the reviewer has determined that further confirmation is necessary to determine the validity of the evaluation result displayed in the inconsistency information display unit 73a. The comment input unit 93 accepts input of comments regarding the inconsistency indicated by the evaluation unit 110. The reviewer can input the validity judgment result and comments regarding the consistency evaluation result displayed in the inconsistency information display unit 73a via the review acceptance unit 9.

[0086] The overall comment input section 94 accepts the reviewer's comments on the entire document to be evaluated 2. The overall comment input section 94 is displayed in the document display section 72a. When the reviewer presses the review submission button 95 after completing the judgment of the validity of the consistency evaluation result, a notification is sent to the review requester or the next reviewer.

[0087] The list display section 75a displays the inconsistencies and their risks identified in the document 2 to be evaluated, as well as the results of the reviewer's assessment of the validity of each inconsistency. The progress of the review by the reviewer is also displayed below the list display section 75a. The review progress indicates the number of inconsistencies that have been judged valid by the reviewer out of all inconsistencies pointed out in the document 2 to be evaluated.

[0088] 13 is a diagram showing a document list screen 30 that lists documents 2 to be evaluated that are stored in the quality control document evaluation system 1. As shown in Fig. 13, the document list screen 30 displays a task display section 34 that displays a list of tasks such as event management and document management, a document information display section 33 that displays the titles of stored documents, creation dates, the status of approval by reviewers, the version numbers of each document, and icons for viewing each document, a keyword input section 31 for inputting keywords for searching for documents, a button 32 for registering a new document, etc. By pressing the button 32 for registering a new document and selecting a document to store, the document is stored in the memory section 13.

[0089] FIG. 14 shows the search results for evaluation target documents 2 stored in the quality control document evaluation system 1. As shown in FIG. 13, the document list screen 30 displays a search result display section 35 that displays a list of searched documents, a keyword display section 38 that displays keywords used in the search, a first sort key 36 that sorts the types of search targets, and a second sort key 37 that sorts by approval / unapproval status. In the example shown in FIG. 14, the search keyword is "safety," and as a result of document search, documents related to safety are displayed in the search result display section 35. In this embodiment, a trained model such as a large-scale language model searches for documents based on keywords.

[0090] The machine learning unit 106 acquires and stores the evaluation results of the consistency between the rule information and the document 2 to be evaluated using the trained model 4, and the validity judgment results for the consistency evaluation results. Then, the machine learning unit 106 additionally trains the trained model 4 using the pair of the evaluation results of the consistency as input data and the judgment results of the validity as labels as training data. This makes it possible to improve the evaluation accuracy of the consistency of the document 2 to be evaluated with the rule information using the trained model 4.

[0091] Next, an example of the flow of the quality control document evaluation process executed by the quality control document evaluation system 1 will be described with reference to FIGS.

[0092] Fig. 15 is a flowchart showing the flow of quality control document evaluation processing. Fig. 16 is a flowchart showing an example of the flow of consistency evaluation processing in quality control document evaluation processing. Fig. 17 is a flowchart showing an example of the flow of checkpoint processing in quality control document evaluation processing. Note that database 3 stores rule information such as laws and regulations, ministerial ordinances, rules, and internal company rules related to the quality of medical devices and pharmaceuticals, such as the QMS Ministerial Ordinance, as well as evaluation-related information such as company information and item information, which are vectorized and stored with a defined document hierarchy.

[0093] As shown in FIG. 15, in step S1, the receiving unit 101 of the processor 10 receives the document 2 to be evaluated input via the input unit 15.

[0094] In step S2, the evaluation unit 110 and the generation unit 103 perform a consistency evaluation process to evaluate the consistency between the document 2 to be evaluated received in step S1 and the rule information stored in the database 3 and related to the document 2 to be evaluated.

[0095] The consistency evaluation process in step S2 will be described in detail with reference to FIG.

[0096] As shown in FIG. 16, in step S21, the data processing unit 111 divides the text data in the document 2 to be evaluated, which has been received in step S1, into a plurality of semantic chunks.

[0097] In step S22, the data processing unit 111 vectorizes each of the semantic chunks divided in step S21.

[0098] In step S23, the layer discrimination unit 112 discriminates to which layer of document layers in quality control the document vectorized in step S22 belongs.

[0099] In step S24, the search processing unit 113 refers to the database 3 and extracts rule information and evaluation-related information related to the document 2 to be evaluated from the database 3. At this time, the search processing unit 113 may calculate the similarity between vectorized documents, between sentences, or between semantic chunks, and extract the rule information and evaluation-related information to be extracted from the database 3 based on the calculated similarity.

[0100] In step S25, the search processing unit 113 inputs the document 2 to be evaluated, the rule information and evaluation-related information extracted in step S24, and a prompt to evaluate the consistency between the extracted rule information and the document 2 to be evaluated as input information to the trained model 4.

[0101] In step S26, the language processing unit 114 evaluates the consistency between the document to be evaluated 2 and the rule information extracted in step S24, based on the information output from the trained model 4 based on the input information in step S25. Based on the evaluation result of this consistency, the language processing unit 114 determines whether there is any inconsistency between the rule information and the document to be evaluated 2.

[0102] In step S27, if it is determined in step S26 that the document 2 to be evaluated does not contain any inconsistencies with respect to the rule information (step S27; NO), the processor 10 ends the consistency evaluation process. If it is determined in step S26 that the document 2 to be evaluated contains any inconsistencies with respect to the rule information (step S27; YES), the processor 10 proceeds to step S28.

[0103] In step S28, the generation unit 103 executes a process of generating information related to the inconsistent part, such as the document that has become inconsistent, the basis information for determining that the document description is inconsistent, the risk classification of the problem of the inconsistent part, the regulatory consistency rate, etc. After that, the consistency evaluation process for the document 2 to be evaluated ends.

[0104] 15, in step S3, if the processor 10 determines, as a result of the consistency evaluation process in step S2, that there is no inconsistency between the document 2 to be evaluated received in step S1 and the rule information (step S3; NO), the processor 10 terminates the quality control document evaluation process. On the other hand, if the processor 10 determines that there is an inconsistency between the document 2 to be evaluated and the rule information (step S3; YES), the processor 10 proceeds to step S4.

[0105] In step S4, the review support unit 105 executes a checkpoint process.

[0106] The checkpoint process in step S4 will be described in detail with reference to FIG.

[0107] 17, in step S41, the review support unit 105 generates checkpoints to be checked by the reviewer. For example, the review support unit 105 generates, as checkpoints, documents that have become inconsistent among the inconsistency-related information generated in step S28, risk classifications of the inconsistencies, and the like.

[0108] In step S42, the review support unit 105 displays the checkpoints generated in step S41 and the review reception unit 9 that receives input from the reviewer of the result of the judgment on the validity of the inconsistent portion.

[0109] In step S43, the review support unit 105 acquires the result of the judgment on validity input by the reviewer via the review receiving unit 9.

[0110] In step S44, the review support unit 105 stores the result of the determination of validity acquired in step S43.

[0111] In step S5, the generating unit 103 generates an improvement plan for the document 2 to be evaluated for the inconsistent portion, and displays it on the display unit 16.

[0112] In step S6, the machine learning unit 106 additionally trains the trained model 4 as supervised data using a pair of the inconsistency detected in step S26 and the reviewer's judgment result on the appropriateness of the inconsistency stored in step S44. This additional training improves the accuracy of the trained model 4's evaluation of the consistency between the rule information and the document 2 to be evaluated. Thereafter, the processor 10 terminates the quality control document evaluation process.

[0113] According to the embodiment described above, the following effects are achieved.

[0114] (1) The quality control document evaluation system 1 is a quality control document evaluation system 1 for evaluating quality control documents, and includes a storage processing unit 102 that stores rule information regarding quality control rules in a database 3, a reception unit 101 that receives input of an evaluation target document 2, which is a quality control document to be evaluated, an evaluation unit 110 that analyzes the semantic relevance between the evaluation target document 2 and the information in the database 3 and evaluates the consistency of the evaluation target document 2 with the rule information using a trained model 4, a generation unit 103 that, when an inconsistency that is not consistent with the rule information is identified in the evaluation target document 2 based on the consistency evaluation result, generates information about the inconsistency and an improvement proposal for the evaluation target document 2, and a display processing unit 104 that displays the information about the inconsistency and the improvement proposal.

[0115] For example, the QMS Ministerial Ordinance and other rules for quality control of medical devices and other products contain detailed stipulations on various matters. Therefore, in order to comply with the QMS Ministerial Ordinance, it is necessary to create documents that meet the various requirements detailed in the QMS Ministerial Ordinance, which is time-consuming and costly.

[0116] According to this embodiment, the consistency between the created evaluation target document 2 and the rule information is evaluated by processing using the trained model 4, and improvement proposals, etc. are generated, thereby reducing the effort and cost required to create a quality control document that conforms to quality control rules that specify a variety of details. Furthermore, since information in a database 3 that can be managed by the user is used to evaluate the evaluation target document 2 using the trained model 4, hallucination can be prevented and a more accurate evaluation of the consistency of the quality control document with the rule information can be obtained. Therefore, quality control documents that conform to quality control rules can be created efficiently.

[0117] (2) In the quality control document evaluation system 1 described in (1), the rule information is text data written in a document related to quality control rules, and the evaluation unit 110 is equipped with a hierarchy determination unit 112 that automatically determines to which hierarchy of documents in quality control the document 2 to be evaluated belongs, and evaluates the consistency of the document 2 to be evaluated with the rule information by evaluating whether the document 2 to be evaluated meets the requirements of the rule information that is hierarchically higher than the document 2 to be evaluated.

[0118] After determining the document hierarchy to which the document 2 to be evaluated in quality control belongs, the system evaluates the consistency with rule information belonging to a higher hierarchy than the document 2 to be evaluated, thereby more reliably obtaining an evaluation of the requirements that the document 2 to be evaluated must comply with. This makes it possible to create quality control documents that comply with the quality control rule information efficiently and more reliably.

[0119] (3) The quality control document evaluation system 1 described in (2) further includes a review support unit 105 that automatically generates checkpoints, which are information regarding inconsistencies that should be confirmed by the reviewer of the document 2 to be evaluated, based on the hierarchy to which the document 2 to be evaluated belongs, and displays the generated checkpoints along with a review reception unit 9 that accepts input from the reviewer of the validity of the consistency evaluation results.

[0120] The reviewer's checkpoints among the information regarding the inconsistencies are displayed together with the review reception unit 9, so that the reviewer can more efficiently determine the validity of the consistency evaluation results using the trained model 4.

[0121] (4) The quality control document evaluation system 1 described in (3) further includes a machine learning unit 106 that accumulates the validity judgment results received by the review receiving unit 9 and additionally trains the trained model 4 using pairs of the consistency evaluation results as input data and the validity judgment results as labels as training data.

[0122] The accuracy of evaluation of the consistency between the document to be evaluated 2 and the rule information using the trained model 4 can be improved.

[0123] (5) In the quality control document evaluation system 1 described in any one of (1) to (4), the evaluation unit 110 divides the document into multiple semantic chunks, vectorizes each of the multiple semantic chunks, quantifies the semantic relevance by calculating the similarity between the multiple vectorized semantic chunks, and performs a process of interpreting the content of the document through natural language understanding using a large-scale language model as a trained model 4.

[0124] After calculating the similarity between vectorized semantic chunks, natural language understanding is performed using a large-scale language model, which allows for a more accurate understanding of the contents of the document 2 to be evaluated and the rule information. This makes it possible to obtain a more accurate evaluation of the consistency between the rule information and the document 2 to be evaluated.

[0125] (6) In the quality control document evaluation system 1 described in any one of (1) to (5), when the generation unit 103 identifies an inconsistency in the document 2 to be evaluated, it classifies the risk of the inconsistency identified in the document 2 to be evaluated, and based on the classification result, performs a comprehensive evaluation of the consistency between the rule information of the entire document 2 to be evaluated.

[0126] In addition to the risk of inconsistencies with the rule information in the document 2 to be evaluated, it is also possible to grasp the consistency with the rule information of the entire document 2 to be evaluated taking into account that risk classification, so users such as document creators and reviewers can more efficiently grasp the appropriateness of the document 2 to be evaluated.

[0127] (7) In the quality control document evaluation system 1 described in any one of (1) to (6), the rule information includes the QMS Ministerial Ordinance for Medical Devices, ISO13485, ISO14971, and related guidelines.

[0128] It enables efficient creation of quality control documents that comply with QMS Ministerial Ordinance, ISO13485, ISO14971 and related guidelines.

[0129] (8) In the quality control document evaluation system 1 described in any one of (1) to (7), the rule information includes an internal rule of the organization to which the user belongs.

[0130] Since the rule information also includes internal organizational regulations, it is possible to create an evaluation target document 2 that complies with not only laws, ministerial ordinances, and external organization standards related to the quality control of pharmaceutical devices and drugs, but also the regulations of one's own organization.

[0131] (9) In the quality control document evaluation system 1 described in any one of (1) to (8), the memory processing unit 102 stores organizational information regarding the organization to which the user belongs in the database 3, and the evaluation unit 110 evaluates the consistency of the document 2 to be evaluated with the rule information based on the organizational information.

[0132] (10) In the quality control document evaluation system 1 described in any one of (1) to (9), the storage processing unit 102 stores item information regarding items subject to quality control in the database 3, and the evaluation unit 110 evaluates the consistency of the document 2 to be evaluated with the rule information based on the item information.

[0133] Here, for example, the rules applied in the QMS Ministerial Ordinance will differ depending on the company information such as the size of the company and business content, and the item information, and the items and contents that should be included in the quality control documents will differ. .

[0134] The requirements in the rule information for the document to be evaluated 2 are evaluated by taking into account organizational information and item information in the database 3, and if inconsistencies are detected, improvement suggestions are displayed, allowing quality control documents to be created efficiently and with greater accuracy.

[0135] (11) The program is a program that causes a computer 18 to execute processing for evaluating quality control documents, and the processing includes a storage processing step of storing rule information regarding quality control rules in a database 3, a reception step of receiving input of a document to be evaluated 2, which is a quality control document to be evaluated, an evaluation step of analyzing the semantic relevance between the document to be evaluated 2 and the information in the database 3 and evaluating the consistency of the document to be evaluated 2 with the rule information using a trained model 4, a generation step of generating information about the inconsistency and improvement suggestions for the document to be evaluated 2 when an inconsistency that is not consistent with the rule information is identified in the document to be evaluated 2 based on the consistency evaluation results, and a display processing step of displaying the information about the inconsistency and the improvement suggestions.

[0136] (12) The quality control document evaluation method is a quality control document evaluation method for evaluating quality control documents, and includes a storage processing step of storing rule information regarding quality control rules in a database, a reception step of receiving input of an evaluation target document 2, which is a quality control document to be evaluated, an evaluation step of analyzing the semantic relevance between the evaluation target document 2 and information in a database 3 and evaluating the consistency of the evaluation target document 2 with the rule information using a trained model 4, a generation step of generating information about the inconsistency and improvement suggestions for the evaluation target document 2 when an inconsistency point that is not consistent with the rule information is identified in the evaluation target document 2 based on the consistency evaluation result, and a display processing step of displaying the information about the inconsistency point and the improvement suggestions.

[0137] Although the embodiments of the present invention have been described above, the present invention is not limited to the above embodiments and can be modified as appropriate.

[0138] In the above embodiment, the search processing unit 113 extracted rule information and evaluation-related information related to the document 2 to be evaluated from the database 3, but the trained model 4 may divide the document 2 to be evaluated into semantic chunks, vectorize it, directly refer to the database 3, and extract rule information and evaluation-related information related to the document 2 to be evaluated from the database 3.

[0139] In the above embodiment, the search processing unit 113 extracts the review-related information together with the rule information related to the document 2 to be reviewed from the database 3, but it is also possible to extract only the rule information.

[0140] In the above embodiment, the quality control document evaluation system 1 generated an improvement plan after performing checkpoint processing as shown in Figure 15, but it may also generate an improvement plan after the consistency evaluation processing and before the checkpoint processing. [Explanation of symbols]

[0141] 1. Quality control document evaluation system 2. Documents to be evaluated 3 Database 4. Pre-trained model 101 Reception 102 Memory Processing Unit 103 Generation part 104 Display processing unit 110 Evaluation Department

Claims

1. A quality control document evaluation system for evaluating quality control documents, comprising: a storage processor configured to store rule information including at least one requirement related to a quality control rule in a database, the at least one requirement being divided into a first plurality of semantic chunks, the first plurality of semantic chunks being vectorized, and the vectorized first plurality of semantic chunks being stored in the database; a receiving unit that receives an input of a document to be evaluated, the document to be evaluated being a quality control document to be evaluated, the receiving unit including at least one sentence; evaluating a semantic relationship between the at least one requirement and the at least one sentence, evaluating the semantic relevance includes dividing the at least one sentence into a second plurality of semantic chunks, vectorizing the second plurality of semantic chunks, and calculating a degree of semantic relevance between each requirement included in the at least one requirement and each sentence included in the at least one sentence by calculating a similarity between the vectorized first plurality of semantic chunks and the vectorized second plurality of semantic chunks; Identifying requirements and sentences having high semantic relevance from the at least one requirement and the at least one sentence based on the calculated degree; and Evaluating the consistency between the identified requirements and the identified sentences using a trained model that has been trained to associate requirements, sentences, and consistency; Identifying the identified sentence as an inconsistent portion that does not conform to the identified requirements based on the evaluated consistency; an evaluation unit that performs the above; A quality control document evaluation system comprising: a display processing unit that displays the sentence identified as the inconsistent portion.

2. The quality control document evaluation system according to claim 1 , wherein the rule information includes the QMS Ministerial Ordinance for Medical Devices, ISO 13485, ISO 14971, and related guidelines.

3. The quality control document evaluation system according to claim 2 , wherein the rule information includes an internal rule of an organization to which the user belongs.

4. A program for causing a computer to execute a process for evaluating quality control documents, The process comprises: a storage process step of storing rule information including at least one requirement related to a quality control rule in a database, wherein the at least one requirement is divided into a first plurality of semantic chunks, the first plurality of semantic chunks are vectorized, and the vectorized first plurality of semantic chunks are stored in the database; a receiving step of receiving an input of a document to be evaluated, which is a quality control document to be evaluated, the document to be evaluated including at least one sentence; an evaluation step for evaluating a semantic relationship between the at least one requirement and the at least one sentence, the evaluation step includes a step of dividing the at least one sentence into a second plurality of semantic chunks, vectorizing the second plurality of semantic chunks, and calculating a degree of semantic association between each requirement included in the at least one requirement and each sentence included in the at least one sentence by calculating a similarity between the vectorized first plurality of semantic chunks and the vectorized second plurality of semantic chunks; an identifying step of identifying a requirement and a sentence having a high semantic relevance from the at least one requirement and the at least one sentence based on the calculated degree; an evaluation step of evaluating the consistency between the identified requirements and the identified sentences using a trained model that has been trained by associating requirements, sentences, and consistency; an identifying step of identifying the identified sentence as an inconsistent portion that does not conform to the identified requirements based on the evaluated consistency; an evaluation step of: a display processing step of displaying the sentence identified as the inconsistent portion.

5. A quality control document evaluation method for evaluating quality control documents, the method being executed by a processor, the method comprising: a storage process step of storing rule information including at least one requirement related to a quality control rule in a database, wherein the at least one requirement is divided into a first plurality of semantic chunks, the first plurality of semantic chunks are vectorized, and the vectorized first plurality of semantic chunks are stored in the database; a receiving step of receiving an input of a document to be evaluated, which is a quality control document to be evaluated, the document to be evaluated including at least one sentence; an evaluation step for evaluating a semantic relationship between the at least one requirement and the at least one sentence, the evaluation step includes a step of dividing the at least one sentence into a second plurality of semantic chunks, vectorizing the second plurality of semantic chunks, and calculating a degree of semantic association between each requirement included in the at least one requirement and each sentence included in the at least one sentence by calculating a similarity between the vectorized first plurality of semantic chunks and the vectorized second plurality of semantic chunks; an identifying step of identifying a requirement and a sentence having a high semantic relevance from the at least one requirement and the at least one sentence based on the calculated degree; an evaluation step of evaluating the consistency between the identified requirements and the identified sentences using a trained model that has been trained by associating requirements, sentences, and consistency; an identifying step of identifying the identified sentence as an inconsistent portion that does not conform to the identified requirements based on the evaluated consistency; an evaluation step of: a display processing step of displaying the sentence identified as the inconsistent portion.

Images