Document review support system and document review support method

Abstract

To accurately identify description portions of a document corresponding to check items, so as to support operations related to document review.SOLUTION: Provided is a document review support system for supporting document review performed using a checklist in which one or more check items are described. The document review support system is configured to: store a document having a hierarchical description structure, and an existing checklist in which corresponding parts of the document to be referred to when checking the check items are hierarchically indicated; based on the hierarchy indicated for the check items in the existing checklist, identify reliable hierarchies of the corresponding parts for the respective check items; and generate information indicating correspondence respectively between the check items and the identified hierarchies.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to a document review support system and a document review support method. [Background technology]

[0002] Patent Document 1 describes a computer system configured for the purpose of efficiently generating a judgment rule used when extracting a passage determined to be important from a document. The computer system compares a plurality of first documents with reference information including a word designated as an important passage, divides the first documents into a first element including the designated word and a second element not including the designated word, acquires a first feature amount of each document based on the words included in the first element and the second element, generates a judgment rule for judging whether or not an important passage is included based on the first feature amount, acquires a second feature amount based on the words included in the second document, and generates information for extracting an important passage included in the second document by comparing the generated judgment rule with the second feature amount. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2011-238159 A Summary of the Invention [Problem to be solved by the invention]

[0004] Design documents created during the development of various systems are reviewed using a checklist containing check items to ensure quality. Such document reviews are usually performed by identifying the location (location) of evidence in the document used to determine whether the document meets the requirements of each check item, which are set based on a variety of perspectives, and then manually checking the content based on the evidence described in that location. Therefore, there is an issue of a large workload for people, especially when the document is a huge number of pages long or there are many check items.

[0005] As a method for reducing the workload of people involved in reviews, for example, it is possible to identify the sections of the document that correspond to each check item as important sections by using the sections of evidence entered by people in a checklist in past review results as reference information in Patent Document 1, and to carry out a review using the identified information. However, the accuracy of the information entered in the checklist is not necessarily guaranteed, and when sections of the document are identified based on that information, sufficient reliability cannot necessarily be ensured.

[0006] The present invention has been made in consideration of the above background, and aims to provide a document review support system and a document review support method that are capable of accurately identifying the portions of a document that correspond to check items and supporting tasks related to document review. [Means for solving the problem]

[0007] One of the present inventions for achieving the above-mentioned object is a system for supporting document review using a checklist containing one or more check items, which is constructed using an information processing device, stores a document having a hierarchical structure and an existing checklist in which the relevant parts of the document that are referenced when reviewing the check items are indicated in a hierarchy, identifies a hierarchy of the relevant parts with high reliability for the check items based on the hierarchy indicated for the check items in the existing checklist, and generates information indicating the correspondence between the check items and the identified hierarchy.

[0008] Other problems and solutions disclosed in the present application will be made clear in the detailed description and drawings. Effect of the Invention

[0009] According to the present invention, it is possible to accurately identify the portions of a document that correspond to check items and to support the work of reviewing a plan. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating a flow of reviewing a design document using a checklist. [Diagram 2] This is an example of a checklist. [Diagram 3] 13 is an example of a description location correspondence table. [Figure 4] FIG. 11 is a diagram illustrating an example of a characteristic word extraction procedure. [Diagram 5] This is a diagram explaining that the accuracy of the relevant parts of the checklist is not necessarily guaranteed. [Figure 6] FIG. 2 is a block diagram illustrating main functions of the support system. [Figure 7] 1 is an example of an information processing device used to realize the support system. [Figure 8] 11 is a flowchart illustrating an example of a teacher data generation process. [Figure 9] FIG. 1 is a diagram showing an example of analysis of the structure of a chapter or section of a design document. [Figure 10] FIG. 13 is a diagram illustrating an example of a calculation process of a chapter / section level. [Figure 11] This is an example of a method for identifying a highly reliable corresponding portion and a less reliable corresponding portion. [Figure 12] This is an example of a method for identifying a highly reliable corresponding portion and a less reliable corresponding portion. [Figure 13] This is an example of a method for identifying a true relevant portion. [Figure 14A] 11 is a flowchart illustrating an example of a relevant portion estimation process. [Figure 14B] 11 is a flowchart illustrating an example of a relevant portion identification process. [Figure 15] 11 is a diagram illustrating an example of a method for calculating a similarity between a highly reliable corresponding portion and a low reliable corresponding portion; FIG. [Figure 16] FIG. 13 is a diagram illustrating an example of a method for generating teacher data. [Figure 17] This is a diagram for explaining a case where the level of abstraction of the description for the same chapter / section level differs for each design document (each case). [Figure 18] 13 is a flowchart illustrating an example of a chapter / section level correction process. [Figure 19] 11A to 11C are diagrams illustrating each step of the chapter / section level correction process. [Figure 20] 13 is a flowchart illustrating an example of a check item removal process. [Figure 21] 13A to 13C are diagrams illustrating each step of the check item removal process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, the embodiment will be described with reference to the drawings. Hereinafter, the embodiment of the present invention will be described with reference to the drawings. The following description and drawings are examples for explaining the present invention, and are omitted and simplified as appropriate for clarity of explanation. The present invention can be implemented in various other forms. Unless otherwise limited, each component may be singular or plural.

[0012] In the following description, the same or similar components may be designated by the same reference numerals, and duplicate descriptions may be omitted. In addition, in the following description, the letter "S" before a reference numeral indicates a processing step. In addition, in the following description, various types of information may be described using expressions such as "table" and "information", but the various types of information may be expressed using data structures other than these.

[0013] FIG. 1 is a diagram showing the comparison of the current state and the state after the mechanism of this embodiment is applied with respect to the work related to the review of design documents 111 created when developing various systems. In this embodiment, the design document 111 is, for example, a design document (basic design document, detailed design document, etc.) created when developing an information processing system, but the type of document to be reviewed is not necessarily limited to a design document, and the mechanism of this embodiment can be widely applied to documents in general.

[0014] As shown in the upper part of the figure, currently, the review of the design document 111 (self-review, design review, etc.) is carried out by, for example, a system engineer (SE) or a QA This is mainly done manually by a person such as E (Quality Assurance Engineer) (hereinafter referred to as the "evaluator"). Specifically, the evaluator first identifies the location (location) in the design document 111 of information (hereinafter referred to as "evidence") used to determine whether or not each check item in the checklist 112 satisfies the requirements. Next, the evaluator checks the evidence written in the identified location, determines whether or not the design document satisfies the requirements of each check item, and enters the determination result in the checklist 112.

[0015] 2 shows an example of the checklist 112 used in the above judgment. The illustrated checklist 112 is a sheet (paper medium or electronic medium) with a predetermined number of lines that includes items such as item ID 1121, check items 1122, judgment results 1123, and corresponding locations 1124.

[0016] Of the above items, item ID 1121 describes an identifier (hereinafter referred to as "check item ID") assigned to each check item. Check item 1122 describes a document indicating the content of the check item (such as requirements that the design document must satisfy). Judgment result 1123 is entered by the evaluator with respect to the check item; for example, if the design document satisfies the requirements of the check item, "OK" is entered in judgment result 1123, and if not, "NG" is entered. Corresponding location 1124 is entered by the evaluator with information indicating the location of evidence in design document 111 being reviewed for the check item.

[0017] Returning to Fig. 1, as shown in the lower part of the figure, in this embodiment, a design document review support system (hereinafter referred to as "support system 100") configured using an information processing device supports the work of a person involved in reviewing a design document 111. Specifically, the support system 100 automatically identifies the description locations in the design document 111 of evidence used to determine whether or not each item in a checklist 112 satisfies the requirements, and supports the work of a person who reviews the design document 111 by providing a description location correspondence table 120, which is information showing the identified results as information showing the correspondence between the chapter / section numbers of the design document 111 and the check items.

[0018] 3 shows an example of the description location correspondence table 120. In the example description location correspondence table 120, a list 1201 of check items is arranged in the row direction, and a list 1202 of chapters, sections, and clauses of the design document is arranged in the column direction. A symbol "○" is set in the column of the chapter, section, and clause (corresponding section) of the design document 111 in which the evidence corresponding to the check item is described. When reviewing the design document 111, the assessor can easily refer to the description locations in the design document 111 of the evidence corresponding to the check item by referring to the description location correspondence table 120, and the assessor can efficiently proceed with the review work of the design document 111.

[0019] Returning to FIG. 1, the support system 100 divides the design document 111 into chapters, sections, and items, and generates training data 114 by identifying which chapter, section, and item each check item in the checklist 112 corresponds to. The support system 100 generates feature word definitions 119 using a machine learning model (hereinafter, "feature word extraction model 115") trained using the training data 114 generated based on existing information (a previously created design document 111 and a checklist 112 used for reviewing it). The feature word definitions 119 include one or more feature words for each check item. This is information that defines characteristic words (words used when identifying the relevant parts corresponding to the check items from the design document 111).

[0020] 4 is a diagram showing an example of a procedure (hereinafter referred to as "characteristic word extraction procedure S400") in which the support system 100 generates characteristic words for each check item using the characteristic word extraction model 115. In the following explanation, it is assumed that the design document 111 has a hierarchical structure, and the contents are described structured by chapters and sections.

[0021] As shown in the figure, first, the support system 100 generates training data 114 based on existing information (design document 111 and checklist 112 in a past review of the design document 111) (S411).

[0022] Next, the support system 100 generates a characteristic word extraction model 115 that is trained using the teacher data 114 and existing information (existing design document 111, existing checklist 112), and generates characteristic word candidates 116, which are information indicating candidate characteristic words for each check item, based on the generated characteristic word extraction model 115 (S412).

[0023] Next, the support system 100 generates standard feature word definitions 117 by brushing up the generated feature word candidates 116, for example, through dialogue with the user via a user interface (S413). The brushing up is performed, for example, by measuring accuracy based on correct answer data and selecting feature words to improve the accuracy.

[0024] Next, the support system 100 compares the standard characteristic word definition 117 with a synonym dictionary 118 that has been generated separately (S414) based on existing design documents 111, thereby adjusting differences in terminology (e.g., differences in terminology (spelling variations) between design documents 111 for different projects) and generating characteristic word definitions 119 (S415).

[0025] Furthermore, the support system 100 may provide a user interface (such as providing an editable screen) for editing (registering, updating, deleting, searching) the various information described above (design document 111, checklist 112, teacher data 114, feature word candidates 116, standard feature word definitions 117, synonym dictionary 118, feature word definitions 119), allowing the user to manually edit this information.

[0026] As shown in the figure, the support system 100 uses an existing checklist 112 when generating teacher data 114. However, the evaluator's main interest when reviewing the design document 111 is "whether the design document 111 to be reviewed satisfies the requirements of the check items," and the accuracy of the contents entered by the evaluator in the relevant section 1124 of the checklist 112 is not necessarily guaranteed.

[0027] 5 is a diagram for explaining that the accuracy of the content entered by the evaluator in the relevant section 1124 of the checklist 112 is not necessarily guaranteed. As shown in the figure, for example, even if the true description section of the evidence in question is chapter, section, and paragraph number "2.2.3", the evaluator may enter, for example, "2.2" as the chapter, section, and paragraph number in the relevant section 214 of the checklist 112. In this way, since the accuracy of the content entered by the evaluator in the relevant section 1124 of the checklist 112 is not necessarily guaranteed, even if the characteristic word extraction model 115 is trained using the teacher data 114 generated by using the existing checklist 112 as is, the extraction accuracy of the characteristic words cannot necessarily be sufficiently ensured.

[0028] Therefore, in this embodiment, the support system 100 is based on the following viewpoints (1) to (3). The reliability of the feature word definition 119 is improved by generating teacher data 114 based on the classification.

[0029] (1) In Design Document 111, the lower the chapter / section level, the more specific the contents and the more narrow the topic. (2) The higher the chapter / subsection level of a design document, the more likely it is that various topics will be mixed together. (3) Among the relevant sections listed in the checklist 112, the lower the chapter / section level, the greater the proportion of topics highly related to the check item, and the higher the reliability.

[0030] Specifically, for example, the support system 100 is provided with the following functions.

[0031] (a) When narrowing down the highly reliable relevant parts, the parts with the lowest chapter / subsection level among the relevant parts shown in the checklist 112 are regarded as the most reliable relevant parts. (b) When narrowing down the highly reliable relevant parts, those parts shown in the checklist that are at the chapter / subsection level or lower than the mode will be considered to be highly reliable relevant parts. (c) When narrowing down the highly reliable relevant sections, those sections shown in the checklist 112 that do not have subchapters or subsections are deemed to be highly reliable relevant sections. (d) Identifying true hits within low-confidence hits based on high-confidence hits.

[0032] 6 is a block diagram for explaining the main functions of the support system 100. As shown in the figure, the support system 100 has the following functions: a storage unit 110, a data acquisition management unit 125, a teacher data generation unit 130, a feature word extraction model generation unit 140, a feature word candidate generation unit 145, a standard feature word definition generation unit 150, a synonym dictionary generation unit 155, a feature word definition generation unit 160, and a description location correspondence table generation unit 165.

[0033] Of the above functions, the memory unit 110 stores a design document 111, a checklist 112, structural analysis results 113, training data 114, a feature word extraction model 115, feature word candidates 116, standard feature word definitions 117, a synonym dictionary 118, feature word definitions 119, the training data 114, and various intermediate data 121.

[0034] The data acquisition management unit 125 acquires the design document 111 and the checklist 112 via a user interface or communication, and manages the acquired design document 111 and checklist 112 in the storage unit 110 .

[0035] The teacher data generation unit 130 generates teacher data 114 based on an existing design document 111 from a past review and its checklist 112. As shown in the figure, the teacher data generation unit 130 includes the functions of a design document structure analysis unit 131, a chapter / section level correction unit 133, a relevant part identification unit 134, and a teacher data output unit 135.

[0036] The characteristic word extraction model generation unit 140 generates a characteristic word extraction model 115 which is a machine learning model trained using the teacher data 114 .

[0037] The feature word candidate generation unit 145 generates feature word candidates 116 based on the feature word extraction model 115. The feature word extraction model 115 extracts feature words for each check item, for example, by statistically processing words included in documents described in chapters and sections corresponding to check items (e.g., determining as feature words words whose occurrence frequency exceeds a predetermined reference value).

[0038] The standard feature word definition generating unit 150 generates the feature word candidates 116 by, for example, the above-mentioned brush-up. By performing a search, standard feature word definitions 117 are generated.

[0039] The synonym dictionary generating unit 155 generates the synonym dictionary 118 based on a plurality of existing design documents 111 (such as a plurality of design documents 111 for different projects (cases)).

[0040] The characteristic word definition generating unit 160 compares the standard characteristic word definition 117 with the synonym dictionary 118 to adjust for differences in terminology (for example, differences in terminology (spelling variations) for each software development project) and generates a characteristic word definition 119.

[0041] The description location correspondence table generating unit 165 generates the description location correspondence table 120 by applying the characteristic word definition 119 to the design document 111. For example, the description location correspondence table generating unit 165 identifies, for each check item, the chapter or section in which each characteristic word appears frequently as the description location.

[0042] 7 shows an example of an information processing device used to realize the assistance system 100. The illustrated information processing device 10 includes a processor 11, a main storage device 12 (memory), an auxiliary storage device 13, an input device 14, an output device 15, and a communication device 16. Examples of the information processing device 10 include personal computers, office computers, various server devices, smartphones, tablets, general-purpose machines (mainframes), and the like.

[0043] The information processing device 10 may be realized, in whole or in part, by using virtual information processing resources provided by using virtualization technology, process space separation technology, or the like, such as a virtual server provided by a cloud system. Also, all or in part of the functions provided by the information processing device 10 may be realized, for example, by a service provided by the cloud system via an API (Application Programming Interface) or the like. All or part of the functions provided by the information processing device 10 may be provided as, for example, Software as a Service (SaaS), Platform as a Service (PaaS), Infrastructure as a Service (IaaS), etc. It may be realized by using the Service as a Service, etc.

[0044] The processor 11 may be, for example, a central processing unit (CPU), a micro processing unit (MPU), a graphics processing unit (GPU), a field programmable gate array (FPGA), or the like. It is composed of chips such as ASIC (Application Specific Integrated Circuit), AI (Artificial Intelligence), etc.

[0045] The main memory device 12 is a device for storing programs and data, and is, for example, a ROM (Read Only Memory). The main memory 12 may be a RAM (Random Access Memory), a non-volatile memory (NVRAM (Non Volatile RAM)), etc. The functions of each component of the assistance system 100 are realized by the processor 11 reading and executing a program stored in the main memory 12.

[0046] The auxiliary storage device 13 is, for example, a solid state drive (SSD) or a hard disk drive. The auxiliary storage device 13 may be a non-transitory storage medium, such as a hard disk, an optical storage device (such as a CD (Compact Disc) or a DVD (Digital Versatile Disc)), a storage system, a reader / writer for a non-transitory storage medium, such as an IC card, an SD card or an optical storage medium, or a non-transitory storage area of ​​a cloud server. Programs and data can be read into the auxiliary storage device 13 from other information processing devices equipped with non-transitory storage media or non-transitory storage devices via a recording medium reader or a communication device 16. The programs and data stored (memorized) in the auxiliary storage device 13 are read into the main storage device 12 as needed.

[0047] The input device 14 is an interface that accepts input of information from the outside, and may be, for example, a keyboard, a mouse, a touch panel, a card reader, a pen-input tablet, a voice input, or the like. power devices, etc.

[0048] The output device 15 is an interface that outputs various information such as the process progress and the process result to the outside. The output device 15 is, for example, a display device (liquid crystal monitor, LCD (Liquid Crystal Display), graphic card, etc.) that visualizes the above-mentioned various information, a device that converts the above-mentioned various information into voice (voice output device (speaker, etc.)), and a device that converts the above-mentioned various information into text (printer, etc.). For example, the information processing device 10 may be configured to input and output information to and from other devices via the communication device 16.

[0049] The input device 14 and the output device 15 constitute a user interface that realizes interactive processing with the user (receiving information, providing information, etc.).

[0050] The communication device 16 is a device that realizes communication with other devices. The communication device 16 is a wired or wireless communication interface that realizes communication with other devices that comply with a predetermined communication protocol via the communication network 5, and is, for example, a NIC (Network Interface Card), a wireless communication module, a USB module, or the like.

[0051] The information processing device 10 may be implemented with, for example, an operating system, a file system, a DBMS (DataBase Management System) (relational database, NoSQL, etc.), a KVS (Key-Value Store), etc.

[0052] 8 is a flowchart for explaining a process (hereinafter referred to as "teacher data generation process S800") performed by the teacher data generating unit 130 of the support system 100. Below, the teacher data generation process S800 will be explained with reference to this figure.

[0053] First, the teacher data generation unit 130 reads the existing design document 111 and the existing checklist 112 (S811). In this example, the teacher data generation unit 130 reads a plurality of existing design documents 111 and a plurality of existing checklists 112 (S811).

[0054] Next, the design document structure analysis unit 131 of the teacher data generation unit 130 analyzes the chapter, section, and clause structure of the design document 111, and outputs the analysis result as the structure analysis result 113 (S812). Note that the design document structure analysis unit 131 analyzes the chapter, section, and clause structure of the design document 111, for example, by using a known document structure analysis technique (for example, the technique described in Patent Document 1 (JP Patent Publication No. 2011-238159)).

[0055] Next, the chapter, section, and item level calculation unit 132 of the teacher data generation unit 130 calculates the level of the chapter, section, and item at the relevant part of the design document 111 corresponding to each check item in the checklist 112 (hereinafter referred to as the "chapter, section, and item level"), and reflects the calculated result in the structural analysis result 113 (S813).

[0056] Next, the chapter, section, and item level correction unit 133 of the teacher data generation unit 130 corrects the chapter, section, and item levels of each of the multiple design documents 111 that have been read (S814). This will be done appropriately.

[0057] Next, the relevant portion identifying unit 134 of the teacher data generating unit 130 identifies highly reliable relevant portions of the design document 111 for each check item of the checklist 112, in units of chapters and paragraphs of the design document 111 (S815).

[0058] Next, the relevant part specifying unit 134 estimates a true relevant part included in the relevant parts with low reliability in the design document 111 (S816).

[0059] Next, the teacher data output unit 135 of the teacher data generation unit 130 generates and outputs the teacher data 114 based on the results of the above processes (S817).

[0060] Next, each step in the teacher data generation process S800 shown in Fig. 8 will be described in detail with a specific example. Note that the process of S814 (correction at the chapter / section level) in the teacher data generation process S800 will be described separately later.

[0061] Fig. 9 is a diagram for explaining an example of analysis of the structure of the chapters and sections of the design document 111 in S812 in Fig. 8. In this example, a structural analysis result 113 is generated by analyzing a plurality of design documents 111 (design document 111 for "Project A", design document 111 for "Project B", design document 111 for "Project C", etc.).

[0062] The structural analysis result 113 illustrated in the bottom row of the figure is a table made up of a plurality of records having the following items: case name 1311, chapter / section number 1312, chapter / section title 1313, and main text 1314. Of the above items, case name 1311 stores the name of the case (case identifier (case ID)). Chapter / section number 1312 stores the chapter / section number. Chapter / section title 1313 stores the title given to the chapter / section of the design document 111. Main text 1314 stores the contents of the main text of the chapter / section number.

[0063] Fig. 10 is a diagram for explaining the process of calculating the chapter, section, and item level in S813 in Fig. 8. As shown in the figure, in this example, the chapter, section, and item level calculation unit 132 of the teacher data generation unit 130 identifies the chapter, section, and item corresponding to each check item for each case, and calculates the chapter, section, and item level for the identified chapter, section, and item. In this example, the chapter, section, and item level is determined according to the depth of the chapter, section, and item. The chapter, section, and item level calculation unit 132 manages the chapter, section, and item level determined as above in chapter, section, and item level data 1211, which is one of the various intermediate data 121.

[0064] FIG. 11 is a diagram for explaining an example of a method in which the relevant part identifying unit 134 of the teacher data generating unit 130 identifies reliable relevant parts and unreliable relevant parts in S815 of FIG. 8. In this example, the relevant part identifying unit 134 identifies, for each check item, the relevant part 1124 in the checklist 112 with the lowest chapter, section, and item level as a reliable relevant part, and identifies the rest as unreliable relevant parts (the above-mentioned function (a)). In this example, the reliable relevant parts and unreliable relevant parts are identified based on the viewpoint that the lower the chapter, section, and item level, the higher the accuracy. The teacher data generating unit 130 manages the results identified above in a relevant part list 1212, which is one of the various intermediate data 121.

[0065] FIG. 12 is a diagram for explaining another example of a method in which the relevant part identifying unit 134 identifies reliable relevant parts and unreliable relevant parts in the design document 111 on a chapter, section, and item basis in S815 of FIG. 8. In this example, the relevant part identifying unit 134 obtains the most frequent value (highest level of occurrence) at the chapter, section, and item level for each check item, and identifies relevant parts 1124 in the checklist 112 that are at the most frequent value or lower than the most frequent value as reliable relevant parts, and others as unreliable relevant parts (the above-mentioned function (b)). In this example, the more cases a part is selected, the more reliable it is, and the more reliable and unreliable it is identified based on this viewpoint. The teacher data generating unit 130 manages the results identified above in the relevant part list 1212, which is various intermediate data 121.

[0066] In addition to the methods shown in Figures 11 and 12, for example, the relevant part identification unit 134 may determine that among the relevant parts shown in the checklist 112, those that do not have any lower-level chapters or sections are highly reliable relevant parts, and determine the rest as low-reliability relevant parts (function (c) described above).

[0067] 13 to 15 are diagrams for explaining the function of the relevant part identifying unit 134 in identifying a true relevant part in S816 of FIG. 8 (the above-mentioned function (d)). As shown in the figures, the relevant part identifying unit 134 estimates a true relevant part included in a low-reliability relevant part. Specifically, when a low-reliability relevant part has a lower chapter / section term, the relevant part identifying unit 134 compares the similarity between the high-reliability relevant part and the higher-reliability relevant part between the higher-reliability chapter / section term and the lower-reliability chapter / section term, and estimates the lower-reliability chapter / section term having a higher similarity than the higher-reliability chapter / section term as the true relevant part. In this case, when the higher-reliability chapter / section term has a higher similarity than all the lower-reliability chapter / section terms, the relevant part identifying unit 134 may estimate the higher-reliability chapter / section level as the true relevant part.

[0068] Fig. 14A is a flowchart for explaining the process (hereinafter referred to as "relevant part estimation process S1400") performed by the relevant part identification unit 134 in the method shown in Fig. 13. Hereinafter, the relevant part estimation process S1400 will be explained with reference to the same figure.

[0069] The process of S1410S to S1410E shown in the figure is a loop process (hereinafter referred to as a "first loop process") that is repeatedly performed by sequentially selecting check items on the check sheet one by one.

[0070] Moreover, the loop process of S1420S to S1420E inside the first loop process is a loop process (hereinafter referred to as "second loop process") that is repeatedly performed by sequentially selecting low-reliability hit portions in the hit portion list 1212 described above one by one.

[0071] In S1430 of the second loop process, the relevant portion identifying unit 134 identifies a portion to be estimated as a true relevant portion based on the selected check item and the selected relevant portion X having low reliability.

[0072] Fig. 14B is a flowchart for explaining the details of the process of S1430 in Fig. 14A (hereinafter referred to as "relevant part identification process (X) S1430"). Below, the relevant part identification process (X) S1430 will be explained with reference to this figure.

[0073] First, the relevant part identification unit 134 judges whether the relevant part X has a lower chapter, section, or item (S1431). If the relevant part X has a lower chapter, section, or item (S1431: YES), the process proceeds to S1432. On the other hand, if the relevant part X does not have a lower chapter, section, or item (S1431: NO), the process proceeds to S1434.

[0074] In S1432, the relevant passage identifying unit 134 calculates the similarity between the relevant passage X and the highly reliable relevant passages in the relevant passage list 1212 described above, for all the chapters and sections lower than the relevant relevant passage.

[0075] In S1433, the corresponding part identifying unit 134 determines whether or not there is a lower chapter, section, or term having a higher similarity than the corresponding part X. If there is a lower chapter, section, or term having a higher similarity than the corresponding part X (S1443: YES), the process proceeds to S1440S. On the other hand, if there is no lower chapter, section, or term having a higher similarity than the corresponding part X (S1443: NO), the process proceeds to S1434.

[0076] In S1434, the relevant part identification unit 134 adds the relevant part X to the highly reliable relevant parts in the relevant part list 1212 of the selected check item. Thereafter, the processing for the currently selected check item and the currently selected low reliability relevant part is terminated, and the processing proceeds to S1410E in FIG. 14A.

[0077] The loop process from S1440S to S1440E (hereinafter referred to as the "third loop process") is This is a loop process in which lower-level chapter / section items Y that have a higher similarity to the relevant portion than the relevant portion X are selected one by one.

[0078] The process of S1430 in the third loop process (hereinafter referred to as "relevant part identifying process (Y) S1430") is a recursive process of the relevant part identifying process (X) S1403, which is performed with the selected chapter / section Y as the relevant part X. When the loop process of S1440S to S1440E ends, the process proceeds to S1410E in FIG. 14A.

[0079] 15 is a diagram for explaining an example of a method for calculating the above-mentioned similarity (similarity between a highly reliable relevant portion and a higher-level section term of a less reliable relevant portion or a lower-level section term of a less reliable relevant portion). The relevant portion identification unit 134 uses word distributed representations such as "TF-IDF" and "word2vec", topic models such as LDA (Latent Dirichlet Allocation), etc. Using existing technology, the text of the chapters and paragraphs of the design document 111 is converted into vectors. Then, the relevant part identification unit 134 calculates the similarity as, for example, cosine similarity. If there are multiple highly reliable relevant parts, for example, the average vector (centroid vector) of the vectors of each relevant part is used as the vector of the highly reliable relevant part to calculate the cosine similarity.

[0080] Fig. 16 is a diagram showing how the teacher data output unit 135 generates the teacher data 114 in S817 of Fig. 8. As shown in the figure, the teacher data output unit 135 generates the teacher data 114 by combining (merging) a highly reliable relevant portion 1611 identified in S815 of Fig. 8 and a highly reliable relevant portion 1612 identified in S816.

[0081] <Chapter-level corrections> 8, when multiple design documents 111 are read in S811, in calculating the chapter, section, and item level in S813, the teacher data generation unit 130 calculates the chapter, section, and item level for each design document 111. However, as illustrated in Fig. 17, the abstraction level of the description for the same chapter, section, and item level may differ for each design document 111 (for each case), in which case the standard for the chapter, section, and item level will shift, and therefore the chapter, section, and item levels of different design documents 111 (cases) cannot be simply compared.

[0082] Therefore, when the standards for the chapter, section, and item level differ between different design documents 111, the chapter, section, and item level correction unit 133 of the teacher data generation unit 130 performs a process of correcting (normalizing) the chapter, section, and item level (the process of S814 in FIG. 8. Hereinafter, this process will be referred to as "chapter, section, and item level correction process S814"). In this embodiment, as an example, a case will be described in which the chapter, section, and item level correction unit 133 corrects (normalizes) the chapter, section, and item level so that it can be compared between cases based on the chapter, section, and item level statistics of each design document 111 (each case).

[0083] Fig. 18 is a flowchart for explaining an example of the chapter, section, and item level correction process S814. Fig. 19 is a diagram for explaining each process shown in Fig. 18. The chapter, section, and item level correction process S814 will be explained below with reference to these diagrams.

[0084] As shown in the figure, first, the chapter / item level correction unit 133 calculates the average value of the chapter / item level of the corresponding part shown in the checklist 112 for each design document 111 (for each case) (S1811 in FIG. 18, S1911 to S1912 in FIG. 19).

[0085] Next, the chapter, section, and term level correction unit 133 selects a case to be used as a reference case (S1812 in FIG. 18, S1913 in FIG. 19). For example, the chapter, section, and term level correction unit 133 first creates a vector in which the chapter, section, and term levels are arranged (as components) for each case, and finds the center of gravity of the vectors of all cases. Next, the chapter, section, and term level correction unit 133 calculates the cosine similarity between the vector of each case and the center of gravity vector, and sets the case with the highest cosine similarity as the reference case. In the example of FIG. 19, the chapter, section, and term level correction unit 133 selects "Case A" as the reference case. .

[0086] Next, the chapter, section, and item level correction unit 133 calculates a relative level from the reference case for each case based on the difference between the chapter, section, and item level average value and the reference case (S1813 in FIG. 18, S1913 in FIG. 19). For example, the chapter, section, and item level correction unit 133 calculates a relative level from the reference case based on the average chapter, section, and item level value of each case, and sets an integer m that satisfies the following as the relative level from the reference case. [Number 1] m - 0.5 ≤ (Difference from the average value with the reference case) < m + 0.5 (Equation 1)

[0087] Subsequently, the section-item level correction unit 133 corrects the section-item level of each case using the relative level (S1814 in FIG. 18, S1914 in FIG. 19). For example, the section-item level correction unit 133 obtains the corrected section-item level from the following equation. [Number 2] Corrected section-item level = Section-item level before correction - Relative level from the reference case (Equation 2)

[0088] By the way, depending on the check item, there may be cases where the section-item level is not necessarily determined, such as when the degree of abstraction of the description is not constant. Therefore, when the section-item level is not necessarily determined, a process of excluding the check item for which the section-item level is not determined before executing the section-item level correction process S814 (hereinafter referred to as "check item exclusion process S2000") may be performed, for example, before executing the process of S1811 in FIG. 18.

[0089] FIG. 20 is a flowchart for explaining the check item exclusion process S2000. Also, FIG. 21 is a diagram for explaining each process shown in FIG. 20. Hereinafter, the check item exclusion process S2000 will be described together with these figures.

[0090] First, the section-item level correction unit 133 generates a correlation matrix of the section-item levels between the check items (S2011 in FIG. 20, S2111 - S2112 in FIG. 21).

[0091] Subsequently, the section-item level correction unit 133 counts the number k of correlations exceeding the threshold value a with other check items for each check item (S2012 in FIG. 20, S2113 in FIG. 21).

[0092] Next, the chapter, section, and item level correction unit 133 excludes check items whose ratio of k to the total number of check items is less than b% (S2013 in Fig. 20, S2113 in Fig. 21). Note that in the example of Fig. 21, the check item "item 3" has only 7% of its correlations with other check items exceeding 0.7, so the chapter, section, and item level correction unit 133 excludes the check item "item 3" as its chapter, section, and item level is not determined.

[0093] In such a case where the chapter, section, and item level is not necessarily determined, the chapter, section, and item level between different design documents 111 can be appropriately corrected (normalized) by excluding check items whose chapter, section, and item level is not determined before executing the chapter, section, and item level correction process S814.

[0094] As described above in detail, according to the support system 100 of this embodiment, the extraction accuracy of characteristic words can be improved by the characteristic word extraction model 115. Therefore, even if the accuracy of the contents entered by the assessor in the corresponding portion 1124 of the checklist 112 is not necessarily guaranteed, the characteristic word extraction model 115 can be trained with high accuracy by the teacher data 114 generated by using the existing checklist 112 as is, and the extraction accuracy of characteristic words can be improved. This makes it possible to accurately identify the portion of the design document 111 that corresponds to the check item, and to support the work related to the review of the design document.

[0095] Although one embodiment of the present invention has been described above, it goes without saying that the present invention is not limited to the above embodiment and can be modified in various ways without departing from the gist of the present invention. For example, the above embodiment has been described in detail to clearly explain the present invention, and is not necessarily limited to an embodiment having all of the configurations described. In addition, it is possible to add, delete, or replace part of the configuration of the above embodiment with other configurations.

[0096] In addition, the above-mentioned configurations, functional units, processing units, processing means, etc. may be realized in part or in whole by hardware, for example, by designing them as integrated circuits. In addition, the above-mentioned configurations, functions, etc. may be realized in software by a processor interpreting and executing a program that realizes each function. Information such as the program, table, file, etc. that realizes each function can be stored in a memory, a hard disk, a recording device such as an SSD (Solid State Drive), an I It can be placed on recording media such as C cards, SD cards, and DVDs.

[0097] The above-described layouts of the various functional units, processing units, and databases of each information processing device are merely examples, and may be changed to optimal layouts in terms of the performance, processing efficiency, and communication efficiency of the hardware and software included in these devices.

[0098] Furthermore, the configuration (schema, etc.) of the database that stores the various types of data described above can be flexibly changed from the perspective of efficient use of resources, improved processing efficiency, improved access efficiency, improved search efficiency, and the like. [Explanation of symbols]

[0099] 100 Support system, 110 Memory unit, 112 Checklist, 113 Structural analysis result, 114 Teacher data, 115 Feature word extraction model, 116 Feature word candidate, 117 Standard feature word definition, 118 Synonym dictionary, 119 Feature word definition, 120 Description location correspondence table, 121 Various intermediate data, 125 Data acquisition management unit, 130 Teacher data generation unit, 131 Design document structure analysis unit, 132 chapter / section / item level calculation unit, 133 chapter / section / item level correction unit, 134 relevant location identification unit, 135 teacher data output unit, 140 feature word extraction model generation unit, 145 feature word candidate generation unit, 150 standard feature word definition generation unit, 155 synonym dictionary generation unit, 160 feature word definition generation unit, 165 description location correspondence table generation unit, S800 teacher data generation process, S814 chapter / section / item level correction process, S1400 relevant location estimation process, S1430 relevant location identification process, S2000 check item exclusion process

Claims

1. A system for supporting document review using a checklist having one or more check items, comprising: The information processing device is configured to: A document having a hierarchical structure; An existing checklist in which the relevant parts of the document to be referred to when confirming the check items are indicated in a hierarchical manner; Remember, Identifying the hierarchical level of the check item with high reliability based on the hierarchical level indicated for the check item in the existing checklist; generating information indicating a correspondence between the check items and the identified hierarchical levels; Document review support system.

2. 2. The document review support system according to claim 1, For each of the check items, a checklist at the lowest level among the checklists created for each of the documents is identified as a relevant part, and the checklist is identified as a relevant part with high reliability. Document review support system.

3. 2. The document review support system according to claim 1, For each of the check items, among the hierarchical levels shown as relevant sections in the existing checklists created for each of the plurality of documents, a section having a frequency of occurrence that is the most frequent value or lower than the most frequent value is identified as the relevant section having high reliability. Document review support system.

4. 2. The document review support system according to claim 1, Identifying, as the highly reliable relevant portion, a check item that does not have a lower hierarchy among the hierarchies indicated as the relevant portion in the existing checklist for the check item; Document review support system.

5. 2. The document review support system according to claim 1, obtain a first similarity between the description of a hierarchical level of a relevant location identified as a highly reliable relevant location among the relevant locations indicated for the check items in the existing checklist and the description of a relevant location not identified as a highly reliable relevant location, and a second similarity between the description of a hierarchical level of a relevant location identified as a highly reliable relevant location and the description of a lower hierarchical level of the relevant location not identified as a highly reliable relevant location; If the second similarity is higher than the first similarity, the lower layer is also identified as the relevant portion with high reliability. Document review support system.

6. 6. A document review support system according to claim 5, When the first similarity is higher than the second similarity, a corresponding portion that has not been identified as the highly reliable corresponding portion is also identified as the highly reliable corresponding portion. Document review support system.

7. 2. The document review support system according to claim 1, The checklist includes a plurality of check items, When identifying the highly reliable relevant portion for each of the plurality of documents, a normalized hierarchy among the plurality of documents is used as the hierarchy of each of the check items described in the existing checklist. Document review support system.

8. The document review support system according to claim 7, a vector having components corresponding to the hierarchical levels of the check items in the existing checklist is calculated for each of the documents; a centroid vector of the vectors of the documents is calculated; and the document having the highest similarity between the vector of the documents and the centroid vector is selected as a criterion; calculating an average value of the hierarchical level for each of the check items described in the existing checklist for each of the plurality of documents, and using a value based on a difference between the average value of each of the plurality of documents and the average value of the document selected as the standard as a normalized hierarchical level for each of the plurality of documents; Document review support system.

9. The document review support system according to claim 7, determining a correlation between the check items for each of the plurality of documents when the hierarchy of each of the check items described in the existing checklist is used as an element; For each of the check items, the number of correlations between the check items and other check items that exceed a preset threshold is calculated; The normalization is performed by excluding the check items whose ratio of the number is smaller than a preset threshold value. Document review support system.

10. 2. The document review support system according to claim 1, generating a feature word extraction model, which is a machine learning model trained using information indicating the correspondence between the check items and the document hierarchies as training data; Document review support system.

11. The document review support system according to claim 10, generating, using the feature word extraction model, feature word definitions which are information indicating the correspondence between the check items and the feature words corresponding to the check items and which are used when identifying the relevant portions of the document corresponding to the check items; Document review support system.

12. 2. The document review support system according to claim 1, The hierarchical structure is a chapter / section structure. Document review support system.

13. 1. A method for assisting in the review of documents using a checklist having one or more check items, comprising: An information processing device, A document having a hierarchical structure; An existing checklist in which the relevant parts of the document to be referred to when confirming the check items are indicated in a hierarchical manner; storing the Identifying a hierarchy of the relevant portion having high reliability for the check item based on the hierarchy indicated for the check item of the existing checklist; generating information indicating a correspondence between the check items and the identified hierarchical levels; The document review support method includes:

14. A document review support method according to claim 13, comprising: The information processing device, a step of identifying, for each of the check items, the lowest level of a hierarchy indicated as a relevant portion in the existing checklist created for each of the plurality of documents as the relevant portion with high reliability; The document review support method further comprises:

15. A document review support method according to claim 13, comprising: The information processing device, a step of identifying, for each of the check items, a portion having a frequency of occurrence that is the most frequent value or lower than the most frequent value among the hierarchical levels shown as a relevant portion in the existing checklist created for each of the plurality of documents, as the relevant portion having high reliability; The document review support method further comprises:

Images